SVP freezes when a video that buffers slow is played

When I play a video file from a remote FTP server (quite slow, but could do real-time playback fine), the video would start playing, but SVP would not activate. Pointing to the SVP tray icon would either say "No active playback" or simply no popup bubble. If it try to open SVP manager, it would not open and freeze SVP. During this issue, it would cause the playing video to stutter or freeze, force closing SVP with task manager would make the video back to normal right away.

I tested with local files and youtube, and it would be fine (since youtube is much faster)

I have tested potplayer, MPC-HC, both with Avisynth Filter and the old ffdshow raw filter. I also tried different renderers (madvr, EVRCP, MPC).

Would this issue be related to the buffer/cache SVP requires before it activates?

Running Windows SVP 4.6.0.273