I got SVP to work flawlessly on windows at 1080p and all settings set to max, but when I tried it on linux I got some video/sound desync even at 720p which I thought was because of linux gpu drivers or something like that.. And it turns out I was able to fix it by passing
--fps=23.976 --no-correct-pts
argument to mpv.. It seems like videos that has fps like ~23.810 will desync after a while when using SVP, and rounding up the video source fps to 23.976 fixes the issue.
Hope this helps for anybody else that is experiencing this issue.