this is the way VLC filter works - it overrides audio/video sync value and shifts video timestamps accordingly, to be able to buffer a few frames inside the filter.
for now you can't do what you want in VLC + SVP
still you can use any other video player...