This is very strange, your system is more than strong enough to handle 120FPS. Something is wrong, somewhere. Having seen the replies, I have by Chainik, he already explored most options for you.
Okay first of all don't set SVP to fixed FPS of 120. Set it to 5x video framerate.
Use MPC (Media Player Classic) + Madvr (in Madvr setting, go to rendering tab on the left menu, and then under it in Smooth Motion, make sure you enable it and select ALWAYS).
Right click SVP and go to video frame menu... select (choose both) KEEP Aspect Ratio and KEEP Normal Size... Untick (deselect) detect and cut off black bars.
Right click SVP and go to Outer Lighting menu... select disabled.
Right click SVP and go to Additional Settings menu... Go to GPU Acceleration and make sure you choose your GTX 1080 Ti
-
IF this fixes your issues and everything runs smoothly. Ask @Chainik to show you how to edit the full settings to make you choose 6x framerate. Because 6x framerate will be smoother than 5x. Your CPU should handle it.
I have a 4790K @ 4.6Ghz and a GTX 1070 and I can easily run 1080p content upscaled to 4K at 4x the framerate.
Native 4K content though is a different story.
EDIT:
OH and finally I forgot, right click desktop, go to nVidia Control Panel, go to Mange 3D Profiles on the left menu at the top, then go to Program Settings and find MPC, under Power Management Mode, choose Prefer maximum performance. And obviously make sure you're using the current latest drivers from nVidia.