What SVP Settings are you using? I fixed the performance issues I was having with 4K videos already by using MPC x64 and changing the settings below.
I'm running a GTX 1070 and a 4790K @ 4.6Ghz on a Sony X800D 4K HDR TV.
First of all, you want to right click SVP and go to "Video Frame" and select "Keep Normal Size".
http://i.imgur.com/9CDlp0N.png
Second of all, I use the settings below for 4K videos, you don't need to have 3X Movie Frame Rate selected, just choose to SCREEN. The most important aspect, is to keep "Decrease Grid Step" on DISABLED.
http://i.imgur.com/36Cg9nm.png
I posted as links instead of images because on normal 1080p screens those images would take a lot of space for most people.
EDIT - MUST DO THESE:
I forgot, in Madvr Settings go to Devices (Left Menu) --> Choose your TV/Monitor --> Properties --> Under 3D Format, choose None.
Go to Rendering (Left Menu) --> Stereo 3D --> Disable ALL.
Go to Processing (Left Menu) --> Artifact Removal --> Disable BOTH.
Go to Scaling Algorithms (Left Menu) --> Image Downscaling --> Choose Lanczos (3 Taps).
--------------------------------------------- > Chroma Upscaling --> Jinc (Tick enable ONLY activate anti-ringing filter).
--------------------------------------------- > Image Upscaling --> Jinc (Tick enable BOTH activate anti-ringing filter & scale in sigmoidal light).