Try changing HDR in madVR to "passthrough" instead of "use pixel shaders" and make sure smooth motion is off ("smooth motion" is a madVR setting, not talking about SVP). I was also having issues with 4K HDR 90fps when I wasn't having performance issues before. I forgot I had those two settings on when I was testing some stuff earlier. I know madVR HDR tonemapping is a huge selling point but just compare it to passthrough anyway. If passthrough fixes your performance issues, try using ReShade as an alternative to pixel shaders.
Edit: This is in the case you forgot to mention you're trying to watch 4K HDR... If you're just watching regular 4K as your post suggests, then this won't help. Make sure madVR smooth motion is off regardless though.
Edit 2: Forgot to mention but even if you're watching 4K video on a 4K screen, there's a setting in "scaling algorithims -> image upscaling" that forces supersampling. Any of the "doubling" algorithms have a setting called "activate doubling/quadrupling" -> "always - supersampling" which means it'll supersample the 4K video regardless of screen resolution.
When I enabled it, the entire video went red and then crashed when I attempted to play. If you don't want to go through the settings like that, then a quick fix is to make sure upscaling is on something like Lanczos. Because you don't need upscaling for 4K video unless you have an 8K TV. For upscaling regular content, I prefer the look of Cubic (SoftCubic100) + SuperRes (different setting than supersampling, it's in "upscaling refinement") + a bit of Lumasharpen though instead of Lanczos or NGU.
Edit 3: In LAV Video Decoder, change "Hardware Acceleration" to "NVIDIA CUVID (old)" instead of "DXVA2 (copy-back)". CUVID helps a lot with 4K SVP. 480p/1080p 165fps SVP still works great without any dropped frames with CUVID too, so there's no downside that I know of.