I found out what caused the problem. It was hardware acceleration in LAV Video Decoder:
Hardware acceleration: None
GPU load: ca. 65%
CPU load: ca. 30%
Works fine.
DXVA2 (native):
GPUload: ca. 70%
CPU load: ca. 30%
Works fine.
All remaining hardware decoders cause heavy stuttering (GPU loads of ca. 90%).
Only DXVA2 (native) is working without stuttering. It seems that in case of a sufficiently powerful CPU there is no benefit from hardware acceleration. To the contrary, it can even be detrimental. Moreover, for HDR in madVR "let madVR decide" seems to work best.
This is my hardware:
Intel(R) Core(TM) i9-10980XE CPU @ 3.00GHz
128 GB RAM
NVIDIA GeForce RTX 3070
Samsung QE-65QN800A