New RIFE filter - 3x faster AI interpolation possible in SVP!!! (Page 11 of 85)

Fourth-generation Tensor Cores - Peak FP16 using the Sparsity feature:

2000 TFLOPS - NVIDIA H100 SXM5 (528 Tensor Cores, 700W TDP)
1600 TFLOPS - NVIDIA H100 PCIe (456 Tensor Cores, 350W TDP)

Third-Generation Tensor Cores - Peak FP16 using the Sparsity feature:

624 TFLOPS - NVIDIA A100 (432 Tensor Cores, 400W TDP)
320 TFLOPS - NVIDIA GeForce RTX 3090 Ti (336 Tensor Cores, 450W TDP)
284 TFLOPS - NVIDIA GeForce RTX 3090 FE (328 Tensor Cores, 350W TDP)
273 TFLOPS - NVIDIA GeForce RTX 3080 Ti (320 Tensor Cores, 350W TDP)
244 TFLOPS - NVIDIA GeForce RTX 3080 12GB (280 Tensor Cores, 350W TDP)
238 TFLOPS - NVIDIA GeForce RTX 3080 10GB FE (272 Tensor Cores, 320W TDP)
174 TFLOPS - NVIDIA GeForce RTX 3070 Ti (192 Tensor Cores, 290W TDP)
162.6 TFLOPS - NVIDIA GeForce RTX 3070 FE (184 Tensor Cores, 220W TDP)
129.6 TFLOPS - NVIDIA GeForce RTX 3060 Ti (152 Tensor Cores, 200W TDP)
102.4 TFLOPS - NVIDIA GeForce RTX 3060 (112 Tensor Cores, 170W TDP)

Second-Generation Tensor Cores - Peak FP16:

130.5 TFLOPS - NVIDIA TITAN RTX (576 Tensor Cores, 280W TDP)
113.8 TFLOPS - NVIDIA GeForce RTX 2080 Ti FE (544 Tensor Cores, 260W TDP)
107.6 TFLOPS - NVIDIA GeForce RTX 2080 Ti (544 Tensor Cores, 250W TDP)
89.2 TFLOPS - NVIDIA GeForce RTX 2080 Super FE (384 Tensor Cores, 250W TDP)
84.8 TFLOPS - NVIDIA GeForce RTX 2080 FE (368 Tensor Cores, 225W TDP)
80.5 TFLOPS - NVIDIA GeForce RTX 2080 (368 Tensor Cores, 215W TDP)
72.5 TFLOPS - NVIDIA GeForce RTX 2070 Super FE (320 Tensor Cores, 215W TDP)
63 TFLOPS - NVIDIA GeForce RTX 2070 FE (288 Tensor Cores, 185W TDP)
59.7 TFLOPS - NVIDIA GeForce RTX 2070 (288 Tensor Cores, 175W TDP)
57.4 TFLOPS - NVIDIA GeForce RTX 2060 Super (272 Tensor Cores, 175W TDP)
51.6 TFLOPS - NVIDIA GeForce RTX 2060 (240 Tensor Cores, 160W TDP)

First-generation Tensor Cores - Peak FP16:

125 TFLOPS - NVIDIA Tesla V100 (640 Tensor Cores, 300W TDP)

---------------------------------------------------------
H100
https://resources.nvidia.com/en-us-tensor-core
A100, V100
https://images.nvidia.com/aem-dam/en-zz … epaper.pdf
RTX 3090 FE, RTX 3080 10GB FE, RTX 3070 FE, TITAN RTX, RTX 2080 Super FE, RTX 2070 Super FE
https://images.nvidia.com/aem-dam/en-zz … per-V1.pdf
RTX 2080 Ti FE, RTX 2080 Ti, RTX 2080 FE, RTX 2080, RTX 2070 FE, RTX 2070
https://images.nvidia.com/aem-dam/en-zz … epaper.pdf
RTX 3090 Ti
https://www.anandtech.com/show/17334/nv … lpowerful-
https://www.techpowerup.com/gpu-specs/g … 0-ti.c3829
RTX 3080 Ti, RTX 3080 12GB, RTX 3070 Ti
https://www.anandtech.com/show/17204/nv … more-money
https://www.techpowerup.com/gpu-specs/g … 0-ti.c3735
https://www.techpowerup.com/gpu-specs/g … 2-gb.c3834
https://www.techpowerup.com/gpu-specs/g … 0-ti.c3675
RTX 3060 Ti, RTX 3060, RTX 2060
https://www.anandtech.com/show/16512/la … eam-at-329
https://www.techpowerup.com/gpu-specs/g … 0-ti.c3681
https://www.techpowerup.com/gpu-specs/g … 3060.c3682
https://www.techpowerup.com/gpu-specs/g … 2060.c3310
RTX 2060 Super
https://www.anandtech.com/show/16290/la … re-for-400
https://www.techpowerup.com/gpu-specs/g … uper.c3441

Hello!
Recently I got a new TV and it had interpolation techniques built into the software so I searched for the same thing for PC and I found this. I tried the standard algorithm and Nvidia optical flow but they were both artifacting so I tried RIFE. It has a lot less artifacts and the flow of the images are smooth I very like it. The only problem is my weak GPU. I have to downscale my source material for it to work realtime. My current setup is:

Source material:
8bit 1080p 23.976 fps 8Mb/s (anime)

Rife via Vulkan v4
Windows 10
MPC-BE x64 + LAV decoder (CUVID) + MadVR
GTX 1660 Super 6GB, 2100Mhz core, 7800Mhz memory

Downscale via Spline64 to 1280x720 (via avisynth)
Interpolate with rife (x2)
Upscale via MadVR Jinc back to 1920x1080 (1920x1078 actual)
GPU usage 71-74%
Power 100-110 W

I can also downscale by integer and use nearest neighbour for resizing for increased smoothness:
Downscale via Pointresize to 960x540
Interpolate with rife (x3)
Upscale via MadVR nearest neighbour
GPU usage 90-94%
Power 110-120W

Actually what I found strange was Rife with high GPU utilization was consuming less power than either NVOF or standard SVP algorithm, both used around 115-130W.
The quality suffers but the interpolation makes movement smooth and nice on those few scenes where there is camera movement. 540p was a bit too pixellated for my like so I remained at 720p interpolation with Rife.
I tried to increase the resolution but it would always make the video choppy and max out mx GPU utilization even when I only upped the resolution by a little like to 1408x792 (frequent dropped frames, GPU usage spiking, EDIT: after a minute of playback gpu usage normalizes and playback gets smooth with around 85% GPU usage EDIT2: it start spiking and dropping frames again even in not very busy scenes so I had to go back to 720p).
What do you think about my current settings? Do you think there is a better way of achieving real time interpolation with my computer and in better quality? I feel like x2 interpolation is smooth enough I don't need x3 but I would like to increase resolution or quality somehow if possible. I tried Cuda but it freezes my player on start, in mpv it worked but it's speed was half of vulkan v4.

I don't always watch content on the TV as it is shared with others. That is why I searched for a universal solution that I can use on PC. I don't want to pre-encode hundreds of episodes into higher framerate because I don't feel like making my my gpu work at 100% for weeks constantly and I don't want the size of my shows to extend and to amass terabytes of storage. That is why I seeked a real-time solution. But maybe you are right, I should just re-encode them with rife in full resolution, that would be the highest quality option. I can run transcoding with vulkan v4 at around 35 FPS and cuda pytorch at around 21-22 FPS at 1080p. Almost realtime but not quite. I just need 12 fps more.

Hello. It seems there's some problem in particular with SVP implementation of RIFE interpolation and/or VapourSynth - it runs out of CPU on the threads it uses for high FPS output.
(That at gpu_threads=8... could use 16 there but filter does not support it.)

720p 60 x2 works. So does 1080p 30 x2. 1080p 24 x3 runs into a CPU cap, getting 105 FPS output. 1080p 60 x2 same, getting 85 FPS output.
Encoding is *much* faster than this.

Hardware: GeForce 3080 (OC) and Ryzen 5950x.

Can anything be done to make it actually not run out of CPU?

Transcoding via SVP Manager with NVENC into H264 8Mb/s by the end it was working at 78 fps but it was climbing throughout the whole transcode. I am not sure if the fps counter in SVP Manager is accurate, is there any other way of keeping track of how fast my GPU is spitting out frames?

It is an LG Oled C1 48". It has a setting named TrueMotion and you can manually set de-judder and de-blur setting on a scale of 0 to 10. It seems like the video retains all of it's quality and the interpolation looks very smooth both in anime and movie footage. I would say it has less artifacts than RIFE x2 but on very few scenes (very choppy camera pans in anime) there is still some judder left. To be fair I believe on those scenes RIFE has problems as well but I haven't cross tested the same footage to compare them yet.

With my current knowledge, I don't understand this figure but I will believe you. Still, compromising resolution for interpolation accuracy is a hard decision and I will have to test more to find the sweet spot.

Today I found out that if I run a very simple script in Vapoursynth filter instead of SVP manager, I get a lot more consistent performance and without frame drops or gpu usage spikes. Now I can run 1536x864 resolution real time without any problems.

Thanks. I'm glad you learned something new thanks to this thread. That was exactly my goal.

I've always been interested in watching videos in a way that reflects reality as much as possible. That's what brought me to SVP long before I started posting here on the forum. It was SVP that allowed me to experience the magic of real-time interpolation even on very archaic hardware.

It wasn't until 2018 that I saw for the first time that even better quality could be achieved with AI: https://developer.nvidia.com/blog/trans … n-with-ai/
A year later, I was admiring on YouTube what DAIN could do, but I saw no hope that this technology could ever be used in real time.

It wasn't until RIFE that I came to believe and to this day I am still fascinated by it and share this fascination with others. However, I am still raising funds for a PC that will allow me to fully enjoy the possibilities of AI.

So I invite anyone interested, like me who does not yet have the right hardware, but even more so those who have the hardware and can share the results of their tests, to join the discussion.

I, for the time being, can at most deal with the theoretical side and that is what I am trying to do. I will soon have some very interesting information on what the possibilities currently are in the field:

- video frame interpolation
- video deblurring
- video super-resolution

Or maybe even all 3 in 1

I don't want to write too many things at once so as not to create too much confusion. However, it will be worth following this and my other thread dedicated to IFRNet: https://www.svp-team.com/forum/viewtopic.php?id=6666

Would a 4080 or 4090 be capable of running RIFE on full HDR 4K ?

How does it perform currently on a 3090?

When watching some 1080p/24fps films I still have slowdowns in some fast/complex scenes, it is always present when watching again the same scene.

It should not be related to hardware performance (3080Ti with only about 40% usage) and when the slowndown occure the load on the GPU decrease.

I think the RIFE filter just stop making interpolation for a few seconds that's why it can be perceived as a slowdown.

I tried to change the GPU threads number, from 2 to 4 but I see no change.

Do you have the same problem?

Is SVP going to update to the last release?
https://github.com/nihui/rife-ncnn-vulkan/releases

I also have the vertical bars artifacts which are quite anoying, but I think this is "normal" in an interpolation algorythm.

Looking at the release notes the rife-v4 model has not been updated.
Maybe the nccn code compiled in SVP has to be updated with an update?

I fixed that problem. Apparently Windows has it's own version of Python hidden in "WindowsApps". I got rid of it, and now that error is gone, but it's still not working.

Now when I try to play a video it plays for a half a second and then freezes indefinitely. There aren't any errors in the logs when this happens.

EDIT: So, in Plex it freezes as I said, but in MPC-HC, it just doesn't work. The performance monitor shows the SVP Index at a constant 0.5 (normal framerate, since I have RIFE set to x2). Setting the profile back to Automatic works as expected, it's just RIFE that isn't working (both the CUDA version and the normal version of RIFE in MPC-HC, although in Plex, the normal version works, and CUDA version freezes playback).

EDIT 2: I just found the note at the bottom of the wiki page saying it doesn't work for real-time transcoding, which is what I've been testing it with. That's unfortunate. Any plans to make that work? Also maybe getting it to use the tensor cores?

Just built vapoursynth-rife-ncnn-vulkan on Apple M1, and it only makes 50 fps out of 704*528@30 source video with v4 model.
Not exactly the performance I expected...
But at least it works, which is good news