New RIFE filter - 3x faster AI interpolation possible in SVP!!! (Page 4 of 85)

When I try to use RIFE following the instructions that the previous posts references post I get:

20:55:20.530 [E]: Playback [3d8c3fef]: VS - Python exception: Ran out of input
20:55:20.530 [E]: Playback [3d8c3fef]: VS - Traceback (most recent call last):
20:55:20.530 [E]: Playback [3d8c3fef]: VS - File \src\cython\vapoursynth.pyx\, line 2244, in vapoursynth.vpy_evaluateScript
20:55:20.530 [E]: Playback [3d8c3fef]: VS - File \src\cython\vapoursynth.pyx\, line 2245, in vapoursynth.vpy_evaluateScript
20:55:20.530 [E]: Playback [3d8c3fef]: VS - File \C:\Users\htpc user\AppData\Roaming\SVP4\scripts\3d8c3fef.py\, line 70, in <module>
20:55:20.530 [E]: Playback [3d8c3fef]: VS - smooth = interpolate(clip)
20:55:20.530 [E]: Playback [3d8c3fef]: VS - File \C:\Users\htpc user\AppData\Roaming\SVP4\scripts\3d8c3fef.py\, line 51, in interpolate
20:55:20.530 [E]: Playback [3d8c3fef]: VS - smooth = RIFE(input_m,model_ver=3.8,fp16=not rife_precision,scale=1.0 if rife_threads<=2 else 0.5,device_index=rife_gpu)
20:55:20.530 [E]: Playback [3d8c3fef]: VS - File \C:\Program Files (x86)\SVP 4\mpv64\lib\site-packages\vsrife\__init__.py\, line 77, in RIFE
20:55:20.530 [E]: Playback [3d8c3fef]: VS - model.load_model(os.path.join(os.path.dirname(__file__), model_dir), -1)
20:55:20.530 [E]: Playback [3d8c3fef]: VS - File \C:\Program Files (x86)\SVP 4\mpv64\lib\site-packages\vsrife\model38\RIFE_HDv3.py\, line 40, in load_model
20:55:20.530 [E]: Playback [3d8c3fef]: VS - self.flownet.load_state_dict(convert(torch.load('{}/flownet.pkl'.format(path))), False)
20:55:20.530 [E]: Playback [3d8c3fef]: VS - File \C:\Program Files (x86)\SVP 4\mpv64\lib\site-packages\torch\serialization.py\, line 608, in load
20:55:20.530 [E]: Playback [3d8c3fef]: VS - return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
20:55:20.530 [E]: Playback [3d8c3fef]: VS - File \C:\Program Files (x86)\SVP 4\mpv64\lib\site-packages\torch\serialization.py\, line 777, in _legacy_load
20:55:20.530 [E]: Playback [3d8c3fef]: VS - magic_number = pickle_module.load(f, **pickle_load_args)
20:55:20.530 [E]: Playback [3d8c3fef]: VS - EOFError: Ran out of input

This then crashes the media player, after playing sound for say 10 second, video just hangs as soon as enabled.

I also tried with Vulkan, so set Test-Time Augmentaion: Disabled
This does not hang, but is unusable in real-time I get a 10 or so frames a second at best as is expected.

21:12:59.698 [i]: Playback [91b99dbe]: playing at 59.94 [23.976 *5/2] 
21:13:25.245 [i]: Playback [91b99dbe]: switching to profile 'RIFE AI engine'
21:13:25.247 [i]: Playback [91b99dbe]: playing at 47.952 [23.976 *2/1] 
21:13:39.714 [i]: Playback [91b99dbe]: source frame rate changed to 8.181
21:13:39.715 [i]: Playback [91b99dbe]: 2 acceptible profiles, best is 'Anime_SubsPlease' [100]
21:13:39.717 [i]: Playback [91b99dbe]: keep previously selected profile 'RIFE AI engine'
21:15:40.833 [i]: AVSF: player closed
21:15:40.836 [i]: Playback [91b99dbe]: disabled while video is stopped
21:15:40.836 [i]: Playback [91b99dbe]: deleted

ERIC

Well I have not gotten very far as you can see I get some script failure using PyTorch based RIFE

As for specs: I'm trying on a AMD 3900x,  64GB of Memory and a 3080 TI card, so I should have the horse power to run RIFE if anything can.

I did try that video quickly using Vulkan; I had no issue with that GPU load is around 20%, will try to get some better data when I have some real  time to play more with RIFE.  Fully agree real-time SVP with RIFE would be a game changer, which is why I'm giving it a shot.

ERIC

Great! With this graphics card theoretically it should be possible to interpolate 720p video in real time using SVP with RIFE filter for VapourSynth (PyTorch version) without any problem!

If we can find a solution for script failure using PyTorch based RIFE, this smaller 240p 15FPS file can be used to test x4 real-time interpolation with RIFE!

That being said, I hope you don't give up and keep actively contributing to this thread until a solution can be found. Since dlr5668 was successful: https://www.svp-team.com/forum/viewtopi … 315#p79315 there must be some way to make it work on the 3080 TI too!

Hi guys, glad I found there are people working on real-time playback with RIFE. I've tried this method myself, hope I'm adding more valuable data to your development. Throwing my specs first：

CPU: Ryzen 7 5800X
GPU: NVIDIA GeForce RTX 3090 24GB
Memory: DDR4 4x16GB (64GB)
System: Windows 11

After some troubleshooting I got my VLC player to work with VS RIFE.
Straight to conclusion: it works perfectly in 720p, but laggy and complicated in 1080p.

Best result's configs I used:
Math precision: FP16
GPU threads: 6

Main take away here is that my GPU was poorly utilized.
In 720p, GPU utilization was about 28%. Using scale=1.0 (i.e. GPU thread=2) it goes up to 48%.
In 1080p GPU utilization was very unstable, when output is smooth it stays at around 60%, but drops to single digits quickly, and frames were droppepd. The result is a very jaggy GPU graph and laggy playback. I feel there are lots of space for optimization, or is my system not configured correctly? Anyway, in terms of raw performance I believe it's achievable to have a smooth playback in 1080p.

Another thing to note is that it ran very stable, I could do everything with the playback, like pause and skip forward/backward. Switching the profile was only a short freeze of 1s.

All video files I tried were 24fps anime videos. To my surprise the output image was not satisfactory to me, artifacts were terrible on texts, and interpolation did not seem much better than default SVP method.

Regarding to script errors:

I got the same error. I looked at the log and figured it was reading an empty file in vs-rife library, therefore the EOF error. Please bear with me that I don't know much about the technologies used here, it seems pytourch is reading this .pkl file below for AI inference perhaps?
SVP 4\mpv64\Lib\site-packages\vsrife\model38\flownet.pkl

I found vs-rife project is providing these files separately from the source code, so I downloaded them and put them into the Lib directory, then everything worked:
https://github.com/HolyWu/vs-rife/releases/tag/model38
Just download the source code zip and overwrite correponding folders and files.
I bet there is a better way to install them with pip, but it's not my profession here, guess Chainik could provide the command.

Chainik, is there a way to get the rife beta for svp on linux using the add programs and features tool or is this only for paid versions of SVP?

Yes, HolyWu has made changes with version 1.3.0.
The problem and solution is described here: https://github.com/HolyWu/vs-rife/issues/6
And here the recently added 3 lines of code in the file __init__.py : https://github.com/HolyWu/vs-rife/commi … 86856aabc0

Can you please update the files for SVP so that the models download automatically?

Sure, I'm happy to test this out and follow this thread.

For the troubleshooting it was really just the script error, nothing else.

Not sure how to configure vulkan vs d3d11 in VLC, I don't have the config file mentioned in previous posts (SVP 4\mpv64\mpv conf). I suppose it's TTA on or off.
Otherwise I've tried all the parameters you mentioned. Some notes:
I edited base.py that fixed scale to 1.0 regardless of GPU threads.
Following parameters did not affect the results at all, I've tried all combinations with both 720p and 240p videos:
GPU threads
Screen refresh rate. I tried 50Hz and 60Hz
VLC windowed and fullscreen mode
Black bar detection. Switching to this tab causes a 2s freeze, but clicking any options in this tab did not have any effects to the video.

Other values:
RIFE model = 3.8
Scale = 1.0
Math precision = FP16

Following are results:

With the sample 720p file:
TTA on: GPU stables at round 46%, playback is smooth at 50Hz. Switching to this profile from automatic has 3s freeze. Resuming pause will cause a 1s frame drop + freeze. Other actions are fluid without freeze.
TTA off: Lags every second, about half of the frames are dropped, GPU jumps between 1% to 100% in a very consistent manner.

Sample 240p file:
TTA on: GPU stables at around 8%, playback is smooth at 30Hz.
TTA off: GPU stables at around 13%, playback is smooth at 30Hz.

Interestingly when TTA is on, FP32 never worked, it freezes video but playback continues with audio, GPU stays idle.

Hope above data gives you a clear idea how it performed.

I don't use Linux myself, nor am I familiar with it, but I agree with your request. The more testers the better.

Thank you very much for the testing I requested and thank you very much for your willingness to continue to help on this thread.

I have read your post and the test results very carefully. I will come back to certain aspects of these results, but for now I am bothered by a few points:

1.

Did you make this change only before the tests I asked you to do, or before when you wrote in the previous post:

If I understand, by setting "GPU threads: 6" early on you were actually setting scale=0.5, according to this post: https://www.svp-team.com/forum/viewtopi … 183#p79183
which obviously doesn't do the quality any good: https://www.svp-team.com/forum/viewtopi … 250#p79250

2.

Can you share a fragment of the above code that you modified in base.py ?

3.

Request for a short test: to see if changing the GPU threads parameter changes anything in performance after the change in base.py you made.

Test A:

Test-Time Augmentation: Enabled [sets RIFE filter for VapourSynth (PyTorch)]
AI model: Generic [with above PyTorch version it doesn't matter, because it is set in the 2nd version of base.py by Chainik permanently to model 3.8]
Math precision: Half (FP16)
GPU threads: 2
Test file:
original demo video from the creator of RIFE at: https://github.com/hzwer/arXiv2020-RIFE
720p (1280x720), 25FPS, 53 s 680 ms, 4:2:0 YUV, 8 bits
direct link: https://drive.google.com/file/d/1i3xlKb … sp=sharing

Test B:

Same, except for one parameter:
GPU threads: 6

Please present the performance results as Chainik did in this post: https://www.svp-team.com/forum/viewtopi … 465#p78465

I think such a scheme with 3 parameters will be the best to present these and future results:

Test A
GPU: XX% (XX% Compute_1)
VRAM: X.X GB
SVP index: X.X

Of course, this could be "Compute_1" "Compute_0", "Cuda" or another virtual node, loaded by RIFE.This most accurately shows the actual utilization of GPU computing power by RIFE.

Hi Eric, could you share what parameters you used? It appears exactly the same behavior when I use FP32 with PyTorch. Please try adjust the parameters, you may find a working set.

About virtual nodes:

NVIDIA CUDA TOOLKIT 11.0.3 https://docplayer.net/192620757-Nvidia- … olkit.html

In Windows Task Manager there should be 4 graphs and the content of each graph should be changeable from the drop down list:

In the quote below, Chainik has shown the load on his NVIDIA GeForce RTX 2060 Mobile graphics card with real-time interpolation using RIFE filter for VapourSynth (ncnn).

The first result shows how the graphics card handles the video file without issue (SVP index: 1.0) with a real-time load of 65% indicated by Compute_1.

The second result shows how the graphics card can't keep up with real-time interpolation (SVP index: 0.76) as indicated by Compute_1's full 100% saturation, despite the total load surprisingly dropping to single digits.

Notice how this is consistent with what you wrote earlier:

Of course with you NVIDIA GeForce RTX 3090 testing the RIFE filter for VapourSynth (PyTorch) it doesn't have to be Compute_1 at all, it could be a completely different node, or a combination of several. Likewise, changing FP32 to FP16 can also result in load on other nodes. No one has tested or described this. You may be the first and it may interest many people. Me for sure Especially since at the moment there is no more powerful graphics card than RTX 3090 for a mere mortal.

lwk7454, I think now it would be worth to check again what is the actual GPU load (most loaded nodes) with the already tested reference 720p file at FP16 and if it reaches 100% nodes saturation for FP32.

Please check also 1080p file, which was already tested earlier on this thread if 100% nodes saturation is reached with it.

source.mkv
Video: 1080p (1920x1080), 23.976 FPS, 35 s 994 ms, 4:2:0 YUV, 8 bits
https://www.playbook.com/s/vadash/wcVXb … Ky5jaQMkLs

If you have a link to any other interesting test demo for interpolation with typical 720p or 1080p, 23.976 FPS, 8 bits that anyone today and in the future can download for comparison testing then let us know.

Fixed test parameters:

Test-Time Augmentation: Enabled [sets RIFE filter for VapourSynth (PyTorch)]
real time playback with x2 interpolation
RIFE model: 3.8
scale=1.0

Variable test parameters:

Math precision: FP16 and FP32
Video files: 720p and 1080p

Test results:
% GPU utilisation - most loaded GPU nodes

The results can be compared to other existing graphics cards, future graphics cards and future better and more efficient RIFE models!