• Thanks: 230
  • Thanks for the post: 1

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

UHD wrote:

I cannot believe it!

1920x1080 23.976 x2 and 54,0fps with RIFE!!!

So, NVIDIA GeForce RTX 3070 Ti is capable of producing enough frames per second in 1080p files today to think about x2 real-time interpolation!!!

Please post more, you are doing a great job!

Now, the most important question: do you have the same problem as Chainik on the Turing microarchitecture when attempting real-time interpolation?

Chainik wrote:

mpv just hangs after switching to RIFE profile in a real time
no idea why

Thank you very much!!!


How can I do that ? I tried

https://i.imgur.com/ss33OUG.png

but it uses old profile

  • Thanks: 1622
  • Thanks for the post: 1

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

> How can I do that ?

switch via profiles menu in runtime

  • Thanks: 230
  • Thanks for the post: 1

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

smooth = RIFE(input_m,model_ver=3.8,scale=0.5,device_index=rife_gpu,fp16=False)

54 fps

smooth = RIFE(input_m,model_ver=3.8,scale=0.5,device_index=rife_gpu,fp16=True)

44 fps

So using fp16 actually reduces fps ? wtf

  • Thanks: 230
  • Thanks for the post: 1

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

Chainik wrote:

> How can I do that ?

switch via profiles menu in runtime

Nice

https://i.imgur.com/uoW1Sez.png

  • Thanks: 1622
  • Thanks for the post: 1

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

just use this file - https://www.svp-team.com/forum/viewtopi … 183#p79183

"scale=0.5" is cheating big_smile

31 Reply by dlr5668 (edited by dlr5668 16-09-2021 15:12:59)

  • Thanks: 230
  • Thanks for the post: 1

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

Chainik wrote:

just use this file - https://www.svp-team.com/forum/viewtopi … 183#p79183

"scale=0.5" is cheating big_smile

scale 1.0 drops to 28 and fp16 has no effect right now (even drops 1 fps)

  • Thanks: 1622
  • Thanks for the post: 1

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

if I understand it correctly, scale=0.5 just downscales the whole frame before the "AI" magic, then upscales the result back --> you just get lower resolution intermediate frames
this is why it's "cheating"

  • Thanks: 230
  • Thanks for the post: 1

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

Chainik wrote:

if I understand it correctly, scale=0.5 just downscales the whole frame before the "AI" magic, then upscales the result back --> you just get lower resolution intermediate frames
this is why it's "cheating"

Yep after reading rife script I understand. Whats about fp16 ? Do we need to retrain models ?

  • Thanks: 1622
  • Thanks for the post: 1

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

> Whats about fp16 ? Do we need to retrain models ?

no idea...

so, 1080p actually plays at 27 fps only, not 54.
54 is with scale=0.5

35 Reply by dlr5668 (edited by dlr5668 16-09-2021 15:32:28)

  • Thanks: 230
  • Thanks for the post: 1

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

Chainik wrote:

> Whats about fp16 ? Do we need to retrain models ?

no idea...

so, 1080p actually plays at 27 fps only, not 54.
54 is with scale=0.5

At least my mpv is not crashing big_smile rewind takes 3-4 sec though

36 Reply by UHD

  • UHD
  • Member
  • Offline
  • Thanks: 39

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

Chainik wrote:

if I understand it correctly, scale=0.5 just downscales the whole frame before the "AI" magic, then upscales the result back --> you just get lower resolution intermediate frames
this is why it's "cheating"

your video has very high resolution such as 4K, we recommend set --scale=0.5 (default 1.0). If you generate disordered pattern on your videos, try set --scale=2.0. This parameter control the process resolution for optical flow model.

https://github.com/hzwer/arXiv2020-RIFE … DME.md#L60

What then does scale=2 and scale=4 do?

  • Thanks: 1622
  • Thanks for the post: 1

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

> What then does scale=2

the opposite? upscale before, downscale after

38 Reply by UHD

  • UHD
  • Member
  • Offline
  • Thanks: 39

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

Chainik wrote:

> What then does scale=2

the opposite? upscale before, downscale after

So, scale=1 will probably be the optimal solution. Somehow I don't believe that upscaling will bring any significant improvement. On the other hand, downscaling is not only cheating, but also a significant reduction in quality.

  • Thanks: 230
  • Thanks for the post: 1

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

UHD wrote:
Chainik wrote:

> What then does scale=2

the opposite? upscale before, downscale after

So, scale=1 will probably be the optimal solution. Somehow I don't believe that upscaling will bring any significant improvement. On the other hand, downscaling is not only cheating, but also a significant reduction in quality.

Yep lets wait for fp16 tweaks. Even x2 would allow us to watch in real time and author claims x4

40 Reply by UHD (edited by UHD 16-09-2021 16:30:27)

  • UHD
  • Member
  • Offline
  • Thanks: 39

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

dlr5668 wrote:

At least my mpv is not crashing big_smile

Great!!! This is the message I was waiting for!


Could you please check such configuration:

fps number encoding with 720p 23.976, scale=1, FP16 and FP32, RIFE 3.8 with SVP

and then:

real time playback x2: 720p 23.976, scale=1, FP16 and FP32, RIFE 3.8 with SVP with MPV

41 Reply by Fadexz (edited by Fadexz 14-12-2021 14:07:33)

  • Thanks: 3
  • Thanks for the post: 1

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

Chainik wrote:

mpv just hangs after switching to RIFE profile in a real time
no idea why

transcoding seems to work

Only when not in windowed mode for some reason, maybe something to do with the sizing change

Edit: Probably the GPU API used aka Vulkan

42 Reply by Fadexz (edited by Fadexz 14-12-2021 14:08:12)

  • Thanks: 3
  • Thanks for the post: 1

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

UHD wrote:

Thank you very much!

I was just writing the 2nd point of my essay, but it looks like it's not necessary. OK, maybe I'll still finish it and paste it in here sometime. It is not important now!

Chainik, please write if you were able to interpolate in real time a higher resolution file using the 3.8 model than last time?

Chainik wrote:

GPU; VRAM; CPU utilization; SVP index

424*240 - 23% (65% compute_1); 0.9 GB; zero; 1.0
640*360 - 9% (100% compute_1); 0.9 GB; zero; 0.76

https://www.svp-team.com/forum/viewtopi … 96&p=2


3.8 model is the fastest!


Based on the evaluation of dozens of videos, the v3.8 model has achieved an acceleration effect of more than 2X while surpassing the effect of the RIFEv2.4 model.

https://github.com/hzwer/arXiv2020-RIFE/issues/176

Most say RIFE 3.1 is the better for speed, 2.3 is the best quality though generally. You compared the RIFE 2 models to RIFE 3 models not RIFE 3.1 to RIFE 3.8. But whatever is fastest would be best for this.

43 Reply by UHD (edited by UHD 20-09-2021 20:37:03)

  • UHD
  • Member
  • Offline
  • Thanks: 39

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

Fadexz wrote:

Most say RIFE 3.1 is the better, 2.3 is the best though generally.

This is what they say on the FrameRateConverter thread: https://forum.doom9.net/showthread.php? … mp;page=25

They extensively test RIFE and work on its implementation into FrameRateConverter. Their developer is actively communicating with HolyWu https://github.com/HomeOfVapourSynthEvo … n/issues/7  so there is a chance VapourSynth RIFE filters will be developed, which SVP could also benefit from.

Unfortunately FrameRateConverter like others that have recently implemented AI RIFE focus on interpolation with reencode. In this case, time is of no importance, so they can use any VapourSynth RIFE filter, for convenience even the one using ncnn with Vulcan.

SVP is famous for its real-time interpolation. Here, any refinement that affects the interpolation time matters. Therefore only the VapourSynth RIFE filter with PyTorch and CUDA https://github.com/HolyWu/vs-rife is currently fast enough to interpolate in real time.

This is also why I quoted the creator of RIFE himself, who claims that the 3.8 model is 2x faster than the 2.4. This is a huge difference, for someone who wants to interpolate in real time.

However, I wouldn't get too attached to the 3.8 model and dwell too much on the differences to older models, because it's almost certain that we're in for a great future with AI RIFE:

To-do List

Multi-frame input of the model

Frame interpolation at any time location

Eliminate artifacts as much as possible

Make the model applicable under any resolution input

Provide models with lower calculation consumption

https://github.com/hzwer/Practical-RIFE

44 Reply by UHD (edited by UHD 20-09-2021 20:59:17)

  • UHD
  • Member
  • Offline
  • Thanks: 39

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

Fadexz wrote:
Chainik wrote:

mpv just hangs after switching to RIFE profile in a real time
no idea why

transcoding seems to work

Only when not in windowed mode for some reason, maybe something to do with the sizing change

Could you please elaborate a bit on this topic?
Did you manage to interpolate a video in real time according to these descriptions:
https://www.svp-team.com/forum/viewtopi … 023#p79023
https://www.svp-team.com/forum/viewtopi … 183#p79183 

Which scale in RIFE? What graphics card? What file resolution? Did it not crash?

45 Reply by UHD

  • UHD
  • Member
  • Offline
  • Thanks: 39

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

Good news!

On 20 September poisondeathray wrote on the FrameRateConverter thread:

The other issue is model availability - I'm finding 2.3 and 2.4 generally better than the 3.x versions . But we can probably ask HolyWu to make other models available for the Vapoursynth Pytorch version

https://forum.doom9.net/showthread.php? … mp;page=26

...and immediately HolyWu has prepared a new 1.3.0 version of the RIFE filter for VapourSynth based on PyTorch: https://github.com/HolyWu/vs-rife/releases/tag/v1.3.0

Please, if anyone has a suitable graphics card capable of working with this filter, try it in collaboration with SVP and the RIFE filter for VapourSynth based on PyTorch for real-time interpolation and please share your results on this thread.

As long as HolyWu is actively developing this filter, we can ask him to add or change something in the code of this filter.

  • Thanks: 1622
  • Thanks for the post: 1

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

I thought you want _faster_, not _better_...

47 Reply by UHD (edited by UHD 25-09-2021 14:16:15)

  • UHD
  • Member
  • Offline
  • Thanks: 39

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

Thanks for asking what I want. I think the answer to that question will lighten the context of my posts a bit.

I want to:

1. Take advantage of the fact that hzwer is actively developing RIFE AI interpolation and actively communicating with the community: https://github.com/hzwer/arXiv2020-RIFE

2. Take advantage of the fact that HolyWu is actively developing the RIFE filter for VapourSynth based on PyTorch and is actively communicating with the community: https://github.com/HolyWu/vs-rife

Of course, I would very much like to see the RIFE filter for VapourSynth based on ncnn developed as well: https://github.com/HomeOfVapourSynthEvo … cnn-Vulkan Here the problem arises that HolyWu is dependent on the nihui project for this filter: https://github.com/nihui/rife-ncnn-vulkan

And this is where the problem lies... nihui has not been developing this project for 4 months and does not respond to users problems: https://github.com/nihui/rife-ncnn-vulkan/issues Of course we should be grateful that he has made the code available for us all to use. We must remember that he makes his work available for free and that any support depends on his good will and the time he has available.

However, this does not change the fact that we do not know if there will ever be any update to the code created by nihui. The situation is similar with NVIDIA's GPU implementation of CUDA inference support for the well-known Tencent NCNN inference engine: https://github.com/atanmarko/ncnn-with-cuda further development of which is essential to be able to efficiently interpolate in real time using Vulkan.

If nothing changes, the long-awaited RIFE interpolation model 4 prepared by hzwer will only be usable in SVP thanks to HolyWu and the actively developed the RIFE filter for VapourSynth based on PyTorch: https://github.com/HolyWu/vs-rife

This is exactly what I want: I ask everyone including you Chainik to test the RIFE filter for VapourSynth based on PyTorch with SVP for real-time interpolation and take advantage of the fact that all possible problems and suggestions for improvements can be reported to HolyWu and hzwer and there is a good chance that they will be addressed for the benefit of all of us who want to have frame interpolation without artifacts.

  • Thanks: 230
  • Thanks for the post: 1

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

vsrife 1.3 broke conversion

06:47:01.966: ===== Starting mpv ======
06:47:01.966: Command line: C:\Program Files (x86)\SVP 4\mpv64\mpv.exe E:/media/shows/[LowPower-Raws] Boku no Hero Academia S4 (BD 1080P x265 Ma10p FLAC)/[LowPower-Raws] Boku no Hero Academia S4 - 08 (BD 1080P x265 Ma10p FLAC).mkv --o=E:/media/shows/[LowPower-Raws] Boku no Hero Academia S4 (BD 1080P x265 Ma10p FLAC)/[LowPower-Raws] Boku no Hero Academia S4 - 08 (BD 1080P x265 Ma10p FLAC).SVP.temporary.mkv --no-audio --no-sub --no-sub-auto --input-ipc-server=mpvencodepipe --input-media-keys=no --no-msg-color --vf=vapoursynth:[C:\Users\vadash\AppData\Roaming\SVP4\scripts\ffff.py]:4:24 --of=matroska --ovc=h264_nvenc --ovcopts=b=35010130,preset=slow,profile=high,rc=vbr,maxrate=45513169,bufsize=70020260,time_base=1000/47952,threads=24
06:47:02.056: (+) Video --vid=1 (*) (hevc 1920x1080 23.976fps)
06:47:02.056: Audio --aid=1 --alang=jpn (*) (flac 2ch 48000Hz)
06:47:03.560: [vapoursynth] Script evaluation failed:
06:47:03.560: [vapoursynth] Python exception: Ran out of input
06:47:03.560: [vapoursynth]
06:47:03.560: [vapoursynth] Traceback (most recent call last):
06:47:03.560: [vapoursynth] File "src\cython\vapoursynth.pyx", line 2244, in vapoursynth.vpy_evaluateScript
06:47:03.560: [vapoursynth] File "src\cython\vapoursynth.pyx", line 2245, in vapoursynth.vpy_evaluateScript
06:47:03.560: [vapoursynth] File "C:\Users\vadash\AppData\Roaming\SVP4\scripts\ffff.py", line 70, in <module>
06:47:03.560: [vapoursynth] smooth = interpolate(clip)
06:47:03.560: [vapoursynth] File "C:\Users\vadash\AppData\Roaming\SVP4\scripts\ffff.py", line 51, in interpolate
06:47:03.560: [vapoursynth] smooth = RIFE(input_m,model_ver=3.8)
06:47:03.560: [vapoursynth] File "C:\Program Files (x86)\SVP 4\mpv64\lib\site-packages\vsrife\__init__.py", line 77, in RIFE
06:47:03.560: [vapoursynth] model.load_model(os.path.join(os.path.dirname(__file__), model_dir), -1)
06:47:03.560: [vapoursynth] File "C:\Program Files (x86)\SVP 4\mpv64\lib\site-packages\vsrife\model38\RIFE_HDv3.py", line 40, in load_model
06:47:03.560: [vapoursynth] self.flownet.load_state_dict(convert(torch.load('{}/flownet.pkl'.format(path))), False)
06:47:03.560: [vapoursynth] File "C:\Program Files (x86)\SVP 4\mpv64\lib\site-packages\torch\serialization.py", line 608, in load
06:47:03.560: [vapoursynth] return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
06:47:03.560: [vapoursynth] File "C:\Program Files (x86)\SVP 4\mpv64\lib\site-packages\torch\serialization.py", line 777, in _legacy_load
06:47:03.560: [vapoursynth] magic_number = pickle_module.load(f, **pickle_load_args)
06:47:03.560: [vapoursynth] EOFError: Ran out of input
06:47:03.560: [vapoursynth]
06:47:03.652: (!!!) Intermediate file may be broken: E:\media\shows\[LowPower-Raws] Boku no Hero Academia S4 (BD 1080P x265 Ma10p FLAC)\[LowPower-Raws] Boku no Hero Academia S4 - 08 (BD 1080P x265 Ma10p FLAC).SVP.temporary.mkv
06:47:03.652: ===== mpv exited with code 62097 =====

49 Reply by UHD

  • UHD
  • Member
  • Offline
  • Thanks: 39

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

Hi dlr5668, thanks for taking a moment to test vs-rife v1.3.0.

If I understand correctly the file you are trying to interpolate is:
[LowPower-Raws] Boku no Hero Academia S4 - 08 (BD 1080P x265 Ma10p FLAC).mkv

RIFE does not currently support 10-bit colour depth, hence perhaps the problems with the Main 10 profile. Have you tried interpolating 8-bit files?


If you still have some time, could you check if it is possible to smoothly interpolate in real time on your NVIDIA GeForce RTX 3070 Ti (GA104) @ 1830 MHz graphics card with the following configuration:

real time playback, x2 interpolation
720p 23.976 video file
SVP with mpv with vs-rife(PyTorch)
RIFE model: 3.8
scale=1
FP32 or(and) FP16

I would love to test it myself, but I don't have a suitable graphics card yet. If I knew what video resolution could be interpolated with vs-rife in real time it would make it more easly for me to choose a graphics card to buy.

  • Thanks: 1622
  • Thanks for the post: 1

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

> it would make it more easly for me to choose a graphics card to buy

buy 3090!