You are not logged in. Please login or register.
SmoothVideo Project → Using SVP → RIFE filter for VapourSynth - AI interpolation in SVP?
RIFE in SVP: https://www.svp-team.com/wiki/RIFE_AI_interpolation
-------------------------------
RIFE filter for VapourSynth: https://github.com/HomeOfVapourSynthEvo … cnn-Vulkan
SVP with real-time RIFE support could provide better results and less artifacts than the motion interpolation algorithms in the best TVs!
Question to developers: is it now possible to try to combine RIFE with SVP?
If anyone still doesn't know what RIFE is here are links to the source:
RIFE: https://github.com/hzwer/Arxiv2020-RIFE
RIFE ncnn Vulkan: https://github.com/nihui/rife-ncnn-vulkan
Also, you can check for yourself the motion interpolation effects of several AI-based methods, including RIFE, in a Windows environment using:
Flowframes: https://nmkd.itch.io/flowframes
https://github.com/n00mkrad/flowframes
how fast is it ?
laptop rtx 2060 --> 704*528 @30->60 --> 24 fps transcoding speed
1080p @24->48 --> 4 fps
(fp32 = false, tta = false)
is it supposed to use all the GPU? 
because I only see up to 10% GPU load regardless of gpu_thread value
---
flowframes shows the same speed and gpu usage so lets assume this is "normal"
bottom line - I think we could integrate this for the SVPcode usage
laptop rtx 2060 --> 704*528 @30->60 --> 24 fps transcoding speed
I think more powerful PC dedicated graphics cards, especially the latest GeForce RTX 30 should perform better. However, even this setup gives hope that at least DVD-quality material can be interpolated in real time. At least for now, as the RIFE algorithms are constantly being optimised.
NVIDIA GeForce RTX 2060 Mobile - relative performance:
https://www.techpowerup.com/gpu-specs/g … bile.c3348
is it supposed to use all the GPU?
No idea what this is all about. However, if 10% GPU usage produces these results then there is hope for an increase in fps as much as 10X!!! I have no idea about programming, but I think a programmer with a programmer will very quickly be able to come to ways to use the full power of the GPU.
It may sound strange in the context of what I am writing about and striving for, but at the moment I don't even have a way to check how Flowframes works. I was supposed to build a PC based on the AMD Ryzen Threadripper 3970X in early 2020, but had to postpone plans.... Now I know that a good GPU will still come in handy. Well... I'm looking forward, especially to SVP on this giant CPU and compare the effects with RIFE on a good GPU. For the moment I'm relying on the opinions of others, and they are extremely enthusiastic about RIFE.
bottom line - I think we could integrate this for the SVPcode usage
Is it possible to implement RIFE so that we can watch motion interpolated videos in real time without transcoding? I care about this the most, and probably not only me. We can transcode movies using Flowframes, however, every transcoding means loss of some quality, unless we use -x265-params-lossless=1, but then we need huge HDD space;-)
Yes, I know. For starters, materials in DVD resolution and below are only with a powerful GPU.
„Q: What's the difference between RIFE CUDA and RIFE NCNN? Which one should I use?
A: The results should be identical, however, RIFE-NCNN also runs on AMD cards, CUDA only on Nvidia. If you have an Nvidia card, use CUDA as it's faster.”
https://github.com/n00mkrad/flowframes
For now we have an attempt to create a universal version of RIFE filter for VapourSynth based RIFE ncnn Vulkan. If in the future there will be a RIFE filter for VapourSynth dedicated for CUDA then there is a chance for some performance increase. I also wonder if it will be possible to increase performance with Tensor Cores?
yeah, CUDA version is like 3 times faster than ncnn (in flowframes) and loads my GPU up to 50%
Great!!!
Does that mean: laptop rtx 2060 --> 704*528 @30->60 --> 72 fps transcoding speed?
That would already make sense!
On the other hand RIFE filter for VapourSynth without CUDA optimization might not be enough.
Now the question is whether to watch motion interpolated videos in real time without transcoding using AI RIFE interpolation and SVP...
...RIFE filter for VapourSynth is necessary?
Or would RIFE alone be sufficient: https://github.com/hzwer/Arxiv2020-RIFE
> Great!!!
not great, because we definitely won't use 2.5 GB of dependencies required to run the "original" RIFE
there's a ncnn/CUDA (https://github.com/atanmarko/ncnn-with- … ocator.cpp) but it was last updated in January... so I dunno...
If I understand correctly RIFE filter for VapourSynth does not need these dependencies?
more like "RIFE over ncnn does not need these dependencies"
and the "official" ncnn only supports Vulkan as of now
That is, it would be best for the author of the RIFE filter for VapourSynth to implement ncnn-with-cuda in his code. That 3x performance gain is worth it. It gets a bit complicated.
Still such a question arises: Flowframes performs 2 operations:
-interpolation of motion using, for example, RIFE algorithms
-transcoding to the resulting video file or image files (PNG).
I am curious how much time could be saved if this second operation was omitted, i.e. what I am aiming at: sending the finished result of motion interpolation directly to the monitor, without transcoding and saving on HDD/SSD?
> how much time could be saved if this second operation was omitted
zero 
encoding via NVENC costs nothing...
even if you want good old software encoder then it will be like 10 times faster than "rife over cuda"
So we have 3 options:
1. Implement the "original" RIFE in SVP - disadvantage: 2.5 GB of dependencies
2. Implement the RIFE filter for VapourSynth in SVP - disadvantage: worse performance
3. Ask the developer of the RIFE filter for VapourSynth to implement the code https://github.com/atanmarko/ncnn-with- … ocator.cpp
Alternatively there is a 4th option...
Chainik maybe you could look into whether you would be able to somehow combine the codes from point 3 above to speed up the RIFE filter for VapourSynth code to be able to watch motion interpolated videos in real time without transcoding using AI RIFE interpolation and SVP? Just as an experiment. You are a motion interpolation enthusiast, just like all of us on this forum. The difference, however, is that you have a knowledge of how it all works that we do not.
> Ask the developer of the RIFE filter for VapourSynth to implement the code
actually there no changes needed in the "RIFE filter for VapourSynth" itslef
the problem is "ncnn with CUDA" is not completed yet, some functions necessary for RIFE are not implemented... and the last commit was on Jan 18.
> Implement the "original" RIFE in SVP - disadvantage: 2.5 GB of dependencies
... plus very long (re-)start time. a video player will hang for 5-10 secs on each seek, for example.
actually there no changes needed in the "RIFE filter for VapourSynth" itslef
the problem is "ncnn with CUDA" is not completed yet, some functions necessary for RIFE are not implemented... and the last commit was on Jan 18.
This is probably even better. NCNN has a much broader application than motion interpolation, so there's a good chance that someone will want to improve NCNN performance by optimizing it for CUDA. It's not clear if the lack of updates are some temporary issues, lack of times, or something out of the blue? On the other hand, it's very interesting how so many new tools dedicated to AI and using AI are being developed in just the last year of time!
... plus very long (re-)start time. a video player will hang for 5-10 secs on each seek, for example.
Are you using a fast NVMe SSD? Maybe it takes that long to load those 2.5 GB of dependencies each time? Have you checked what is the bottleneck in that 5-10 seconds: GPU, CPU, HDD/SDD?
Would it be possible to check if the same amount of time would be needed if you put all data with all dependencies in RAM Disk?
...and thank you very much for that:
https://github.com/nihui/rife-ncnn-vulkan/issues/22
I keep my fingers crossed!
I cannot believe it! You did it!
Tell me, did you somehow manage to get the performance of the CUDA?
Is it possible to get smooth x2 playback with RIFE on your GPU on some resolution? For example 480p? 720p?
> did you somehow manage to get the performance of the CUDA?
nope
> Is it possible to get smooth x2 playback with RIFE on your GPU on some resolution? For example 480p? 720p?
not a chance
---
240p works great, 360p is too much
240p works great, 360p is too much
That is what I was about to ask. Every GPU should have a resolution threshold at which it can handle and one where it can't.
Here is what we know:
NVIDIA GeForce RTX 2060 Mobile
240p works great
360p is too much
Can you share what your GPU, VRAM, CPU utilization was at 240p and 360p?
SmoothVideo Project → Using SVP → RIFE filter for VapourSynth - AI interpolation in SVP?
Powered by PunBB, supported by Informer Technologies, Inc.