New RIFE filter - 3x faster AI interpolation possible in SVP!!! (Page 5 of 85)

I'm back.

Today I'd like to propose tests that are related to two problems that arose when testing frame interpolation during real-time playback using the RIFE filter for VapourSynth (PyTorch):

1.

2.

My proposition is to run the same tests again as last time, but this time not during real-time playback but during re-encoding of video files.

The re-encoding speed expressed in frames per second will give us a very precise measure of what the potential is for the fastest graphics card on the market today: NVIDIA GeForce RTX 3090. A value for a 1080p file above 48FPS would give us the answer that the current RIFE 3.8 model is too slow for real-time interpolation without downscaling (scale=1.0).

Comparing the results of a 720p and 1080p file will give us an answer as to whether the load increases linearly with the number of pixels and give us benchmarks for comparison with other graphics cards.

Tests for FP32 precision can confirm to us where the problem lies. I hope everything will work and we will see how the performance drop in relation to FP16.

Percentage load tests on the graphics card will show us if it is possible to interpolate at full power.

And here are the details of the proposed 4 tests:

Fixed test parameters:

Test-Time Augmentation: Enabled [sets RIFE filter for VapourSynth (PyTorch)]
re-encoding with x2 interpolation
RIFE model: 3.8
scale=1.0

Variable test parameters:

Math precision: FP16 and FP32
Video files: 720p and 1080p

Test results:

re-encoding speed [FPS]
GPU utilisation [%]

Video files:

Test file 1:
original demo video from the creator of RIFE at: https://github.com/hzwer/arXiv2020-RIFE
720p (1280x720), 25FPS, 53 s 680 ms, 4:2:0 YUV, 8 bits
direct link: https://drive.google.com/file/d/1i3xlKb … sp=sharing

Test file 2:
video tested earlier on this thread by dlr5668
source.mkv
1080p (1920x1080), 23.976 FPS, 35 s 994 ms, 4:2:0 YUV, 8 bits
https://www.playbook.com/s/vadash/wcVXb … Ky5jaQMkLs

lwk7454, I would be very grateful to you if you could find some time for these 4 tests. I am very curious about the results.

Here are the tests done for RIFE interpolation in Flowframes. Last update done 9 months ago, so it's not the fastest 3.8, and we don't know if FP32 or FP16 was used. Graphics card drivers and Windows OS have also changed since then:

https://github.com/n00mkrad/flowframes/ … chmarks.md

Thank you! This is what it was all about! Very interesting results and a bit surprising.

The first thing that comes to mind after reviewing the results is how powerful the NVIDIA GeForce RTX 3090 is. Nearly 70FPS for HD video using an interpolation algorithm based on machine learning (artificial intelligence) and without downscaling (scale=1.0) is pretty impressive.

The second thing that comes to mind after reviewing the results is how much of the NVIDIA GeForce RTX 3090 graphics card's potential is still unfulfilled: 56-66% CUDA utilization and worse FP16 results compared to FP32.

I think a lot of people don't realize what potential there is in RIFE and don't even look at this thread. Yes I know that not everyone has a top of the line graphics card (I know because I don't have one yet either), but it is up to all of us now to make progress and practical use of algorithms based on machine learning for real time interpolation. Up to us, because firstly we can show that we need them. Secondly, because we can signal where there is still room for improvement in terms of efficiency and quality.

The third thing that comes to mind after reviewing the results is how big the differences are when it comes to real-time interpolation during playback versus during re-encoding: the percentage usage of CUDA power and the possibility of using FP32.

In upcoming posts later today and over the next few days, I will outline the areas that I think would be worth testing based on the results that lwk7454 presented and explain why.

So I've tried the free version of Flowframes. Seems its RIFE Pytorch implementation isn't any better than SVP, except GPU usage was a bit unstable, therefore I'm writing down approximated average usages here. Results:

720p, FP16
FPS: 61.32
Cuda: ~45%
720p, FP32
FPS: 63.28
Cuda: ~45%

1080p, FP16
FPS: 26.84
Cuda: ~50%

1080p, FP32
FPS: 27.02
Cuda: ~50%

I've tried Vulkan implementation too, but it only has RIFE 3.1 instead of 3.8:
720p, FP16
FPS: 36.79
Compute_1: 98%
Cuda: 0%
720p, FP32
FPS: 38.85
Compute_1: 98%
Cuda: 0%

Interesting to see Flowframes RIFE with Vulkan has much better performance than SVP.

2. Math precision: FP32 vs. FP16

All tests, the results of which were presented to us by lwk7454, clearly show that for the 3.8 model the performance of FP32 calculations is higher than that of FP16 calculations. This regularity is maintained both for interpolation with SVP+vsrife and for Flowframes.

A similar regularity for this model was observed by dlr5668 at scale=0.5:
https://www.svp-team.com/forum/viewtopi … 246#p79246

Now I think I know what Flowframes creator n00mkrad meant when he wrote on 21 August:

https://github.com/hzwer/arXiv2020-RIFE/issues/188

The question is: did FP16 work better with older models, and if so which ones?

cheesywheesy wrote:

https://github.com/hzwer/arXiv2020-RIFE/issues/188

This is even more interesting since the author of the above quote also uses an NVIDIA GeForce RTX 3090 graphics card:

https://github.com/hzwer/arXiv2020-RIFE/issues/188

Now I don't know if by writing about speed x4 the author meant the performance of the card itself or some older RIFE model.

After the following quote from hzwer, the creator of RIFE, I assumed that the 3.8 model will be the fastest:

https://github.com/hzwer/arXiv2020-RIFE/issues/176

It may be the fastest, but for FP32 precision. It would be interesting to see if older models were faster and by how much for FP16 precision. Especially since there are quite frequent opinions that older models gave better quality:
https://github.com/hzwer/Practical-RIFE/issues/5
https://github.com/hzwer/Practical-RIFE/issues/3
https://github.com/hzwer/arXiv2020-RIFE/issues/176

In vs-rife we have 6 models available:
1.8
2.3
2.4
3.1
3.5
3.8
https://github.com/HolyWu/vs-rife/tree/master/vsrife

The last one we have tested. I know that it would take some time to test the other 5 models for FP16 and FP32, because it is as many as 10 tests, but I think it would be interesting to know at exactly what point there was a performance degradation for FP16 and if there is any model that gives higher performance with FP16 than 3.8 with FP32. The results of such tests I obviously intend to pass on to the RIFE developer with a question about the possibility of future optimisation for model 4.

If that's too much to test then I'm curious about performance in this order of importance: 2.4; 3.1; 2.3; 1.8; 3.5.

Fixed test parameters:

Test-Time Augmentation: Enabled [sets RIFE filter for VapourSynth (PyTorch)]
re-encoding with x2 interpolation
scale=1.0

Variable test parameters:

Math precision: FP16 and FP32
RIFE model: 2.4 optional: 3.1; 2.3; 1.8; 3.5

Test results:

re-encoding speed [FPS]
GPU utilisation [%].

Video file:

original demo video from the creator of RIFE at: https://github.com/hzwer/arXiv2020-RIFE
720p (1280x720), 25FPS, 53 s 680 ms, 4:2:0 YUV, 8 bits
direct link: https://drive.google.com/file/d/1i3xlKb … sp=sharing.

lwk7454, I would be very grateful if you could find some time to run these tests.

Many thanks for the tests and for commenting on them.

What can I add? It is a great pity that no one has done such tests before and shared the results publicly. I am very curious what the developer of RIFE will say about this. After all, he knows the RIFE code best and how it works. If anyone would like to perform such tests on another graphics card I highly encourage you to do it. In about a week I will post the results on the RIFE project website: https://github.com/hzwer/arXiv2020-RIFE/issues

Once again all results together:

FP32

FPS 63.28 - Cuda ~45% - v3.8
FPS 59.43 - Cuda ~50% - v3.1
FPS 58.22 - Cuda ~70% - v2.4
FPS 58.39 - Cuda ~70% - v2.3
FPS 55.98 - Cuda 87% - v1.8

FP16

FPS 61.32 - Cuda ~45% - v3.8
FPS 60.03 - Cuda ~40% - v3.1
FPS 57.06 - Cuda ~50% - v2.4
FPS 57.20 - Cuda ~55% - v2.3
FPS 58.87 - Cuda ~70% - v1.8

lwk7454, when it comes to re-encoding tests I think we already have the full picture regarding the potential of today's most efficient consumer graphics card with relation to all major RIFE models and both computational precisions. I hope you'll continue to follow this thread and the place where I intend to share your results with a wider group of people interested in RIFE along with its creator: https://github.com/hzwer/arXiv2020-RIFE/issues  Maybe someone will come up with some ideas on how to increase the utilization of the graphics card power? I'm especially counting on the creator of RIFE, who is very helpful, as you can see in this example: https://github.com/hzwer/arXiv2020-RIFE/issues/207  If you would like to post results and questions yourself, let us know. After all, it is thanks to your work that we know there is still a lot of potential in the NVIDIA GeForce RTX 3090 graphics card to use.

I have a question. Does anyone read this forum and know enough to be able to test some very innovative interpolation model based on machine learning (artificial intelligence)?

This is a special model that could be either revolutionary or still need a lot of work to make it useful to the average interpolation enthusiast.

It differs significantly from RIFE or other top models of traditional AI frame interpolation such as: SoftSplat, XVFI, FLAVR, ABME, QVI and EQVI.

I have high hopes for it but also a lot of concerns.

The model requires at least:

NVIDIA GPU
CUDA 9.0
CuDNN 7.6.5
Pytorch 1.1.0

Only one model and only for someone who has the REAL SKILLS to test it in practice and of course the willingness

This still relevant to you?
If so yeah I use Vulkan in mpv, it doesn't allow for any overlays and stuff in fullscreen because it takes control of the display or something like displays directly, I don't know much about it.
I just use whatever settings to make it run easiest as that's really all that matters, to make it actually run.
I just gave it a test and it doesn't crash now so must have already been fixed.

Cant install VS 57 for it

C:\Program Files (x86)\SVP 4\mpv64>python -m pip install VapourSynth==57
Collecting VapourSynth==57
  Using cached VapourSynth-57.zip (567 kB)
  Preparing metadata (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: 'C:\Program Files (x86)\SVP 4\mpv64\python.exe' -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\vadash\\AppData\\Local\\Temp\\pip-install-9t2tm3ve\\vapoursynth_e1cac919435143d8881edb08ccb11258\\setup.py'"'"'; __file__='"'"'C:\\Users\\vadash\\AppData\\Local\\Temp\\pip-install-9t2tm3ve\\vapoursynth_e1cac919435143d8881edb08ccb11258\\setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base 'C:\Users\vadash\AppData\Local\Temp\pip-pip-egg-info-pvu45zqm'
       cwd: C:\Users\vadash\AppData\Local\Temp\pip-install-9t2tm3ve\vapoursynth_e1cac919435143d8881edb08ccb11258\
  Complete output (14 lines):
  Traceback (most recent call last):
    File "C:\Users\vadash\AppData\Local\Temp\pip-install-9t2tm3ve\vapoursynth_e1cac919435143d8881edb08ccb11258\setup.py", line 64, in <module>
      dll_path = query(winreg.HKEY_LOCAL_MACHINE, REGISTRY_PATH, REGISTRY_KEY)
    File "C:\Users\vadash\AppData\Local\Temp\pip-install-9t2tm3ve\vapoursynth_e1cac919435143d8881edb08ccb11258\setup.py", line 38, in query
      reg_key = winreg.OpenKey(hkey, path, 0, winreg.KEY_READ)
  FileNotFoundError: [WinError 2] The system cannot find the file specified

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "C:\Users\vadash\AppData\Local\Temp\pip-install-9t2tm3ve\vapoursynth_e1cac919435143d8881edb08ccb11258\setup.py", line 67, in <module>
      raise OSError("Couldn't detect vapoursynth installation path")
  OSError: Couldn't detect vapoursynth installation path
  ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/f9/80/66051b33405d4ecbc6647021110c0f51e75f10195ad3a920f475a1cadcbc/VapourSynth-57.zip#sha256=da28cba8c453ede130cbe79b5954c73a3d948e13a75c2f3942767e082c4227d9 (from https://pypi.org/simple/vapoursynth/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
ERROR: Could not find a version that satisfies the requirement VapourSynth==57 (from versions: 39, 40, 41, 42, 43, 44, 45, 46, 47, 47.1, 47.2, 48, 49, 50, 51, 56, 57)
ERROR: No matching distribution found for VapourSynth==57

C:\Program Files (x86)\SVP 4\mpv64>

Yep vsrife installs now

    Uninstalling vsrife-1.2.0:
      Successfully uninstalled vsrife-1.2.0
Successfully installed VapourSynth-57 vsrife-2.0.0

Now time to edit svp script

00:54:18.798: [vapoursynth] File "C:\Users\vadash\AppData\Roaming\SVP4\scripts\ffff.py", line 5, in <module>
00:54:18.798: [vapoursynth] core = vs.get_core(threads=4)
00:54:18.798: [vapoursynth] AttributeError: module 'vapoursynth' has no attribute 'get_core'
00:54:18.798: [vapoursynth]
00:54:18.807: (!!!) Intermediate file may be broken: E:\media\shows\The Wheel of Time\Season 1\The.Wheel.Of.Time.S01E06.PROPER.1080p.WEB.h264-GOSSIP[rartv].SVP.temporary.mkv

Yes, information from people who have successfully run real-time interpolation using the RIFE filter for VapourSynth (PyTorch) is all the time most important to me, as this is how I will use the filter most often.

My main goal is to interpolate 1080p files in real time using SVP and the RIFE filter for VapourSynth (PyTorch), once I buy a sufficiently powerful graphics card. Therefore, in addition to real-time testing, I am also interested in the performance of each graphics card and looking for ways to optimise interpolation using the RIFE algorithm.

At the moment the priority is to implement the new 4.0 RIFE model to work with SVP. I have a request that you stay with us on this thread and help test this model.

I will come back to the topic of real-time interpolation and have some more questions for you. Thank you for your answer and also for not forgetting my questions.