Search options (Page 1 of 12)

Super!

Hopefully also the fastest RAM - will come in handy with RIFE in 4K

One more thing to make it not so beautiful: the number of input frames required for interpolation

2 frames RIFE
3 frames BiT
4 frames VIDUE
4 frames DeMFI

There may be complications when using scene detection and preparing the algorithm of what to interpolate and what not. This is of course something we will worry about in the future and I think it can be solved somehow. Using more than 2 frames obviously has advantages, such as when the motion accelerates. Using more input frames was also planned by the developer of RIFE: https://github.com/hzwer/Practical-RIFE#to-do-list

A little bonus to whet your appetite;)

zzh-tech previously created a similar database with real motion blur, but only for Video Deblurring. You can see the effects for the AI Video Deblurring model here:
https://github.com/sollynoay/MMP-RNN/tr … NN#results

Go to:
Deblurring results on BSD

There is a link to the file there: BSD_MMPRNN_test.zip (3.6G)

Inside are 3 versions of the video frames:

_input - this is a frame of video recorded with motion blur
_gt - this is a frame of video recorded with a second synchronised camera without motion blur
_mmprnn - this is the effect of the algorithm performing Video Deblurring on _input files

the first camera records: 15 fps with 16 ms motion blur
the second camera records:  15 fps with 2 ms motion blur.

This is an example of what the RIFE algorithm is missing.

Unfortunately, this method is relatively slow, at least compared to RIFE, see Table 1: https://arxiv.org/pdf/2211.11423.pdf

Tensor Cores optimisation will probably significantly speed up the inference. For now, however, we need to have a model that is most useful to us, then we will think about what to do next.

I don't think anything is going to replace RIFE in the near future in terms of performance speed, but for lower resolution video, for video encoding where the highest possible quality is needed we may be looking for something better, albeit slower.

If graphics cards double in power every 2 years, who knows if BiT won't replace RIFE for real-time interpolation in a while, even though it will be slower. After all, for 4K the bottleneck for RIFE is probably no longer the graphics card but RAM.

Then there is the issue of directing researchers to move towards Joint Video Deblurring and Frame Interpolation, which I want to do, through my project, which still needs a lot of work: https://github.com/AIVFI/Video-Frame-In … g-Rankings

What do we have and what more do we need?

We have a BiT++ model trained on the RBI database with real motion blur: https://drive.google.com/drive/folders/ … vnDo9C04HO

However, it turns out that the RBI database is too small and better results are obtained by the model trained on Adobe240 and then on RBI. This is indicated by the data in the paper: https://arxiv.org/abs/2211.11423 which I further confirmed with the author of the BiT model: https://github.com/zzh-tech/BiT/issues/4

Another point is that models trained to maximise PSNR and SSIM are not the optimal ones for practical purposes, as I mentioned earlier: https://www.svp-team.com/forum/viewtopi … 250#p82250

What we need is a model trained on perceptual loss that achieves the best result (lowest value) of LPIPS in the RBI with real motion blur tests. You can read why this is so important in the introduction to my rankings: https://github.com/AIVFI/Video-Frame-In … g-Rankings

In summary, we need a model:
- Joint Video Deblurring and Frame Interpolation
- trained on perceptual loss
- trained on Adobe240 and then on RBI

I am going to make a request to zzh-tech for such a model. I haven't had time to look into this, and I know that the way I make this request may determine whether we get such a model or not. So it is not such a simple matter.

I intended to write a request first, and then ask here on the forum for support of my request by all those who care about the best possible video frame interpolation.

I know that some people just want ready-made, working solutions in SVP and I remember well what indifference there was when I first wrote here about RIFE: https://www.svp-team.com/forum/viewtopic.php?id=6196

However, back then we were in a much better position, because we had both practical models and a ready-made RIFE filter for VapourSynth. In general, we were very lucky that the developer of RIFE was interested in building practical models. Unfortunately, most researchers who develop AI models mostly care about publishing and presenting the paper at a prestigious scientific conference and then getting citations. Practical application often does not matter at all. This is the sad truth.

Therefore, it is up to us to push for the development of models that will be most useful to us. It is for this purpose that I have started and want to update my repository on GitHub: https://github.com/AIVFI/Video-Frame-In … g-Rankings

Unfortunately, I am not a programmer and this repository is the pinnacle of my abilities. I want it to be visible on GitHub, because I believe that through it maybe some talented researchers will develop something that will be most useful to us and suitable for implementation in SVP. My repository is to show what is currently the real SOTA in the field of interpolation and what we need most.

Interestingly, already 2 of the 25 stars that my repository has received on GitHub are from the authors of Video Frame Interpolation models:
https://github.com/kinoud
https://github.com/danier97

It means that what I am doing makes sense. Life has taught me that if you want something you have to try to do everything you can to achieve it. Often it seems like we have no influence and yet there is always something that can be done. I am not a programmer, but I am trying here on this forum and on GitHub to do something that can maybe make me enjoy the best possible interpolation. I'm already happy with how far it's come with RIFE, but I want more and better because I know it's possible.

I will now try to prepare a request for a more practical BIT model and hope you will support my request on GiTHub. This may have an impact on whether we get such a model or not. 

I would like to prepare such a request later this weekend, but I know that writing about deadlines in my case is inappropriate to say the least. I've taken on extra job to earn money for NVIDIA GeForce RTX 4090, but apparently I've been a bit overwhelmed by it all time-wise and otherwise. However, I am not giving up.

Before I prepare a request, please write what you think about all the above.

What could be much better than RIFE for interpolating video frames?

Let me perhaps start with the basics. Why do we use RIFE and SVP?

Simple: a typical video has 24 frames per second, and with RIFE and SVP we can create intermediate frames and, for example, with x5 interpolation, have 120 frames per second and a smooth experience watching a movie or series.

The problem is that this 120 fps is far from what we would experience if we had the same movie natively recorded in 120 fps. My point is not that RIFE isn't perfect, because no algorithm can losslessly restore what was happening in front of the camera.

The point is that in typical 24 fps footage, artefacts in the form of motion blur have been artificially added to trick our brains into giving the impression of movement. As in photography, so in video, each frame has a specific exposure time. In photography, however, apart from a few artistic exceptions, we want the image to be clear without any blurring and therefore the exposure time to be as short as possible. In 24 fps video, the opposite is true: the long exposure time for each of the 24 frames is intended to simulate the effect of a larger number of frames through blurring. The effect is that during a running scene, for example, you don't see the runner's hand at all, just a blurry patch. This is the case at 24 fps with a typical film exposure time of 1/48 s. You can read more about this here: https://www.red.com/red-101/shutter-angle-tutorial

If the above example run had been recorded at 120 fps or 240 fps the runner's hand would be sharp and clear on every frame. RIFE and other typical AI algorithms will do well to interpolate 24 fps to 120 fps when the exposure time of each frame is very short at, say, 1/240 s and the details on each frame are sharp. However, there are no such videos, well, unless we create them ourselves, but then the question arises: why not record straight away in 120 fps or 240 fps? Then no RIFE would be necessary. Unfortunately, in most cases we have to make do with what we have, which is 24 fps and heavily blurred motion. In such a 24 fps video, the runner's hand will be blurred in the original and it will be blurred in the 120 fps created by the RIFE.

This is where the problem lies. We have a smooth 120 fps video obtained by interpolation with RIFE, but our brain nevertheless perceives that something is wrong. Something is artificial and many people hate interpolation precisely for this. Many explain to themselves that the magic of cinema is not there, that this was not the director's intention, that the effect is a soap opera.... however, this is just looking for an explanation that our brain hates what it sees. Can it be explained somehow?

Yes, it can be explained by the 'uncanny valley' effect: https://en.wikipedia.org/wiki/Uncanny_valley This whole magic of cinema is nothing more than the fact that 24 fps is so far from reality that our brain immediately knows it's a movie, entertainment and not reality. This is the same as an industrial robot for the 'uncanny valley' effect. We're 100% sure it's a robot so it doesn't cause us concern or uncertainty about what we're dealing with.

Now let's get back to what video frame interpolation does: 120 fps takes us away from the magic of cinema and brings us closer to reality. Of course, we all want movies to be more imersive and as real as possible: 3D, higher resolution and more frames per second. The problem is that there remains too much artificiality in the interpolation of video frames with RIFE (motion blur even at 120 or 240 fps), which causes us to fall right into that famous 'uncanny valley' when approaching reality. The film is smooth: 120 fps, but there is something wrong with it. Every frame has an unnaturally long blur for 120 fps. Despite 120 fps, certain details, instead of being clear and sharp, are blurred and distorted by motion blur.

Of course, we accept the compromises because we know that RIFE is still the best way to get us closer to the ideal and we have no alternative. In the same way that for many years we accepted artifacts from the basic SVP algorithm because there was no alternative. However, there are some people to whom, if we show the effect of RIFE or SVP interpolation, will still prefer the original 24 fps. Often this is due not to stubbornness or conservatism, but to the simple fact that there is something wrong with the interpolated image.

My dream is to interpolate the video frames of a 24 fps movie in such a way that we get the effect we would get from watching that movie shot natively in 120, 240 or 500 fps. Yes, we already have 500 Hz monitors and this effect can be achieved!

Image Scaling
https://www.svp-team.com/wiki/Manual:Resizing

If you want to change the settings precisely:
https://www.svp-team.com/wiki/Manual:Advanced_SVP
https://www.svp-team.com/wiki/Manual:All_settings

https://www.svp-team.com/forum/viewtopi … 114#p82114

v2 only improve performance. Quality remains the same, as it is still based on the current RIFE v4.6 model

https://docs.nvidia.com/deeplearning/te … dding.html

Chainik will probably know better what this is all about.

Thanks Mardon85 for the further tests and I'm very glad you found a way to interpolate x3 in real time. I think the setting you showed might also help someone.

One more thing can increase efficiency:

This update came out a few days ago: https://github.com/AmusementClub/vs-mlr … c649dfb212

I guess all you need to do is replace the vsmlrt.py file and download the rife v2 models. And maybe remove explicit padding, I am not a programmer.

Share your impressions on how this modification, and what Mardon85 suggests, affects performance.

30.5% increase in performance with HAGS OFF during encoding:
https://www.svp-team.com/forum/viewtopi … 819#p81819

On the other hand, with real-time playback we have:

10.8% increase in performance with HAGS OFF
=(3200*1800)/(3040*1710)

over 59.6% performance increase with 6.7% more RAM bandwidth
=(3840*2160)/(3040*1710)

I know that with a more precisely run test the performance increase is unlikely to exceed the increase in RAM bandwidth, but I think this sufficiently confirms that with a top-of-the-line PC for the present times the bottleneck for 4K HDR files is RAM bandwidth.