726 Reply by UHD (edited by UHD 05-04-2023 23:45:29)

  • UHD
  • Member
  • Offline
  • Thanks: 39
  • Thanks for the post: 1

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

Blackfyre, of course not and there is a good reason for this. The RIFE models that are described in that paper you linked to: RIFE, RIFE-Large and RIFEm are optimised to achieve the best possible PSNR and SSIM. Unfortunately, many researchers stop there, because that is enough for publications and citations. However, there are researchers who also provide models optimised for human perception and measures that better reflect it: LPIPS, FloLPIPS, VFIPS.

RIFE developer hzwer did not include models optimised for human perception in the paper you linked to, but he did in a separate repository to which you have a link here: https://github.com/hzwer/Practical-RIFE The practical models are, of course, based on the models contained in the scientific paper you provided a link to and so, for example, the v4.0-v4.6 practical models are based on the RIFEm model. The difference is, the v4.0-v4.6 practical models are better optimised in terms of human perception, inference performance and trained on more data. These models use SVP, including the latest v4.6, but not directly.

To further improve performance and take advantage of the fastest Tensor cores of NVIDIA graphics cards for AI inference, SVP uses versions of these models from this repository: https://github.com/AmusementClub/vs-mlrt The authors of this repository have recently included a second version of the implementation of these models, and described the changes in this update: https://github.com/AmusementClub/vs-mlr … c649dfb212

To summarise:

The best current model is:
rife_v4.6_ensemble.onnx from the v2 package: https://github.com/AmusementClub/vs-mlr … e_v2_v1.7z

If someone is not able to use the full capacity of their screen with this model, for example to interpolate 4K files, it is worth using a faster model, with a small loss of quality:
rife_v4.6.onnx from the v2 package: https://github.com/AmusementClub/vs-mlr … e_v2_v1.7z

If anyone is interested in much, much better quality than RIFE interpolation I will write about it soon, here on this forum. Please give me another week, maybe two weeks, until I update my repository on GitHub.

727 Reply by Blackfyre (edited by Blackfyre 05-04-2023 23:55:49)

  • Thanks: 153
  • Thanks for the post: 1

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

Great write-up @UHD thank you for the details. Appreciate it and I will look at the links in detail later today.

I noticed with the current version with MPV there are still some artifact issues that I hope can be handled better in the future.

For example, within the film Tenet, around the 35 minute mark the texture on the clothes has bad artifacting.

Tried to capture it here, link will be up for 24 Hours only. Source is 4K Bluray Remux of the film:

https://streamable.com/q9vj6q

Also, after the 11th minute mark there are artifacts on the antennas at the back of the boat as it travels on water:

https://streamable.com/7kdd13

Not sure if versions above have fixed such issues, but it would be nice to have improved artifact masking.

Really appreciate RIFE, it is by far the best motion interpolation method I have seen thus far.

---

EDIT:

I backed up, then replaced the ones you linked with the ones inside (C:\Program Files (x86)\SVP 4\rife\models\rife), but it did not work. I will wait for the update via the app.

728 Reply by UHD (edited by UHD 06-04-2023 00:11:57)

  • UHD
  • Member
  • Offline
  • Thanks: 39

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

Blackfyre wrote:

EDIT:

I backed up, then replaced the ones you linked with the ones inside (C:\Program Files (x86)\SVP 4\rife\models\rife), but it did not work. I will wait for the update via the app.


Chainik wrote:

replace SVP 4\rife\vsmlrt.py

https://www.svp-team.com/forum/viewtopi … 114#p82114


Blackfyre wrote:

Not sure if versions above have fixed such issues

v2 only improve performance. Quality remains the same, as it is still based on the current RIFE v4.6 model

  • Thanks: 47

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

UHD wrote:

If anyone is interested in much, much better quality than RIFE interpolation I will write about it soon, here on this forum. Please give me another week, maybe two weeks, until I update my repository on GitHub.

Oh? Patiently waiting for your post now...

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

UHD wrote:

If anyone is interested in much, much better quality than RIFE interpolation I will write about it soon, here on this forum. Please give me another week, maybe two weeks, until I update my repository on GitHub.

This is very exciting by the way. I don't know how anything can be better than RIFE, basically RIFE with better Artifact Masking for me is the ultimate goal (just artifact masking getting better and better). Because smoothness wise, RIFE is already insane, so I can't imagine better. Maybe efficiency? Because RIFE is very demanding.

Couple of questions if you don't mind me asking:

1. Will it be as efficient or more efficient than RIFE interpolation? Or will it be more demanding?

2. Is this live interpolation? Or will it have to be via transcode. I only use the live options on 4K content.

As an RTX 3090 + Ryzen 5800X owner, I am really looking forward to it. Thank you.

731 Reply by dawkinscm (edited by dawkinscm 07-04-2023 00:40:52)

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

Blackfyre wrote:
UHD wrote:

If anyone is interested in much, much better quality than RIFE interpolation I will write about it soon, here on this forum. Please give me another week, maybe two weeks, until I update my repository on GitHub.

This is very exciting by the way. I don't know how anything can be better than RIFE, basically RIFE with better Artifact Masking for me is the ultimate goal (just artifact masking getting better and better). Because smoothness wise, RIFE is already insane, so I can't imagine better. Maybe efficiency? Because RIFE is very demanding.

I agree. Smoothness is practically a non issue now because Rife is so smooth and SVP with Rife has much fewer artefacts. But artefacts are still noticeable and in some cases Rife introduces artefacts that weren't there before. Like the one in Gravity which looks like someone pulling a dark coat a cross the stars in the background.

732 Reply by Fortune424 (edited by Fortune424 11-04-2023 05:01:34)

  • Thanks: 2
  • Thanks for the post: 1

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

To anyone confused, these are the full instructions to get the best setup going right now as far as I am aware. Please correct me if I'm wrong. With all due respect to Chainik, I think they're too smart and could benefit from giving a little more detail in the instructions for us non-developers.

1. Install SVP - select whatever player you want, I'm using MPV because back in the previous pages people claimed it was faster than MPC-HC/BE.
2. Go to Utilities -> Additional Programs and Features
3. Install the RIFE AI thing and the Tensor RT thing at the bottom. I also chose MPV Shared Libraries, not sure if it matters.
4. Restart the program.
5. Download the V2 model folder from here. https://github.com/AmusementClub/vs-mlr … e_v2_v1.7z SEE FOOTNOTE - MAYBE STOP HERE FOR NOW
6. Extract the "rife_V2" folder into your models folder (SVP 4\rife\models) so it sits beside the "rife" folder.
7. You should have a path like "C:\Program Files (x86)\SVP 4\rife\models\rife_v2\rife_v4.6.onnx" when this is done.
8. Download the updated vsmlsrt.py from here: https://www.svp-team.com/forum/misc.php … download=1
9. Replace "SVP 4\rife\vsmlrt.py"
10. Restart SVP 4.
11. Now you can select your settings as before. This seems optimal for me:
https://i.imgur.com/5UpwsF7.png
12. The thing here that confused me is that you don't have to select the V2 model specifically. It seems like SVP chooses the V2 model automatically if the folder structure exists like that. So you'll be using the non-ensemble V2 rife 4.6 model by default if you have your thing set up like mine.
13. I'm doing x3 for 21:9 1080P and below, and x2 for 4K and 16:9 1080P. Seems to work okay with a 3090 though I've only tested a couple videos, I'll report back.
14. The threads doesn't seem to make much of a difference.

Thank you to Chainik and the SVP team (IS there a team?) for making a great program, and thanks to UHD and everyone else who has contributed to this thread. The RIFE interpolation looks so good. I look forward to seeing it progress, and eagerly await the theoretical RTX 5090. Told myself I'd sit out this current generation now that GPUs cost more than my first car.

---



NOTE: If you read further down, we've determined V2 may be substantially reducing visual quality (at least at 4K). Recommend sticking with the out of the box RIFE model configuration for now. (V1 4.6 + TensorRT + performance boost).



---

733 Reply by dawkinscm (edited by dawkinscm 07-04-2023 17:37:45)

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

I could have done with this info a few weeks ago smile But at least it made me go deeper and look into the code itself which has helped.

Fortune424 wrote:

12. The thing here that confused me is that you don't have to select the V2 model specifically. It seems like SVP chooses the V2 model automatically if the folder structure exists like that. So you'll be using the non-ensemble V2 rife 4.6 model by default if you have your thing set up like mine.

Yes I did comment on this a few weeks ago but I got not reply and in general I haven't been getting any replies from the devs. There is a single piece of "wrapper" code for all models. I'm not a Python person but it looks like it defaults to Rife V2  for the Tensor backend if the Rife v2 folder and onnyx models exist. Otherwise it defaults back to Rife V1.

I don't need it to be any better for smoothness and as I said before, it is mostly way better for artefacts. But I just wish there was some way to adjust for artefacts because it does sometimes introduce it's own kind of artefacts.

734 Reply by dlr5668 (edited by dlr5668 08-04-2023 08:18:57)

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

v2 doesnt work with performance boost = disabled. But aside from this its fine

  • Thanks: 2

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

dawkinscm wrote:

I could have done with this info a few weeks ago smile But at least it made me go deeper and look into the code itself which has helped.

Fortune424 wrote:

12. The thing here that confused me is that you don't have to select the V2 model specifically. It seems like SVP chooses the V2 model automatically if the folder structure exists like that. So you'll be using the non-ensemble V2 rife 4.6 model by default if you have your thing set up like mine.

Yes I did comment on this a few weeks ago but I got not reply and in general I haven't been getting any replies from the devs. There is a single piece of "wrapper" code for all models. I'm not a Python person but it looks like it defaults to Rife V2  for the Tensor backend if the Rife v2 folder and onnyx models exist. Otherwise it defaults back to Rife V1.

I don't need it to be any better for smoothness and as I said before, it is mostly way better for artefacts. But I just wish there was some way to adjust for artefacts because it does sometimes introduce it's own kind of artefacts.

Yeah, me too. Took me a couple days to figure it out. I had a slow few hours on top of a mountain yesterday at work and read through this whole thread and it started to make sense. Now that I've used it some more it's not perfect. I notice anything with text whether it's part of the show or hardcoded subtitles/information looks funky and there is occasionally issues during scene changes and stuff. In typical motion though I think it's better looking than anything else still.

736 Reply by Blackfyre (edited by Blackfyre 07-04-2023 21:20:21)

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

For the ones above, after testing yesterday, the quality with RIFE that comes with SVP is better than the v2 I updated to via the links above.

Current RIFE with SVP shows less artifacts for example on intros when names are showing up, and smoother overall.

rife_V2 is creating artifacts and the quality seems to drop as well on 4K content looking like 1080p, at least with my MPV settings.

Forgot to back up my vsmlrt.py and had to remove RIFE and add it back. Now picture quality and smoothness is back to being better.

By the way @Chainik, if it's possible to add a fourth option under maintenance called "Check File Integrity". So, it checks all the SVP files and replaces them with the default ones from SVP. Overriding any changes we make to RIFE or MPV folder, etc.

737 Reply by dawkinscm (edited by dawkinscm 07-04-2023 21:59:48)

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

Blackfyre wrote:

For the ones above, after testing yesterday, the quality with RIFE that comes with SVP is better than the v2 I updated to via the links above.

Current RIFE with SVP shows less artifacts for example on intros when names are showing up, and smoother overall.

rife_V2 is creating artifacts and the quality seems to drop as well on 4K content looking like 1080p, at least with my MPV settings.

Forgot to back up my vsmlrt.py and had to remove RIFE and add it back. Now picture quality and smoothness is back to being better.

By the way @Chainik, if it's possible to add a fourth option under maintenance called "Check File Integrity". So, it checks all the SVP files and replaces them with the default ones from SVP. Overriding any changes we make to RIFE or MPV folder, etc.

I think the v1/v2 debate may be dependant on local environment because I went back to v1 and didn't notice any improvements over v2. If anything it was slightly worse. Then I tried ensemble and the issues with the star field in Gravity became much worse. This scene seems to be Rife's Kryptonite. The same scene on Automatic has no issues.

  • Thanks: 2

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

dlr5668 wrote:

v2 doesnt work with performance boost = disabled. Buts aside from this its fine

The annoying thing about the performance boost for me has been movies that use multiple aspect ratios / resolutions. For example I'm watching Resident Evil (2002) BluRay right now and there's a 16:9 camera, an almost 16:9 camera, a 21:9 camera, 4:3 CGI for the computer vision/security camera stuff, etc. It has stopped to generate the training info a few times. For anime/web content it's not so bad - you've basically just got 21:9 and 16:9 versions of 480P through 4K and then the occasional portrait/4:3 video.

I wonder if we can create a repository for the training info per GPU. It seems anyone interested in this sort of thing, at least for now, is going to gravitate around the last couple generations of high end NVIDIA cards. In this thread for example, every other person is on a 3090 or 4090.

@BlackFyre I think I see what you're talking about with the quality reduction. I also tried Avatar Way of Water (like a 34GB 4K file) and it looked kind of lower res than it should. I'll have to test back and forth.

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

mpv can be too wild to config imo. Can you retest with MPC-HC ?

740 Reply by Xenocyde (edited by Xenocyde 08-04-2023 10:26:19)

  • Thanks: 47

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

I switched last night to V2 as per the tutorial posted above. I can't really notice much of a difference in smoothness on my RTX 3070 laptop GPU for 1080p content. Artifact handling seems a bit better maybe.

However, I still get the framerate drops that last around 10 seconds. These usually happen after 1 h or 20-30 minutes during 2h+ movies, very rarely in TV series episodes of ~40 minutes. The SVP index drops from 1 to 0.8-0.7 usually, sometimes lower, and the GPU 3D utilization in task manager jumps from 70-80% to 100% while RAM utilization sits sub 3 GB most of the times. This is happening in MPV by the way. I need to test with MPC-HC.

741 Reply by dawkinscm (edited by dawkinscm 08-04-2023 10:36:33)

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

dlr5668 wrote:

mpv can be too wild to config imo. Can you retest with MPC-HC ?

Both MadVR and MPV require a lot of reading but "by default" MadVR has a GUI while MPV does not. I'm new to MPV but I like the additional flexibility of the text file and the better picture quality. But MPV is easy to over-configure with lines of config that are not required. For the most part, leaving the basic commands at default works and then tweaking from there works best.


Xenocyde wrote:

I switched last night to V2 as per the tutorial posted above. I can't really notice much of a difference in smoothness on my RTX 3070 laptop GPU for 1080p content. Artifact handling seems a bit better maybe.

However, I still get the framerate drops that last around 10 seconds. These usually happen after 1 h or 20-30 minutes during 2h+ movies, very rarely in TV series episodes of ~40 minutes. The SVP index drops from 1 to 0.8-0.7 usually, sometimes lower, and the GPU 3D utilization in task manager jumps from 70-80% to 100% while RAM utilization sits sub 2 GB most of the times. This is happening in MPV by the way. I need to test with MPC-HC.

I moved away from MPC (with MadVR) because it was unusable with Rife. With MPV I get better performance, almost perfect smoothness, less artefacts and much better interpolated picture quality than with MPC/MadVR so I can never go back. With a 3070 laptop GPU you are always going to have issues but maybe with MPC it will work better for you.

Note: Edited to say "unusable" -> I moved away from MPC (with MadVR) because it was "unusable" with Rife.

742 Reply by Xenocyde (edited by Xenocyde 08-04-2023 12:08:02)

  • Thanks: 47

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

dawkinscm wrote:

I moved away from MPC (with MadVR) because it was usable with Rife. With MPV I get better performance, almost perfect smoothness, less artefacts and much better interpolated picture quality than with MPC/MadVR so I can never go back. With a 3070 laptop GPU you are always going to have issues but maybe with MPC it will work better for you.

I installed MPC-HC and configured it to run through VapourSynth. Have you tested with this setup? I'm testing now, will report any improvements on the 3D load later.

L.E: Tried a 40 minute episode. Everything seems the same in MPC-HC quality-wise, except the 3D load is 10% lower on average. MPV hits 70%, while MPC-HC barely hits 60%. This could be good news, but need to test further with a long movie, as the frame drops rarely occur in 40 minute episodes with MPV as well.

743 Reply by dawkinscm (edited by dawkinscm 08-04-2023 20:12:40)

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

Xenocyde wrote:
dawkinscm wrote:

I moved away from MPC (with MadVR) because it was usable with Rife. With MPV I get better performance, almost perfect smoothness, less artefacts and much better interpolated picture quality than with MPC/MadVR so I can never go back. With a 3070 laptop GPU you are always going to have issues but maybe with MPC it will work better for you.

I installed MPC-HC and configured it to run through VapourSynth. Have you tested with this setup? I'm testing now, will report any improvements on the 3D load later.

L.E: Tried a 40 minute episode. Everything seems the same in MPC-HC quality-wise, except the 3D load is 10% lower on average. MPV hits 70%, while MPC-HC barely hits 60%. This could be good news, but need to test further with a long movie, as the frame drops rarely occur in 40 minute episodes with MPV as well.

Update since response from @Chainik
Using MPV is similar to using MPC with MadVR so if you are just using MPC-HC then that's probably why you are getting a lower 3D load. Now that I've go MPC+Rife working with VapourSynth, I see that is almost as smooth as with MPV and if I hadn't discovered MPV I would be pretty happy. But I can't see me going back to using MPC when MPV is basically MPC+MadVR but with better picture quality and fewer micro-stutters with Rife.

  • Thanks: 1622
  • Thanks for the post: 1

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

> You never know, I might get a response this time

how do you think, what devs might think about replying the same question for the 100th time? hmm

if you're using your own vapoursynth installation you must be knowing what you're doing

745 Reply by dawkinscm (edited by dawkinscm 08-04-2023 20:12:52)

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

Chainik wrote:

> You never know, I might get a response this time

how do you think, what devs might think about replying the same question for the 100th time? hmm

if you're using your own vapoursynth installation you must be knowing what you're doing

First of all thank you @Chainik I searched and couldn't find any responses to my question about Rife v2 being automatically selected when using Tensor which is why I wasn't happy when I got no response. But in the end it was to my benefit because it made me go through the code. I am pretty sure I had VapourSynth working before so I'm not sure why I installed an external version which ended up breaking it so again I appreciate your feedback. But it did lead to me discovering MPV so the two issues turned into two positives. Apologies and thanks again smile

  • Thanks: 47

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

Tested MPC-HC with Avatar 2. 3D load was hitting 80% and after around 1h it jumped to 100% for about 5 seconds. MPV has similar load, max 85%. After that, I decided to test once again with HAGS turned off. 3D load sits at 15% even in MPV! Haven't noticed spikes for the remainder of the movie (around 2 more hours). Can't remember the exact improvements with HAGS off for V1 ... maybe 3D load was 20% lower because I was still getting spikes to 100% although less frequent. V2 is waaay better in this regard.

747 Reply by aloola (edited by aloola 11-04-2023 00:11:44)

  • Thanks: 86
  • Thanks for the post: 1

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

Blackfyre wrote:

For the ones above, after testing yesterday, the quality with RIFE that comes with SVP is better than the v2 I updated to via the links above.

Current RIFE with SVP shows less artifacts for example on intros when names are showing up, and smoother overall.

rife_V2 is creating artifacts and the quality seems to drop as well on 4K content looking like 1080p, at least with my MPV settings.

I have the same issue too.


I tested a 4k file with RIFE V2 4.4, 4.6, the image quality has degraded a lot.

https://cdn.discordapp.com/attachments/290709370600423424/1095130933562187847/image.png

I need more people to confirm the bug.

edit: I've just tested with MPV, has the same problem too.


NO RIFE: https://cdn.discordapp.com/attachments/ … _06_AM.png

with RIFE V2: https://cdn.discordapp.com/attachments/ … _37_AM.png

748 Reply by Fortune424 (edited by Fortune424 11-04-2023 00:34:44)

  • Thanks: 2

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

I think I can confirm. It really is painfully obvious when you do a side by side comparison. I bet V2 looks worse than the 1080P BluRay would. I guess the free performance boost was too good to be true, for now.

Here's from the opening of the 4K BluRay of Avatar 2:

https://i.imgur.com/gqoKFqP.png

MPV with stock SVP settings, RIFE 4.6 w/ TensorRT on a 3090.

  • Thanks: 153
  • Thanks for the post: 1

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

Fortune424 wrote:

I think I can confirm. It really is painfully obvious when you do a side by side comparison.

I didn't even do side-by-side, the moment I upgraded and played the first video, I was like what am I looking at it lol Good to see it confirmed in pictures though above.

I don't think it's a worry anyway, since SVP Devs likely knew the issue and that's why SVP still runs with v1

Like I said before, quality and smoothness of RIFE is pretty much unmatched right now. Personally, the only improvements I want to see from hereon is for artifact masking.

750 Reply by dawkinscm (edited by dawkinscm 11-04-2023 04:51:32)

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

Fortune424 wrote:

I think I can confirm. It really is painfully obvious when you do a side by side comparison. I bet V2 looks worse than the 1080P BluRay would. I guess the free performance boost was too good to be true, for now.

Here's from the opening of the 4K BluRay of Avatar 2:

MPV with stock SVP settings, RIFE 4.6 w/ TensorRT on a 3090.

I don't think it does look worse than a 1080 BluRay. I use VR to watch 3D and as far as I can tell, I don't get the same issues. Maybe there's a resolution issue, but I checked and all the detail is there when compared to the original Blu-ray when using Rife v2 with no quality difference to Rife v1. I also checked a number of difficult scenes from other movies and the same applies there too. So maybe it's a 4K resolution issue but I'm not interested in 4K playback using SVP so I haven't tried that. Stock MPV settings uses gpu and I do notice a marked improvement in quality when using gpu-next with proper configuration so you may wish to try that as well.