Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

dawkinscm wrote:

Update:  My VR headset also has increased CPU and decreased GPU usage effectively spreading resource usage across both CPU and GPU. I checked my Nvidia settings and there are still no changes. So this might be some kind of Microsoft/Nvidia optimisation.

So I think this is actually an SVP fix for image scaling rather than a Nvidia/Microsoft fix. I mentioned previously that I had issues with Dr Strange 2 but that's because it was 3D SBS and for whatever reason SVP did not handle it correctly. But since the latest updates, it is handled in the same way as files of the same resolution in other 3D and non 3D formats.  Correct scaling is responsible for the improvement I see with my GPU, not Nvidia/Microsoft.

1,727 Reply by Chainik

  • Thanks: 1595
  • Thanks for the post: 1

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

DolbyVision blinking must be fixed at last big_smile

plus DV colors (i.e. brightness) were incorrect because of wrong colorspace conversion before/after RIFE processing

1,728 Reply by Chainik

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

Drakko01 wrote:
Blackfyre wrote:
Drakko01 wrote:

I pointed out one example of this in Doctor Strange Multiverse of madness, after defender Strange is killed by the Ribboned Creature and America chavez run underneath.

Can you provide the timestamp? I have that in high quality and I can test tomorrow.

3.06 and 3.16 the ribbons of the creature , maybe I misinterpreted and its something else.Thanks for taking the time.

---

smooth.scene.limits.scene = 8000;

will do the trick for those scenes
---
updated defaults in svpflow libs, ver. 273-1

1,729 Reply by dawkinscm (edited by dawkinscm 09-07-2024 12:15:41)

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

Chainik wrote:
Drakko01 wrote:
Blackfyre wrote:

Can you provide the timestamp? I have that in high quality and I can test tomorrow.

3.06 and 3.16 the ribbons of the creature , maybe I misinterpreted and its something else.Thanks for taking the time.

---

smooth.scene.limits.scene = 8000;

will do the trick for those scenes
---
updated defaults in svpflow libs, ver. 273-1

With this latest update, all 3 SC algorithms are produce very similar results. But an example of why IC is slightly better is in Dr Strange 2 where he throws his cloak to catch the girl. At the point she is caught the scene changes to show her being carried back. With both NVOF and SVP this is a very sharp and jarring cut. With IC there's still an issue but it's less jarring and looks more natural.

1,730 Reply by Chainik

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

> At the point she is caught the scene changes to show her being carried back. With both NVOF and SVP this is a very sharp and jarring cut. With IC there's still an issue but it's less jarring and looks more natural.

So MVs see the scene change while IC doesn't, and still it's "better". OK then big_smile

1,731 Reply by dawkinscm (edited by dawkinscm 09-07-2024 15:15:15)

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

Chainik wrote:

> At the point she is caught the scene changes to show her being carried back. With both NVOF and SVP this is a very sharp and jarring cut. With IC there's still an issue but it's less jarring and looks more natural.

So MVs see the scene change while IC doesn't, and still it's "better". OK then big_smile

All 3 "see the scene change" but two of them makes the scene change look like a jump cut. There's a difference between reacting to the scene change and over-reacting to it. We are grateful for SVP and so we are only trying to help you make SVP even better. But in this particular instance you seem determined to not even try to understand what we are saying.

1,732 Reply by Chainik

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

there's no way to "over-react"
if it see the SC it repeats the frame before SC or blends two frames
if it miss the SC then you'll see something mixed by RIFE -> which is your case with IC
that's it

1,733 Reply by dawkinscm (edited by dawkinscm 09-07-2024 16:19:15)

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

Chainik wrote:

there's no way to "over-react"
if it see the SC it repeats the frame before SC or blends two frames
if it miss the SC then you'll see something mixed by RIFE -> which is your case with IC
that's it

But it's not missing it is it?. Wouldn't there be a clear difference between IC and disabled if it was?. But there is a difference between all 3 SC algorithms and disabled. It's just that whatever IC is doing while still there, is less jarring.  But even if you are correct then all that means is that what you call "garbage" might simply the AI doing it's job. Isn't that a good thing too?

1,734 Reply by Chainik

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

> But an example of why IC is slightly better is in Dr Strange 2 where he throws his cloak to catch the girl. At the point she is caught the scene changes to show her being carried back.

what is the timecode?

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

Chainik wrote:

> But an example of why IC is slightly better is in Dr Strange 2 where he throws his cloak to catch the girl. At the point she is caught the scene changes to show her being carried back.

what is the timecode?

He throws the cloak at 9:32 which catches her at 9:33 then the scene change happens. It's a sharp cut but I never noticed it before until I used SVP and NVOF.

1,736 Reply by Chainik

  • Thanks: 1595
  • Thanks for the post: 1

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

it's clearly a scene change and SVP do everything right - see f1.jpg before SC and f2.jpg after SC

> what you call "garbage" might simply the AI doing it's job1

see attached rife.jpg
for the 3rd time - if you OK with this - then OK, "SC = disabled" is right for you
and I'll continue thinking that proper SC detection is a must for every frame interpolation algorithm

Post's attachments

f1.jpg 165.83 kb, 14 downloads since 2024-07-09 

f2.jpg 172.03 kb, 14 downloads since 2024-07-09 

rife.jpg 140.18 kb, 14 downloads since 2024-07-09 

1,737 Reply by flowreen91 (edited by flowreen91 09-07-2024 21:34:56)

  • Thanks: 73

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

Chainik wrote:

it's clearly a scene change and SVP do everything right - see f1.jpg before SC and f2.jpg after SC

When trying to transcode with following settings https://gyazo.com/219cab646d448fdd8b58d99d07504a2d with Repeat Frames to see the difference between scene changes i'm getting the following:

DoctorStrange2.SVP_sc06nvofmotionvectors.mp4 and DoctorStrange2.SVP_sc99nvofmotionvectors.mp4 have same file sizes
meaning changing rife_sc value doesn't increase the scene detection strength and they both can't detect the cape scene to repeat it.

DoctorStrange2.SVP_sc06svpmotionvectors.mp4 and DoctorStrange2.SVP_sc99svpmotionvectors.mp4 have same file sizes
meaning changing rife_sc value doesn't increase the scene detection strength and they both can't detect the cape scene to repeat it.

DoctorStrange2.SVP_sc15imagecomparison.mp4 doesn't detect the cape scene to repeat it.
DoctorStrange2.SVP_sc06imagecomparison.mp4 detects the cape scene and repeats correctly the frame to skip interpolating it.

Questions:
1. is that cape scene not detectable by nvof and svp motion vectors? cause it always tries to interpolate it, creating 10 frames of rife.jpg
2. or since they are new functionalities they aren't correctly passed to transcoding functionality?
3. if works ok, is there a way to increase/decrease scene detection strength for nvof/svp motion detector until it detects the cape scene as a scene change to be able to repeat those frames?

Thanks!

You can check the examples here:
https://drive.google.com/drive/folders/ … tb3OOUgkz7

1,738 Reply by dawkinscm (edited by dawkinscm 09-07-2024 22:40:19)

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

Chainik wrote:

it's clearly a scene change and SVP do everything right - see f1.jpg before SC and f2.jpg after SC

> what you call "garbage" might simply the AI doing it's job1

see attached rife.jpg
for the 3rd time - if you OK with this - then OK, "SC = disabled" is right for you
and I'll continue thinking that proper SC detection is a must for every frame interpolation algorithm

So I just checked I do get the same before and after frames as shown in f1.jpg and f2.jpg, but for IC.  When I use either SVP or NVOF SC, I get the same issue as shown in Rife.JPG. It would be interesting to see what others are seeing here because I checked and rechecked using SVP SC to be sure,  so something strange is happening here. This might explain why I didn't like the look of what I see when using SVP SC.

1,739 Reply by Drakko01

  • Thanks: 36

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

dawkinscm wrote:
Chainik wrote:

it's clearly a scene change and SVP do everything right - see f1.jpg before SC and f2.jpg after SC

> what you call "garbage" might simply the AI doing it's job1

see attached rife.jpg
for the 3rd time - if you OK with this - then OK, "SC = disabled" is right for you
and I'll continue thinking that proper SC detection is a must for every frame interpolation algorithm

So I just checked I do get the same before and after frames as shown in f1.jpg and f2.jpg, but for IC.  When I use either SVP or NVOF SC, I get the same issue as shown in Rife.JPG. It would be interesting to see what others are seeing here because I checked and rechecked using SVP SC to be sure,  so something strange is happening here. This might explain why I didn't like the look of what I see when using SVP SC.

For me the frame between F1 and F2 are much better that the rife.jpg and all methods look almost the same.

1,740 Reply by dawkinscm (edited by dawkinscm 10-07-2024 04:26:37)

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

Drakko01 wrote:
dawkinscm wrote:
Chainik wrote:

it's clearly a scene change and SVP do everything right - see f1.jpg before SC and f2.jpg after SC

> what you call "garbage" might simply the AI doing it's job1

see attached rife.jpg
for the 3rd time - if you OK with this - then OK, "SC = disabled" is right for you
and I'll continue thinking that proper SC detection is a must for every frame interpolation algorithm

So I just checked I do get the same before and after frames as shown in f1.jpg and f2.jpg, but for IC.  When I use either SVP or NVOF SC, I get the same issue as shown in Rife.JPG. It would be interesting to see what others are seeing here because I checked and rechecked using SVP SC to be sure,  so something strange is happening here. This might explain why I didn't like the look of what I see when using SVP SC.

For me the frame between F1 and F2 are much better that the rife.jpg and all methods look almost the same.

They do all look almost identical. But the issue is that I am seeing the opposite to @Chainik. I see f1 and f2 with IC and Rife.jpg with SVP and NFOF. He says he sees Rife.jpg with IC.

1,741 Reply by Blackfyre (edited by Blackfyre 10-07-2024 05:13:11)

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

Chainik wrote:

DolbyVision blinking must be fixed at last big_smile

plus DV colors (i.e. brightness) were incorrect because of wrong colorspace conversion before/after RIFE processing

Amazing huge difference with DV content, great work. Now I need to re-watch House of the Dragon lol feels like a new life has been given to it.

However, some Disney+ TV Shows DV now exhibit different behavior with DV. For some reason SVP smoothness does not work properly with them now. It looks like it is dropping frames every few seconds on panning shots, but there is 0 dropped and 0 delayed frames. Very weird. Only with Disney+ TV show I am testing, with House of the Dragon and others it is working fine.

1,742 Reply by Chainik

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

Blackfyre
> For some reason SVP smoothness does not work properly with them now.

this is the cost of non-blinking
it has to repeat frame when DV data changes, which may happen at random places

1,743 Reply by Chainik

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

dawkinscm
> He says he sees Rife.jpg with IC

This isn't what I'm saying.
I'm saying you'll see either
1. repeated frame 1 - when SC detected + "Repeat frame", regardless of the method used (IC or MVs)
2. blended frame 1 and frame 2 - when SC detected + "Blend adjacent frames",  regardless of the method used (IC or MVs)
3. some garbage mixed by RIFE - when SC missed, regardless of the method used (IC or MVs)

And I'm also saying that MVs are more accurate than IC, regardless of the "threshold" value.

1,744 Reply by dawkinscm (edited by dawkinscm 10-07-2024 10:27:43)

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

Chainik wrote:

dawkinscm
> He says he sees Rife.jpg with IC

This isn't what I'm saying.
I'm saying you'll see either
1. repeated frame 1 - when SC detected + "Repeat frame", regardless of the method used (IC or MVs)
2. blended frame 1 and frame 2 - when SC detected + "Blend adjacent frames",  regardless of the method used (IC or MVs)
3. some garbage mixed by RIFE - when SC missed, regardless of the method used (IC or MVs)

Thanks. This makes things clearer so here is what I am seeing:

For Image Comparison I see point 1 and point 2 for "repeated frames" and "blend adjacent frame" respectively.
For SVP and NVOF and disabled I see point 3.

1,745 Reply by Chainik

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

flowreen91
> When trying to transcode ... to see the difference between scene changes

Hint: press '.' key in mpv to step frame by frame. Or "Ctrl+right" in MPC-HC.

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

To summarise my previous comments: I did not see the Rife "garbage" frame shown in Rife.jpg for IC but I am seeing it for SVP and NVOF.

1,747 Reply by flowreen91 (edited by flowreen91 10-07-2024 10:47:09)

  • Thanks: 73
  • Thanks for the post: 1

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

Chainik wrote:

flowreen91
> When trying to transcode ... to see the difference between scene changes

Hint: press '.' key in mpv to step frame by frame. Or "Ctrl+right" in MPC-HC.

By spamming '.' in following example we can see that the cape scene frames are correctly repeated only for the Image comparison algorithm:

SVP motion vectors:
https://gyazo.com/e72e6ff3a090edd396057d0b26eee3f7
NVOF motion vectors:
https://gyazo.com/1b5a7afb1ceabf2ecd0f4a9c6b250de8
Image comparison:
https://gyazo.com/050c30df626410bde53291b9d8c47f76

Is there anything we can do to increase NVOF and SVP algorithms strength in order for them to correctly detect it and not interpolate that cape scene change?

1,748 Reply by dawkinscm (edited by dawkinscm 10-07-2024 10:59:16)

  • Thanks: 176
  • Thanks for the post: 1

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

flowreen91 wrote:
Chainik wrote:

flowreen91
> When trying to transcode ... to see the difference between scene changes

Hint: press '.' key in mpv to step frame by frame. Or "Ctrl+right" in MPC-HC.

By spamming '.' in following example we can see that the cape scene frames are correctly repeated only for the Image comparison algorithm:

SVP motion vectors:
https://gyazo.com/e72e6ff3a090edd396057d0b26eee3f7
NVOF motion vectors:
https://gyazo.com/1b5a7afb1ceabf2ecd0f4a9c6b250de8
Image comparison:
https://gyazo.com/050c30df626410bde53291b9d8c47f76

Is there anything we can do to increase NVOF and SVP algorithms strength in order for them to correctly detect it and not interpolate that cape scene change?

Thank you! Watching on a large screen I could see the difference but I didn't know why until @Chainik suggested frame-step and this is exactly what I saw so I'm glad you were able to reproduce it here. IC is properly processing that scene. The only "garbage" Rife frames are with SVOP and NVOF which explains why I didn't like what I saw.

1,749 Reply by Chainik

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

> IC is properly processing that scene.

with a threshold value <=15
making a lot of false SC detections in other scenes

dawkinscm
> The only "garbage" Rife frames are with SVOP and NVOF which explains why I didn't like what I saw.

but you said the opposite thing - "With both NVOF and SVP this is a very sharp and jarring cut" big_smile

1,750 Reply by Chainik

  • Thanks: 1595
  • Thanks for the post: 1

Re: New RIFE filter - 3x faster AI interpolation possible in SVP!!!

> Is there anything we can do to increase NVOF and SVP algorithms strength

yep
smooth.scene.limits.scene
smooth.scene.limits.blocks

the bad thing is that values that work good with resizing to 720p doesn't seems to work with 1080p and 4K