• Thanks: 4

Topic: some ideas about how to improve motion estimation

Hey guys,

two ideas:

1) Have you thought about making use of the motion information from the video bitstream? Obviously this information is missing for key frames, and it's probably not completely reliable. But it might serve as a first guess, or as a low CPU variant.

2) I've seen Didée (doom9) comment that some of the AviSynth algos don't really track motion, but just try to find a block match. Is that the case for SVP? Or is SVP actively trying to find *continuous* motion vectors over multiple frames?

Maybe both 1) and 2) could be combined? E.g. as an ultra low CPU/GPU power algo, you could rely on the video bitstream motion information, and refine that by tracking motion vectors over multiple frames. E.g. let's say that block A moves 5 pixels right from frame X to frame X+1, and another 5 pixels right from frame X+1 to X+2. And let's say frame X+3 is a key frame. So motion information is missing for the key frame. So SVP could just "continue" all previous motion vectors in the same speed for the key frame. Maybe it would make sense to do some block comparisons just to make sure that continuing the motion vectors seem "likely" for the key frame, just to be safe. Or maybe SVP could run its usual algos just for key frames, but rely on the video stream motion vectors for the other frames? It might make sense to ignore "random" motion vectors which are not continuous, and just use those which stay similar over at least 3 frames or so.

Just some thoughts I had...

Re: some ideas about how to improve motion estimation

SVP is not a part of decoder. How it could possibly get MVs from the stream?
Besides stream vectors are not the same. As Didee said - they're not intended to find a motion but just a closest matched block.
While "motion" is a coherent vectors field.

  • Thanks: 4

Re: some ideas about how to improve motion estimation

Yes, but if you *could* get access to the MVs from the stream, would you consider it useful to improve SVP performance? That's the key question. It might save some performance, or not?

I know that you currently don't have access to the video bitstream, but it's not impossible to get access. Here are some ways that could be achieved:

1) Obviously, you could integrate a decoder. Shouldn't be that hard, libav/ffmpeg does all the heavy lifting. But it's still somewhat ugly. So the other solutions might be better.
2) For DirectShow playback, you could ask nevcairiel to add an interface to LAV Video Decoder that allows you to get access to the bitstream / MVs.
3) For DirectShow playback, you could hook the decoder's input pin's "Receive" method. Not that difficult. madVR already does that to get some added information from the video bitstream.

I know, it's not an easy solution. Interpreting the video bitstream to extract the MVs might already require many hours of development, and that doesn't even include getting access to the bitstream in the first place.  And when you do get access to the bitstream, there's still the problem of matching the bitstream to the decoder output. And maybe the bitstream MVs don't provide very good motion information. But I just thought I'd drop the idea here, in case it might help SVP to improve its quality/performance ratio.

  • Thanks: 1622
  • Thanks for the post: 1

Re: some ideas about how to improve motion estimation

madshi
It might save some performance, or not?

I don't think so. First estimation is already very fast, all CPU horsepowers are wasted on refinement.


you could ask nevcairiel to add an interface

Hmm, this's interesting. You just gave me another idea.
We could ask you to add Vapoursynth processing into madVR smile You know, many people dream of SVP integrated into madVR...

  • MAG79
  • SVP developer
  • Offline

Re: some ideas about how to improve motion estimation

madshi
The main difference between decoders MVs and "truemotion" MVs is the aim of MVs. And from here type of MVs.
Decoder uses block-matching one-to-one, one-to-many, many-to-one types of MVs to get maximum compression. So decoder generally has MVs not shown the real motion direction.
SVP (MTools and SVPFlow libs) uses modified block-matching algorythms to get in ideal one-to-one type of MVs to get real motion.

  • Thanks: 33

Re: some ideas about how to improve motion estimation

As 4k and 120hz displays start to come out, it may be necessary to have some form of integration between madVR and SVP to be able to run both on a 4k 120hz display.

Currently, if you use SVP and then madVR, madVR has only 8.3ms to upscale each frame to 4k.

If madVR was to be processed first, madVR would have much more time to process each frame and SVP would have higher quality frames to work with. However, SVP would then have RBG data instead of YUL which would be heavier to process so the algorithm would have to be tweaked.

According to my own testing, any final sharpening has to be done after SVP otherwise the output looks weird.

The only way to process SVP after madVR would be to have an integration between both.

This would be the only way to support 4k 120hz displays. Not sure how it could be done, but something to keep in mind. Before these displays become popular and the technology requires it.

Re: some ideas about how to improve motion estimation

4K in RGB will eat any 6-core i7.

And I'm not sure madshi will be happy with the idea of returning rendered frames back to system memory.

8 Reply by Mystery (edited by Mystery 28-06-2015 18:19:03)

  • Thanks: 33

Re: some ideas about how to improve motion estimation

So is there any hope for 4k?

The problem is that the amount of pixels grows exponentially, but the CPU/GPU performances don't.

Re: some ideas about how to improve motion estimation

4K after madVR? No.

Just in case madshi will come back - don't miss that message!

Re: some ideas about how to improve motion estimation

Chainik wrote:

4K in RGB will eat any 6-core i7.

Then get an 8-core i7. tongue

Or wait a year for Zen.

  • Thanks: 4
  • Thanks for the post: 4

Re: some ideas about how to improve motion estimation

Sorry for the very late reply. Somehow the forum notification about the new posts didn't seem to reach me.

Chainik wrote:

We could ask you to add Vapoursynth processing into madVR smile You know, many people dream of SVP integrated into madVR...

I'm absolutely open to that. Not right now, too busy with other stuff, but sooner or later. Would you mind sending an email to my email account (madshi (at) gmail (dot) com), so that when I'm ready to look into that, we can talk via email, if that's alright with you? I think that would be more efficient than using the forum.

  • Thanks: 14

Re: some ideas about how to improve motion estimation

SVP/MadVR integration? This is gonna be good smile

  • Thanks: 33

Re: some ideas about how to improve motion estimation

Yep. I see better results when changing the order of operations. When doing frame-quadrupling, best results are when running SVP in-between the two doubles. BUT, as it was mentioned earlier, you'd then have the problem of SVP working with RGB data instead of YUV which would be much more demanding on ressources.

Re: some ideas about how to improve motion estimation

madshi wrote:
Chainik wrote:

We could ask you to add Vapoursynth processing into madVR smile You know, many people dream of SVP integrated into madVR...

I'm absolutely open to that. Not right now, too busy with other stuff, but sooner or later. Would you mind sending an email to my email account (madshi (at) gmail (dot) com), so that when I'm ready to look into that, we can talk via email, if that's alright with you? I think that would be more efficient than using the forum.

That would be SO great. Still hoping for news on that :-)

15 Reply by SamE (edited by SamE 04-06-2018 01:54:59)

  • SamE
  • Member
  • Offline

Re: some ideas about how to improve motion estimation

I really hoped something would come out of this sad
But so far almost 3 years later madshi haven't implemented anything like a vaporsynth interface into madVR.

And as far as i understand it we really need this because otherwise we can't get proper 2160p 10-bit HDR BluRay/Remuxes working in MadVR/mpc-be/hc with SVP.
Now that madVR has really good HDR to SDR tonemapping/conversion + NGU on the chroma that really makes the UHD sharpness pop, i can't imagine not using madVR for UHD.
But i am really missing SVP in all of this.

Can i ask if anything came out of the supposed discussions madshi suggested you had in private to solve this?
Is there any sort of plan on how this will be solved?

Maybe a separate directdraw vaporsynth filter with 10-bit and HDR metadata handling could be developed or maybe vaporsynth integration in mpc-be/hc like you did with mpv.
Because this seems to me anyway as the last puzzle piece we need for the next level of SVP.