Topic: Move to and add support for NVIDIA Optical Flow SDK 4.0

I was reading the documentation for NVOFA FRUC (see https://docs.nvidia.com/video-technolog … index.html), and it looks like this would make optical flow-based frame interpolation possible for any NVIDIA GPUs since the Turing generation.

I suppose there needs to be 2 different options offered now. 1) Only use motion vectors from NVOFA and let SVP do the rest, and 2) Allow NVOFA FRUC to generate the interpolated frames.

(2) is new since SDK 4.0 and would not have been easily possible before. I don't know how the FRUC frames' quality compares with SVP, but it would be an appreciated feature regardless of if it is technically worse. There is sample code provided from NVIDIA that shows how relatively simple it is to integrate NVOFA FRUC into applications.

  • Thanks: 1622
  • Thanks for the post: 1

Re: Move to and add support for NVIDIA Optical Flow SDK 4.0

work in progress, but with a low priority cause the current FRUC API only allows x2 interpolation which makes it not so interesting as it could be

3 Reply by OpenSourceAnarchist (edited by OpenSourceAnarchist 29-12-2022 17:51:08)

Re: Move to and add support for NVIDIA Optical Flow SDK 4.0

Great to hear, thanks for letting me know! I brought up support with MPV (see https://github.com/mpv-player/mpv/issues/11060) and it could end up getting upstreamed into FFmpeg as a filter. Could SVP handle 10/12 bit support? "The api itself says it can only accept nv12 and rgb, which would make proper handling of 10/12 bit content impossible, as you really don't want to do interpolation on the final scaled rgb and neither do you want to do rgb conversion in a filter as opposed to in the vo."

Chainik wrote:

the current FRUC API only allows x2 interpolation which makes it not so interesting as it could be

In this case, it seems the API does not really care whether you passed one if its own interpolated frames back to itself. Perhaps non-2x interpolation could be hacked in through SVP holding on to the outputted interpolated frames and passing them back in with the real next video frame, up to an arbitrary number of times to handle 1.5, 3x, 4x, 5x, etc.? The logic for when and how to do this would need to be on SVP's side obviously. Personally, I only use 2x but I understand why other ratios are desirable.