1 Topic by Sagan (edited by Sagan 18-03-2024 22:05:43)

Topic: Support for video with inconsistent animation (READ DESC)

Some videos/cartoons tend to have the characters animate at a lower framerate, whereas cars and flying objects seem to animate more smoothly. For example, Work It Out, Wombats has the characters animate at ~15fps and the video is rendered at 30fps. Remy and Boo is another perfect example. In the past (July 2023), SVP4 smoothens out a few parts but is unable to do the rest, making the video look blurry and warped.

Does SVP4 + MPC (K-Lite Codec Pack) support these kinds of videos? And if so, how do I set it up to smoothen them out? Also, what kind of video is this called (the inconsistent animations part)?

Windows 10 Pro
Nvidia GeForce RTX 2060 Super
Intel Xeon E5-1620 v3 @3.5GHz
1080p Sceptre LED display refreshing at 75Hz
32GB RAM

  • Thanks: 1

Re: Support for video with inconsistent animation (READ DESC)

RIFE with v4.7-4.10 models are said to be optimized for anime:
https://github.com/megvii-research/ECCV … me-ov-file

Re: Support for video with inconsistent animation (READ DESC)

Kadet89 wrote:

RIFE with v4.7-4.10 models are said to be optimized for anime:
https://github.com/megvii-research/ECCV … me-ov-file

Here are the methods I tested (VapourSynth):
Automatic+60fps+Animation on Remy and Boo episode converted to 20 fps
Automatic+60fps+Animation on Remy and Boo episode converted to 15 fps
RIFE Nvidia TensorRT 4.6+48fps
RIFE Nvidia TensorRT 4.9+48fps
RIFE Nvidia TensorRT 4.14+48fps
RIFE Nvidia TensorRT 4.14 ensemble+48fps
RIFE Nvidia TensorRT 4.10 ensemble+48fps
RIFE Vulkan 4.4+48 fps
Any configuration above with Movie×y/Movie frame rate×y, where y is the multiplier for the frame rate

It seems like SVP 4 Pro has a hard time on Remy and Boo-style animations, specifically (not just this one title, though). In one case, one character moves one frame, while the next character moves the next frame, so converting the video to a lower fps and then smoothing it is probably the best option as of now. The other case, the background and transitions animate very smoothly, while character animations, don't work as well.

I think the main problem with SVP 4 Pro is that it doesn't smoothen out some animations that have a greater distance between the first and the second frames, such as characters running.

Alva's World, Luo Bao Bei, and Work It Out, Wombats have been tested and the best working one for me is either the Automatic+60fps or the RIFE Nvidia TensorRT configs.

I have attached a sample video for testing purposes (attachments don't seem to work for me in the forums, so it has been uploaded to OneDrive for now)
https://1drv.ms/v/s!Ak5br0jCI8hDiIwzDD8 … Q?e=ketbEO

  • Thanks: 99

Re: Support for video with inconsistent animation (READ DESC)

Try latest RIFE 4.15 too because

dawkinscm wrote:

It is a clear improvement over previous models for reducing double images in fast movement.

SVP allows u to fill the gaps between their source frames

Sagan wrote:

For example, Work It Out, Wombats has the characters animate at ~15fps and the video is rendered at 30fps.

but if the animators were cost cutting trolls then not even AI will be able to smooth out their trash animations.

  • Thanks: 1

Re: Support for video with inconsistent animation (READ DESC)

CGI with such animation style is a rare thing, no wonder RIFE can't handle it properly. I bet there is no such source data in the training materials. You could inform the authors of the models about this problem, probably they will improve interpolation of such videos in future models.