1 Topic by Skaven252 (edited by Skaven252 18-11-2012 13:59:37)

  • Thanks: 3

Topic: Fixing dropped frames in action cam footage? (for stereoscopic 3D)

I'm using two action sport cameras to shoot stereoscopic video. As in, I have two video files I need to pair into one stereoscopic video. I do this in Vegas Movie Studio.

I ran into a problem: if one of the cameras is low on batteries, it keeps recording, but starts dropping a lot more frames than the other. The framerates between the two videos can get so different that Vegas Movie Studio refuses to pair them into a stereoscopic subclip (required for 3D video in that program).

I usually re-encode the videos with an AviSynth script to remove fisheye distortion. I can use the same script to force the videos to the same framerate (like 29.970) so they become possible to pair in Vegas. But because of the dropped frames they will drift out of sync. sad

The videos have an audio track, which does not suffer from dropped frames - it is always consistent. VirtualDub, for example, can change the video's framerate to match the audio track's length. That's a good starting point, but the resulting video's framerate is still too different from the other to pair in Vegas.

So, I was wondering: is it possible to use SVP to repair those dropped frames, and give both videos the same framerate, matched to the audio track length? I haven't used SVP yet, but I thought I'd ask first.

(I guess one potential problem is that SVP couldn't really know when exactly along the timeline do those dropped frames occur?)

Thanks!

  • MAG79
  • SVP developer
  • Offline

Re: Fixing dropped frames in action cam footage? (for stereoscopic 3D)

Hi, Skaven252

You can try script DoubleDropFix.avs from Russian topic: Replacing of dropped frames (drops) and SVP.
This script can automatically find and restore single drops and double-drops only.

Double-drop replacing demo:

http://www.svp-team.com/forum/misc.php?item=1809

You can download some video samples there wink

If you have interest I can rewrite this script from MVTools to SVP libs to get more quality.

3 Reply by Skaven252 (edited by Skaven252 18-11-2012 15:33:01)

  • Thanks: 3

Re: Fixing dropped frames in action cam footage? (for stereoscopic 3D)

Thanks for the quick reply! So there is a way to do it then. smile  How does it know which frames are missing? Can it use the audio track as the "measuring stick"?

Do you think there might be any problems caused by the fact it's not just one, but two videos that need to be consistent to each other?

I'm definitely interested, and thanks for your offer to help! But I'm quite a n44bie with AviSynth and haven't even downloaded and installed SVP yet. I guess I'll need to bash my head against this for a while first, just to get started.

  • MAG79
  • SVP developer
  • Offline

Re: Fixing dropped frames in action cam footage? (for stereoscopic 3D)

Skaven252
How does it know which frames are missing?
Automatically. By compare every two consequent frames.

Can it use the audio track as the "measuring stick"?
Not for now. Why you ask it? You told audio is perfect. Without any missed chunks.
Or I don't understand you. Please upload sample with drops and I will take a look to "out of sync" you are talking about.

Do you think there might be any problems...
I think we need to try script first. And only after that we can answer wink

It is not hard to start. Give me samples with drops I will help you.

5 Reply by Nick 3DvB (edited by Nick 3DvB 19-11-2012 00:20:43)

  • Thanks: 4

Re: Fixing dropped frames in action cam footage? (for stereoscopic 3D)

I think he wants to use the audio stream as a reference, to lock the positions of the "real" frames, because it is very important to keep the left and right eyes in sync for 3D.  It might be easier to do some tests with SVP, if he uses DirectShowSource in the avisynth script he is using for DeFish warping then SVP manager should intercept it automatically, it has an option for "variable frame-rate repair" so maybe that would work?  It is just an idea though, it is probably better to do it  in a separate step using the script you posted for MVTools 2.5.

PS - Have you had any time to think about my idea in the 3D support thread?

Please let me know if I can explain anything better, about how the Rift HMD works etc.

thanks

  • MAG79
  • SVP developer
  • Offline

Re: Fixing dropped frames in action cam footage? (for stereoscopic 3D)

As I know, any VFR video can be loaded in avisynth with conversion to CFR. And audio sync stay perfect. It is not the trouble.
I wonder if so huge soft as Vegas can't open it correctly.

7 Reply by Nick 3DvB (edited by Nick 3DvB 19-11-2012 00:33:09)

  • Thanks: 4

Re: Fixing dropped frames in action cam footage? (for stereoscopic 3D)

I don't think audio sync will be a problem, more that the camera view for each eye I will be in the wrong position / time, even small errors will cause  problems with the stereoscopic effect.  I agree that software like Vegas / FinalCut / Premier are not very robust about strange input formats, much better to fix it with something else first.

8 Reply by Skaven252 (edited by Skaven252 19-11-2012 06:27:14)

  • Thanks: 3

Re: Fixing dropped frames in action cam footage? (for stereoscopic 3D)

I have two .mov video files:
left camera video.mov - 29,951 fps; length 2:50,680
right camera video.mov - 30,928 fps; length 2:50,655

Both videos were recorded simultaneously  (started and stopped the same time) on two separate cameras. The cameras are identical models, but one was low on battery and kept dropping frames (for some reason this gives a higher frame rate value?  hmm ).

- They are the same length, and play in sync if inserted into a Vegas project (Vegas's timeline is source media framerate independent)
- BUT: They will not pair into a stereoscopic subclip because of too much framerate difference in the media files
- If I re-encode the files with VirtualDub to just simply force them into the same framerate (29,970), they will drift off sync because of the rate difference
- I can adjust the Undersample rate of the re-encoded off-sync video to get it to (almost) match with the other video, but a video with an undersample setting still cannot be paired - I would have to re-render it out of Vegas into an intermediate video file first ( = lossy).

I suppose I don't necessarily even need SVP to fix the video - all that needs to be done is to resample the video so that it stays the same length, but just has a correct framerate so that it pairs with the other clip. I can do this in Vegas: simply render out the video to an intermediate file. But presumably SVP would give a smoother result? (Does AviSynth have frame resampling options built in?)

The other problem is that Vegas Movie Studio only renders to AVC mp4, which I'm not sure if AviSynth can read - because I also need to apply a fisheye removal to the video. And only the DeFish plugin in AviSynth does it right (of all the available options I have tried).

  • MAG79
  • SVP developer
  • Offline

Re: Fixing dropped frames in action cam footage? (for stereoscopic 3D)

Skaven252
I have two .mov video files
Can you share it to me? Public or private. As you wish. If it is not top secret video of-course. wink

left camera video.mov - 29,951 fps; length 2:50,680
right camera video.mov - 30,928 fps; length 2:50,655
It is average framerate. You need to know base framerate to convert VFR to CFR without losing of quality and to save original speed.

They are the same length, and play in sync if inserted into a Vegas project
This is good.

They will not pair into a stereoscopic subclip because of too much framerate difference in the media files
I need to look to video.

If I re-encode the files with VirtualDub to just simply force them into the same framerate (29,970), they will drift off sync because of the rate difference
Something wrong with open or with conversion.

Does AviSynth have frame resampling options built in?
Yes. It does, see DirectShowSource(convertfps=true). It converts VFR to CFR.

I also need to apply a fisheye removal to the video
You can process video in avisynth without removing fisheye. I think it will be better to work with source than with Vegas (wrong/buggy) output. It gives better chance to repair drops.

  • Thanks: 3

Re: Fixing dropped frames in action cam footage? (for stereoscopic 3D)

Thanks!

Sure, I can send you the files for testing if you like. They're about 652 Mb in total, though, so we'll need to figure out a feasible way to pipe them over to you.

  • MAG79
  • SVP developer
  • Offline

Re: Fixing dropped frames in action cam footage? (for stereoscopic 3D)

It is interesting to me. smile

variants:
- via gmail.com
- via any cloud storage service
- peer-to-peer by uTorrent

  • Thanks: 3

Re: Fixing dropped frames in action cam footage? (for stereoscopic 3D)

Hmm... DirectShowSource? Alas, those movies are in QuickTime so I'm using QTSource. I suppose that one does not support convertfps=true?

The DeFish plugin can be added to the AviSynth script to follow after the SVF framerate conversion, I suppose? (then I can open this .avs in VirtualDub and re-encode it to an x.264 file for stereo pairing editing in Vegas)

  • Thanks: 3

Re: Fixing dropped frames in action cam footage? (for stereoscopic 3D)

Would WeTransfer.com work for you? It's pretty quick to use.

  • MAG79
  • SVP developer
  • Offline

Re: Fixing dropped frames in action cam footage? (for stereoscopic 3D)

Of course. I have sent my email to you in PM.

15 Reply by Skaven252 (edited by Skaven252 19-11-2012 06:56:25)

  • Thanks: 3
  • Thanks for the post: 1

Re: Fixing dropped frames in action cam footage? (for stereoscopic 3D)

MAG79
If it is not top secret video of-course.

Nope, it's just a test shoot, with a random street corner in Stockholm with some people passing by, staring at my weird camera setup (or me). smile

  • Thanks: 3
  • Thanks for the post: 1

Re: Fixing dropped frames in action cam footage? (for stereoscopic 3D)

Here's an AviSynth script I normally use to defish these videos (by opening this script in VirtualDub and saving to x.264 with x264vfw).

LoadPlugin ("C:\Program Files (x86)\Video\AviSynth 2.5\plugins\defish_mt.dll")
LoadPlugin ("C:\Program Files (x86)\Video\AviSynth 2.5\plugins\QTSource.dll")
LoadVirtualDubPlugin("C:\Program Files (x86)\Video\VirtualDub_1911\plugins\DeBlock.vdf","DeBlock", 0)

QTInput("video.mov", color=1, quality=100, mode=3, audio=1)
DeBlock(1, 1, 1, 1)
defish(fov=101, scaling="fitxy")
ConvertToYV12()

(color = 1 converts it to RGB32, which DeFish requires)

  • MAG79
  • SVP developer
  • Offline

Re: Fixing dropped frames in action cam footage? (for stereoscopic 3D)

Skaven252
those movies are in QuickTime so I'm using QTSource. I suppose that one does not support convertfps=true?
Yes. You are right. It just combines all frames together without real interframe delays. I'm not sure it is IMHO.

The DeFish plugin can be added to the AviSynth script to follow after the SVF framerate conversion, I suppose?
Do you mean VFR?
VFR=variable frame rate smile

Yes. You can combine filters in avisynth as you wish. In this case for example:
1. open with VFR->CFR conversion
2. apply DeFish
3. replace drops

  • Thanks: 3

Re: Fixing dropped frames in action cam footage? (for stereoscopic 3D)

SVF framerate conversion, I suppose?
Do you mean VFR?
VFR=variable frame rate

I mistyped "SVP" as "SVF", oops. What I meant was that just converting from VFR->CFR isn't enough; SVP has to be there to morph the dropped frames beautifully. As a part of the whole pipeline.

So I reckon the "3. replace drops" part is done by SVP, then? After VFR->CFR opening, how does SVP know which frames have been dropped and need to be interpolated?

  • MAG79
  • SVP developer
  • Offline

Re: Fixing dropped frames in action cam footage? (for stereoscopic 3D)

Skaven252
"3. replace drops" part is done by SVP, then?
Yes.

After VFR->CFR opening, how does SVP know which frames have been dropped and need to be interpolated?
By straight comparing each current frame to next frame.

  • MAG79
  • SVP developer
  • Offline
  • Thanks: 1104
  • Thanks for the post: 1

Re: Fixing dropped frames in action cam footage? (for stereoscopic 3D)

First results of analyze MOV-(gdsmux)->MKV-(mkv2vfr)->VFR
wide ipd test cam1 t02.vfr:

# timecode format v1
Assume 23.976
0,552,29.9518
553,553,31.2500
554,1105,29.9512
1106,1467,29.9471
1468,1468,31.2500
1469,4109,29.9498
4110,5111,29.9516

wide ipd test cam2 t02.vfr:

# timecode format v1
Assume 23.976
0,2106,29.9498
2107,2107,14.9254
2108,2341,29.9501
2342,2342,31.2500
2343,3873,29.9497
3874,3874,31.2500
3875,4266,29.9511
4267,5109,29.9478

As you can see base framerate is not constant (intervals with bolded values). It is from 29.9471 to 29.9518 fps.
I think it is not precisely (maybe) because standard is 29.97 fps. I will make another analyze to confirm it.

  • Thanks: 3

Re: Fixing dropped frames in action cam footage? (for stereoscopic 3D)

Oh. So what happens when you VFR->CFR? Does it still maintain video length, as in, if there are dropped frames, it will duplicate previous frames to fill the gaps, so that the length of the video stays the same?

I'm asking because I'm a curious n44bie, iz all. smile

  • Thanks: 3

Re: Fixing dropped frames in action cam footage? (for stereoscopic 3D)

Thanks!

MAG79
First results of analyze MOV-(gdsmux)->MKV-(mkv2vfr)->VFR

Hmm. Oh. So AviSynth alone isn't enough, looks like there are a lot of new steps required. Sounds like a lot of work... thanks for your time and trouble!

  • MAG79
  • SVP developer
  • Offline

Re: Fixing dropped frames in action cam footage? (for stereoscopic 3D)

Skaven252
what happens when you VFR->CFR?
it will duplicate previous frames to fill the gaps, so that the length of the video stays the same?
Exactly wink

It is not all. I'd like to understand what is inside MOVs. Constant framerate by intervals with multiplied index from base framerate or real fluent variable framerate.

  • Thanks: 3

Re: Fixing dropped frames in action cam footage? (for stereoscopic 3D)

Wow.. the framerate drops down to even 14.9254 at one point, that's quite a gap. But SVP should be able to handle it just fine & dandy, right? smile

  • Thanks: 3

Re: Fixing dropped frames in action cam footage? (for stereoscopic 3D)

Note to self: to avoid this problem in the future, I should not only ensure the batteries are fully charged, but also shoot at a lower bitrate/quality. I'm currently shooting at maximum bitrate, but I guess it brings little visual advantage, while really pushing the SD card to its limits.

Also, I better check what the speed class of the SD cards I'm using actually is...