Multicam Support

[eric-with-a-c]

Are there any plans for adding a schemadef for multicam clips, or is the preferred direction to just use a stack to represent multicam clips?

The downside to using a stack is that if you round trip through OTIO back to an NLE you would lose the multicam clips and they would just become nested tracks.

[jminor]

Yes, we would like to add a schema for this, but we don't have it on the schedule. Would you like to propose a schema for it?

[ssteinbach]

Just to be clear, is the assumption with a multicam item that:

• each 'take' referenced by the multiclip is the same duration and has the same start time?
• that each 'take' referenced by the multiclip is of the same 'kind'? (they're all Video or all Audio)
• there is one 'primary' or 'pick' or 'select' that can be used by the adapter if it doesn't support multicam clips

Could this also be used for stereo pairs?

We've discussed two different ways this could be expressed:

• As a media reference with multiple targets
• as a composition with multiple clips in it

Definitely open to suggestion on this if you have any ideas about what makes sense to you!

[eric-with-a-c]

@ssteinbach before I jump in with answers to things you may not be asking, when you refer to a 'take' are you referring to each piece of media that's included inside the multicam clip?

[ssteinbach]

@ssteinbach before I jump in with answers to things you may not be asking, when you refer to a 'take' are you referring to each piece of media that's included inside the multicam clip?

Yup! Sorry about that. Thats exactly what I meant.

[jminor]

Incidentally, I was just looking at how AAF handles this via their concept of a Selector. It is an object that has one or more "alternates" of which, zero or one is "selected". In the OTIO schema, this would be a subclass of Item, and a peer with Clip. We should compare this with how FCPXML models a multicam clip. Is it more like a sub-composition, or a special kind of clip?

[eric-with-a-c]

For multicams in FCP X there are two pieces. The first is the multicam clip itself which is a container for several pieces of media, both audio and video, each one being referred to as an angle (even if it's just audio). Each piece of media that make an angle typically have a different start and end time. The angles are synced/aligned using either timecode or audio. For syncing to work using audio there must be scratch audio in the video files.

This is what the clip looks like in the XML:

<media id="r9" name="Untitled Multicam Clip" uid="AjO7aKUbSFGPA/UPveBtQA" modDate="2019-02-21 18:27:35 -0500">
    <multicam format="r2" tcStart="0s" tcFormat="NDF">
        <mc-angle name="IMG_0023" angleID="5TZ2uW/AT16v4hs32EW++g">
            <gap name="Gap" offset="0s" duration="6100/3000s" start="3600s"/>
            <asset-clip name="IMG_0023" offset="6100/3000s" ref="r3" duration="30400/3000s" audioRole="dialogue" tcFormat="NDF"/>
        </mc-angle>
        <mc-angle name="IMG_0233" angleID="0ZqhPpuoSiWU1Z0lXyOHbQ">
            <asset-clip name="IMG_0233" offset="0s" ref="r4" duration="135600/3000s" audioRole="dialogue" tcFormat="NDF"/>
        </mc-angle>
        <mc-angle name="IMG_0268" angleID="ctboIx5cRdKIGi8EEDDYxQ">
            <gap name="Gap" offset="0s" duration="3200/3000s" start="3600s"/>
            <asset-clip name="IMG_0268" offset="3200/3000s" ref="r5" duration="110600/3000s" audioRole="dialogue" tcFormat="NDF"/>
        </mc-angle>
        <mc-angle name="IMG_0513" angleID="erFvoohVTVe2JSOXweZCaA">
            <gap name="Gap" offset="0s" duration="9800/3000s" start="3600s"/>
            <asset-clip name="IMG_0513" offset="9800/3000s" ref="r6" duration="31200/3000s" audioRole="dialogue" tcFormat="NDF"/>
        </mc-angle>
    </multicam>
</media>

The second part is the actual use of the multicam clip. So if you put the multicam clip on the timeline, and play through it, you can cut between the different angles. Each of these cuts is represented in the XML by referring to the multicam clip and identifying which angle should be used, what its start time is, what its offset is, and if the audio, the video, or both should be used from that angle.

<project name="Untitled Project" uid="2A71E7F4-5F65-4F7C-86CE-F65A0E915CF3" modDate="2019-02-21 18:27:35 -0500">
    <sequence duration="135600/3000s" format="r2" tcStart="0s" tcFormat="NDF" audioLayout="stereo" audioRate="48k">
        <spine>
            <mc-clip name="Untitled Multicam Clip" offset="0s" ref="r9" duration="3300/3000s">
                <mc-source angleID="5TZ2uW/AT16v4hs32EW++g" srcEnable="all"/>
            </mc-clip>
            <mc-clip name="Untitled Multicam Clip" offset="3300/3000s" ref="r9" duration="12400/3000s" start="3300/3000s">
                <mc-source angleID="0ZqhPpuoSiWU1Z0lXyOHbQ" srcEnable="all"/>
            </mc-clip>
            <mc-clip name="Untitled Multicam Clip" offset="15700/3000s" ref="r9" duration="10400/3000s" start="15700/3000s">
                <mc-source angleID="erFvoohVTVe2JSOXweZCaA" srcEnable="all"/>
            </mc-clip>
            <mc-clip name="Untitled Multicam Clip" offset="26100/3000s" ref="r9" duration="5800/3000s" start="26100/3000s">
                <mc-source angleID="0ZqhPpuoSiWU1Z0lXyOHbQ" srcEnable="all"/>
            </mc-clip>
            <mc-clip name="Untitled Multicam Clip" offset="31900/3000s" ref="r9" duration="8400/3000s" start="31900/3000s">
                <mc-source angleID="ctboIx5cRdKIGi8EEDDYxQ" srcEnable="all"/>
            </mc-clip>
            <mc-clip name="Untitled Multicam Clip" offset="40300/3000s" ref="r9" duration="10400/3000s" start="40300/3000s">
                <mc-source angleID="ctboIx5cRdKIGi8EEDDYxQ" srcEnable="all"/>
            </mc-clip>
            <mc-clip name="Untitled Multicam Clip" offset="50700/3000s" ref="r9" duration="9100/3000s" start="50700/3000s">
                <mc-source angleID="0ZqhPpuoSiWU1Z0lXyOHbQ" srcEnable="all"/>
            </mc-clip>
            <mc-clip name="Untitled Multicam Clip" offset="59800/3000s" ref="r9" duration="6300/3000s" start="59800/3000s">
                <mc-source angleID="0ZqhPpuoSiWU1Z0lXyOHbQ" srcEnable="all"/>
            </mc-clip>
            <mc-clip name="Untitled Multicam Clip" offset="66100/3000s" ref="r9" duration="19700/3000s" start="66100/3000s">
                <mc-source angleID="0ZqhPpuoSiWU1Z0lXyOHbQ" srcEnable="audio"/>
                <mc-source angleID="ctboIx5cRdKIGi8EEDDYxQ" srcEnable="video"/>
            </mc-clip>
            <mc-clip name="Untitled Multicam Clip" offset="85800/3000s" ref="r9" duration="9500/3000s" start="85800/3000s">
                <mc-source angleID="0ZqhPpuoSiWU1Z0lXyOHbQ" srcEnable="all"/>
            </mc-clip>
            <mc-clip name="Untitled Multicam Clip" offset="95300/3000s" ref="r9" duration="7900/3000s" start="95300/3000s">
                <mc-source angleID="0ZqhPpuoSiWU1Z0lXyOHbQ" srcEnable="audio"/>
                <mc-source angleID="ctboIx5cRdKIGi8EEDDYxQ" srcEnable="video"/>
            </mc-clip>
            <mc-clip name="Untitled Multicam Clip" offset="103200/3000s" ref="r9" duration="8300/3000s" start="103200/3000s">
                <mc-source angleID="0ZqhPpuoSiWU1Z0lXyOHbQ" srcEnable="all"/>
            </mc-clip>
            <mc-clip name="Untitled Multicam Clip" offset="111500/3000s" ref="r9" duration="24100/3000s" start="111500/3000s">
                <mc-source angleID="0ZqhPpuoSiWU1Z0lXyOHbQ" srcEnable="audio"/>
                <mc-source angleID="ctboIx5cRdKIGi8EEDDYxQ" srcEnable="video"/>
            </mc-clip>
        </spine>
    </sequence>
</project>

The start in the mc-clip is relative to the multicam clip, not the media that makes up the angle that's being referred to, if that makes sense.

Maybe this is a long of saying that multicam clips are both a special kind of clip and a kind of composition. It seems like we would need a special kind of media reference in OTIO that could group all the different angles together, with some kind of offset or gap to align them with each other, and a way of referring to a particular angle within that media reference from an OTIO Clip.

In case it's not obvious, I haven't thought too much about this and am just spitballing. 😂

[ssteinbach]

@eric-with-a-c when you say "So if you put the multicam clip on the timeline, and play through it, you can cut between the different angles." do you mean that the "active" clip is an animated parameter, or that the multicam clip behaves kind of like a nested track with cuts of clips inside of it?

Is the tcStart a time code in which the clip starts playing and the previous stops? Can you have multiple things playing at once?

[eric-with-a-c]

I would say that it's more like a nested track with cuts applied to it. That act of cutting between angles is very similar to switching between feeds of a live broadcast.

Yes, you could have multiple things playing at once. For example, you might be playing the video from one angle but the audio from another.

Just for posterity, here are the links to the Final Cut Pro X and Adobe Premiere Pro documentation on multicam editing.
https://support.apple.com/kb/PH12656
https://helpx.adobe.com/premiere-pro/using/create-multi-camera-source-sequence.html

Davinci Resolve also supports multicam, but all their documentation is in PDF format so you have to download it from their support site. The NLE I'm least familiar with is Media Composer (sorry), but I think they have a way of doing multicam work too, I just don't have access to any of that documentation.

[virokannas]

One way to visualize how FCXP handles multicam is to think of it as a separate subtimeline, and what gets presented is a "window" that has a start and end, and between them both the video and audio playheads might jump between tracks in that subtimeline. This jumping is the animated parameter, which gets presented as a "cut" but it's not a hard cut from a clip to another as far as the main timeline is concerned.

One way to present this at the top level is to flatten multicam clips and just separating these "angle cuts" into individual clips. That process, however, is not reversible.

As @eric-with-a-c mentioned, there can be up to two things (video and audio) playing at once inside the multicam clip. Not sure if that's the same for 5.1 audio or 3D or other things like that.

[jminor]

Thanks, this is super helpful. It sounds to me like this needs to be an OTIO Composition - like Stack and Track, which can be nested within each other, rather than just a special Clip. That will allow any sort of sub-composition inside. As to how we encode the animated cuts... we'll have to think about that a bit.

I'd like to learn more about how AAF does this (and any other relevant format that OTIO supports) so we can find a nice common ground that will facilitate interchange (if possible).

[virokannas]

Just to elaborate a little bit on the previous comment from over a year ago: the animated parameter would need to be separate for the audio and video components to fully support what FCPX is doing. I would imagine at this point that is the most complicated scenario it would need to accomplish.

To be able to round-trip back to fcpxml from OTIO, if it's an OTIO Composition, it would still need the necessary metadata to imply it's a multicam clip so that it can be reconstructed as one.

[russellaugust]

As far as I can tell (and from what Adobe has told me), Adobe does not support multicam in it's XMLs so I don't think there will be a way in it's current form to get multicam edits from Premiere -> OTIO. Even Extendscript doesn't truly support multicam.

[virokannas]

@russellaugust That sounds like a very API-specific limitation for Premiere. It does have multicam support - is this something that could be accessed using a binary plug-in? It would still be nice if OTIO had a schema for storing this information. There will probably always be specific things some NLEs can't accommodate, but in such cases even having a one-way mapping would at least enable working through OTIO.

To do a one-way mapping the multicam clips (both video and audio) could turn that animated parameter into in/out points for the underlying media clips and kick them out to a "flattened" track. If the original multicam info could be stuffed into a metadata field of some sort, at least the parts that weren't modified could be mapped back as-is?

I don't know how Avid handles multicam timelines, maybe someone has more insight on that? It would be great if the schema would at least accommodate all current multicam-capable NLEs. @eric-with-a-c already provided an FCPX example, maybe something similar could be posted for Media Composer?