Repository navigation
fix(core): music keeps playing smoothly across video cuts in preview - #4939
Conversation
While no Web Audio buffer plays, the preview playhead follows an audio element. It followed the first one in page order, so a sub-composition whose shots each carry a short sound clip took the playhead away from the music bed. At every cut the next shot's sound starts late, the playhead waited for it, the music played on, and media sync then pulled the music back: a jump at each cut. The followed clip still keeps the playhead; otherwise the in-window clip that runs longest takes it.
…lip keeps it A bed whose file ends before its authored length no longer wins the pick and leaves the playhead on wall time. Pins the rule that a longer clip starting late cannot take the playhead from the followed one.
jrusso1020
left a comment
jrusso1020
left a comment
There was a problem hiding this comment.
Approving at ff4ae940.
The fix. With no clip being followed, the playhead now picks the in-window clip that runs longest, not the first one in page order. A clip it already follows keeps it, and an ended clip is skipped. I ran audioClockSource, clock, clock-stall, clock-drift and media at this head: 229 pass. Against main's init.ts, the new cuts test fails. I also deleted each of the two pick lines (the ended skip and the followed early return), and each deletion fails its own test, as the body says.
The real seeks you asked about all behave the same as on main:
- Scrubbing while playing.
clock.seekdoesn't drop the followed clip. A seek inside its window holds onel.seekingas before. A seek outside the window drops it at the window check, and the longest clip takes over. - Pausing. Pausing still detaches the source, so Play always starts a fresh pick.
- Clips that need resyncing. That logic is untouched: every clip the playhead doesn't follow still goes through
syncRuntimeMedia's hard, strict and force tiers. - Audio that starts mid-film. It can't take over from a clip that is already playing. Your "longer bed starts late" test covers this.
- The paused-but-ready case. It still detaches without trying the next clip, exactly as the old
breakdid.
Per-clip audio vs its own video. The cost of a late start moves from the music onto the shot sound. I recorded seeks in your cuts test: on main the runtime writes the music's currentTime, and at this head it writes the shot sounds' instead, pulling each one onto the playhead. The muted shot video follows the playhead, so each shot sound comes back into sync with its picture through the normal strict sync. It lags by its cold-start delay until then. On main, any shot sound placed after the bed in the page already got exactly this treatment, so it's not a new class of behaviour. If a shot ever carries dialogue, its first beat is where to listen. Your measurements only checked the music.
Non-blocking: a looping bed with no data-duration still loses the pick. resolveMediaElementDurationSeconds ranks it by one pass of its file, though its window is open-ended. I probed your cuts fixture with the bed as loop and no data-duration, a 2 s file, playing from 3 s. The playhead still holds 250 ms for each shot, the same as main. With data-duration="10" it holds 0 ms. Studio clips normally carry data-duration, so this is your call. The fix would be to rank a looping clip by its window end.
— Rames
80708ab to
ff4ae94
Compare
|
Correction to my review's non-blocking note on a looping bed with no — Rames |
Edit accuracy: accurate 1216 (base branch 1216), smooth 1064 of thoseThe gate passes. Quarantined, measured but not gated (1)
Unstable (1)
|
2 participants
What
In Studio preview, a music bed now plays straight through video cuts. Before, when each shot of a sequence carried its own short sound clip (the shot's camera audio, faded in and out), the music jumped at every cut: a short gap and a skip back of 60 to 900 ms, depending on how slowly the next shot's sound started.
Export was never affected: the rendered audio of the fixture below is byte-identical on main and on this branch, and its music has no jump.
Why
While no Web Audio buffer plays, the preview playhead follows an audio element (the clock's tier 2). It followed the first in-window
<audio>in page order. A sub-composition placed before the music in the page put its shot sounds first, so the playhead followed a 0.48 s shot sound instead of the music. At each cut the playhead moved to the next shot's sound, which starts late (seek, play, data). Since #4679 the playhead holds while the clip it follows catches up rather than stepping back, so it held for that delay. The music kept playing through the hold, drifted ahead of the playhead, and media sync then seeked it back. That seek is the jump you hear.Related work
Refs #4679, whose rule that the playhead holds for the clip it follows is what turned a late shot sound into a music jump. That rule stays.
How
init.ts: the clock's audio pick moves intofollowedOrLongestRunningAudio. The clip the playhead already follows keeps it, as before. Otherwise the in-window clip that runs longest takes it (startplusresolveMediaElementDurationSeconds, the existing owner of a media element's timeline length), not the first in page order. With a music bed in the window, the bed leads and a cut hands nothing over. A clip that has ended is never picked, so a bed whose file runs out early hands the playhead to a clip still playing. The buffering freeze and the never-step-back rule are unchanged, but whenever nothing is followed yet (at Play, after a seek out of the window, at a handover) they now apply to the longest clip rather than the first in page order: with a voice-over under a longer music bed, the playhead follows the bed and the picture waits for the bed if it is still loading, while the voice is the clip media sync corrects. A clip with nodata-durationranks as open-ended until its metadata loads.Test plan
Unit tests added/updated
Manual testing performed
Documentation updated (if applicable)
Comments follow CONTRIBUTING.md "Comments": they say why, not what, and a bug fix says what the code must do and how to reproduce the bug
audioClockSource.test.ts, new case: three 1 s shot sounds earlier in the page, each starting 250 ms late, and a 10 s music bed. Every frame the playhead must sit on the music's time, and the runtime must never write the music'scurrentTimeafter Play. It fails on main: the playhead lags the music by a frame from the start, and with that check removed main seeks the music from 0.30 s back to 0.05 s. Passes 3 runs in a row with the fix.Two more cases pin the pick: a longer bed that starts late cannot take the playhead from a playing voice it already follows, and a bed whose file ended early hands it to the clip still playing. Removing either line in the pick fails its case.
Core runtime suite: 66 files, 1746 tests pass with the fix.
tsc,oxlint,oxfmt, the comment checks and the dead-code audit pass.Measured in Studio on a Linux box with headful Chrome, playing across the cuts with every media event and the music element logged per frame:
Independent review
No blocking or major findings. Three minor ones, all taken: an ended bed could win the pick (now skipped, with a test), the "followed clip keeps the playhead" rule had no test (added), and the behaviour change when nothing is followed is now stated above. One nit kept:
&& !el.loopin the ended check is redundant, since a looping element never reportsended.Before
Studio on main (70dde41), a fixture built for this PR from generated test patterns and tones: ten 0.48 s shots in a sub-composition, each a muted
<video>plus its own sound clip with a 40 ms fade, under one music bed (a rising tone with a click every quarter second) that starts with the first shot. Play from 00:00.4 to 00:06.2. The readout compares the music element's time with the playhead and counts the music's seeks during playback: 10 seeks, worst gap 203 ms, and the clicks stumble at the cuts.cuts-before-70dde41b5.mp4
After
The same steps with this branch's change on the same main: 0 seeks, the music stays on the playhead, the clicks stay even.
cuts-after-70dde41b5.mp4