Skip to content

Commit f0bc3d1

Browse files
committed
feat(camera): several cameras in the picture, with layout sections
Up to four recorded cameras can now be on screen together. Layout sections on the timeline choose a template — screen + small cameras, one camera full + small ones, two cameras side by side, or another camera full — and which camera goes into each place; the picture glides between sections. Each camera can be turned, mirrored, cropped and perspective-corrected (a homography calibrated once with four corner handles, optionally placed by detected ArUco markers from a printable sheet). - Compositor: a list of camera layers per frame instead of one webcam rect, planned in `camera_layers.rs` (glide and fade between neighbouring sections, incl. a direct glide to and from an adjacent Full Camera section), extra camera decoders opened only for cameras a layout shows (preview and export, a camera that ends early never shortens the clip), a homography and a transparency lane in `LayerCB` (HLSL, WGSL, MSL), the corrected camera cover-fitted into its box. - Editor: Full Camera regions stay as they are (camera 1, desk view); every other layout lives in `cameraLayoutRegions`, clip-anchored like Full Camera regions, in the same timeline lane with no overlap across the two lists. "Add layout" menu, a layout inspector (template, camera per place, reset windows), PiP places movable and resizable in the preview, a "Cameras" section with per-camera settings, and a calibration dialog (perspective and crop) with a loupe, target format, margin and a live rectified preview. Marker detection uses js-aruco2 (MIT; the OpenCV 4x4 dictionary codes are BSD-3, both listed in THIRD-PARTY-NOTICES.md). - Every change is one undo step; projects without layout sections render exactly as before.
1 parent 0857621 commit f0bc3d1

158 files changed

Lines changed: 14249 additions & 663 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎THIRD-PARTY-NOTICES.md‎

Lines changed: 46 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,9 @@ application resources and satisfies the attribution and source-offer obligations
66
that come with them.
77

88
npm dependencies are not listed here: they are resolved from `package.json` and
9-
distributed by their own registries, not redistributed inside our binaries.
9+
distributed by their own registries, not redistributed inside our binaries. The
10+
exception is code bundled into the renderer whose licence asks for its notice in
11+
the distributed form (js-aruco2 below): the bundler strips the source headers.
1012

1113
---
1214

@@ -172,6 +174,49 @@ distributed by their own registries, not redistributed inside our binaries.
172174
notice as OpenScreen's own [LICENSE](LICENSE). Published by Recordly under MIT,
173175
before its relicensing in March 2026.
174176

177+
## js-aruco2 — ArUco marker detection
178+
179+
- **Components**: `src/aruco.js` and `src/cv.js` of the npm package `js-aruco2`
180+
2.0.0, compiled into the renderer bundle (`dist/assets/*.js`).
181+
- **Used by**: the camera calibration dialog, which finds the four printed corner
182+
markers in a camera still.
183+
- **License**: MIT — Copyright (c) 2020 Damiano Falcioni, Copyright (c) 2011 Juan
184+
Mellado, under the same permission notice as OpenScreen's own
185+
[LICENSE](LICENSE).
186+
- **Upstream**: <https://github.com/damianofalcioni/js-aruco2>, release 2.0.0.
187+
188+
## OpenCV — ArUco 4×4 dictionary codes
189+
190+
- **Component**: the marker codes in js-aruco2's
191+
`src/dictionaries/aruco_4x4_1000.js`, compiled into the renderer bundle. They
192+
are taken from OpenCV's `predefined_dictionaries.hpp`; the marker sheet the
193+
editor prints draws markers 0–3 of this dictionary.
194+
- **License**: BSD 3-Clause — Copyright (C) 2013, OpenCV Foundation, all rights
195+
reserved. Third party copyrights are property of their respective owners.
196+
197+
> Redistribution and use in source and binary forms, with or without
198+
> modification, are permitted provided that the following conditions are met:
199+
>
200+
> - Redistributions of source code must retain the above copyright notice,
201+
> this list of conditions and the following disclaimer.
202+
> - Redistributions in binary form must reproduce the above copyright notice,
203+
> this list of conditions and the following disclaimer in the documentation
204+
> and/or other materials provided with the distribution.
205+
> - Neither the names of the copyright holders nor the names of the
206+
> contributors may be used to endorse or promote products derived from this
207+
> software without specific prior written permission.
208+
>
209+
> This software is provided by the copyright holders and contributors "as is"
210+
> and any express or implied warranties, including, but not limited to, the
211+
> implied warranties of merchantability and fitness for a particular purpose
212+
> are disclaimed. In no event shall copyright holders or contributors be liable
213+
> for any direct, indirect, incidental, special, exemplary, or consequential
214+
> damages (including, but not limited to, procurement of substitute goods or
215+
> services; loss of use, data, or profits; or business interruption) however
216+
> caused and on any theory of liability, whether in contract, strict liability,
217+
> or tort (including negligence or otherwise) arising in any way out of the use
218+
> of this software, even if advised of the possibility of such damage.
219+
175220
## Rust crates — compiled into the compositor addon
176221

177222
- **Component**: `compositor_view.node`, under

‎crates/compositor-view-napi/src/lib.rs‎

Lines changed: 41 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ use openscreen_compositor::frame_geometry::FootageQuad;
1515
use openscreen_compositor::gif_export::{GifExportParams, GifStats};
1616
use openscreen_compositor::gif_export_control::{GifExportCancelled, GifExportControl};
1717
use openscreen_compositor::live::{LiveView, PausedPreviews};
18-
use openscreen_compositor::scene::Scene;
18+
use openscreen_compositor::scene::{Scene, SceneClipCamera};
1919
use openscreen_compositor::{config, pipeline};
2020
use std::collections::HashMap;
2121
use std::path::PathBuf;
@@ -368,6 +368,9 @@ pub fn present_time(id: i32, seconds: f64) {
368368
/// Remplace les sources du clip actif sans recréer la vue ni son thread de rendu. L'identité
369369
/// timeline et le playhead source sont atomiques avec le switch : deux clips partageant les
370370
/// mêmes fichiers restent distincts, et les deux décodeurs ouvrent directement la bonne frame.
371+
///
372+
/// `additional_cameras`: cameras 2-4 of the clip (index k-1 = camera k, an empty `path` = no
373+
/// camera in that slot). Absent = none; the view decodes only those its layout regions show.
371374
#[napi]
372375
pub fn set_active_clip(
373376
id: i32,
@@ -376,18 +379,33 @@ pub fn set_active_clip(
376379
webcam_offset_sec: f64,
377380
clip_index: u32,
378381
source_time_sec: f64,
382+
additional_cameras: Option<Vec<ClipCameraInput>>,
379383
) {
380384
if let Some(v) = registry().lock().unwrap().get(&id) {
385+
let cameras = additional_cameras
386+
.unwrap_or_default()
387+
.into_iter()
388+
.map(|c| SceneClipCamera { path: c.path, offset_sec: c.offset_sec })
389+
.collect();
381390
v.set_active_clip(
382391
&screen_path,
383392
&webcam_path,
384393
webcam_offset_sec,
394+
cameras,
385395
clip_index as usize,
386396
source_time_sec,
387397
);
388398
}
389399
}
390400

401+
/// An additional camera of a clip (= TS `CompositorClipCamera`).
402+
#[napi(object)]
403+
pub struct ClipCameraInput {
404+
pub path: String,
405+
/// Camera source time = screen source time - this.
406+
pub offset_sec: f64,
407+
}
408+
391409
/// Installe la scène de l'app (JSON `SceneDescription`) sur la vue : layout preset piloté par
392410
/// l'app au lieu de la fixture. JSON invalide → ignoré côté natif.
393411
#[napi]
@@ -504,6 +522,26 @@ pub struct ClipInput {
504522
pub webcam_offset_sec: f64,
505523
/// `false` évite une ouverture ffmpeg vouée à échouer et réserve du silence à ce clip.
506524
pub has_audio: bool,
525+
/// Cameras 2-4 of the clip (index k-1 = camera k, an empty `path` = none). Absent = none.
526+
pub additional_cameras: Option<Vec<ClipCameraInput>>,
527+
}
528+
529+
/// `ClipInput` → `pipeline::ClipSource`, shared by the MP4 and the GIF export.
530+
fn clip_source(c: ClipInput) -> pipeline::ClipSource {
531+
pipeline::ClipSource {
532+
screen: c.screen_path,
533+
webcam: c.webcam_path,
534+
source_start_sec: c.source_start_sec,
535+
source_end_sec: c.source_end_sec,
536+
webcam_offset_sec: c.webcam_offset_sec,
537+
has_audio: c.has_audio,
538+
additional_cameras: c
539+
.additional_cameras
540+
.unwrap_or_default()
541+
.into_iter()
542+
.map(|cam| pipeline::ClipCamera { path: cam.path, offset_sec: cam.offset_sec })
543+
.collect(),
544+
}
507545
}
508546

509547
/// The export's pixel size. The SHAPE belongs to the scene: `scene.output` is what the live
@@ -701,17 +739,7 @@ pub fn export_multi(
701739
params: Option<ExportParamsInput>,
702740
on_progress: Option<JsFunction>,
703741
) -> Result<AsyncTask<ExportMultiTask>> {
704-
let clips = clips
705-
.into_iter()
706-
.map(|c| pipeline::ClipSource {
707-
screen: c.screen_path,
708-
webcam: c.webcam_path,
709-
source_start_sec: c.source_start_sec,
710-
source_end_sec: c.source_end_sec,
711-
webcam_offset_sec: c.webcam_offset_sec,
712-
has_audio: c.has_audio,
713-
})
714-
.collect();
742+
let clips = clips.into_iter().map(clip_source).collect();
715743
Ok(AsyncTask::new(ExportMultiTask {
716744
out_path,
717745
clips,
@@ -882,17 +910,7 @@ pub fn export_gif(
882910
// Deliberately the same argument shape as `export_multi`: the caller builds
883911
// one clip list and one scene, and picks the container. Cursor comes from
884912
// the scene like every other effect — there is no GIF-specific input left.
885-
let clips = clips
886-
.into_iter()
887-
.map(|c| pipeline::ClipSource {
888-
screen: c.screen_path,
889-
webcam: c.webcam_path,
890-
source_start_sec: c.source_start_sec,
891-
source_end_sec: c.source_end_sec,
892-
webcam_offset_sec: c.webcam_offset_sec,
893-
has_audio: c.has_audio,
894-
})
895-
.collect();
913+
let clips = clips.into_iter().map(clip_source).collect();
896914
let gif_params = params
897915
.map(|p| GifExportParams {
898916
width: p.width,

0 commit comments

Comments
 (0)