Repository navigation
wip: CI snapshot for several cameras in the picture - #1
Closed
christian-wr wants to merge 229 commits into
Closed
christian-wr wants to merge 229 commits into
christian-wr wants to merge 229 commits into
Conversation
1.5 drew a 41 px arrow in a 1080p export and played down the cursor effects, which are part of the product. 1.95 draws 53 px. The 1.5 floor stays: reset on the size slider goes to the new default.
b69d121 left aspectRatio out of the patch and the active comparison, so two presets saved at different formats both read as active and switching between them wrote an identical patch. The format is back in both, and the menu checks at most one row. A new project still never takes the format from a preset.
The row being renamed is a name form, so it cannot carry the active mark: with a matching twin the menu showed a match and no check at all. The mark now skips it.
The named destinations were presets over the very controls the Advanced fold held, and covered less of them (no 720p, no codec). Drop both layers: format, quality, frame rate and codec sit in one always-open panel. Dead i18n keys removed across the 15 locales.
The switch renders no text of its own, so aria-pressed announced a state without ever saying which setting it was. Reuse the shared Toggle, which already takes an ariaLabel for exactly this, instead of the hand-rolled button. Reported by CodeRabbit on getopenscreen#883.
The reactive shrink followed the zoom depth, (1 + 1/zoom) / 2: 78% at the default 1.8x, 60% at 5x. It now drops to a flat 70% for the whole region, whatever the zoom level, easing in and out on the zoom's own envelope. ZoomState carries the region strength that envelope already computes, and reactive_scale reads it instead of the zoom scale.
A zoom region at scale 1 (a rotation-only region) shrank the camera to 70% while the screen did not move. The shrink now also requires a zoom above 1.
1.95 was a 30% bump over 1.5 and still left the size slider's handle near the left edge, which read as too small for effects that are part of the product. 3.6 sits at 47% of the 1.5-6 range, a 98 px arrow in a 1080p export.
…rate
A Logitech BRIO was recorded at 640x480 and 4 Mbit/s, then upscaled into the
picture-in-picture, which is what "the camera looks pixelated" actually was.
Nothing in the stack ever asked for a resolution. `getUserMedia({ video: {
deviceId } })` and a Media Foundation source reader handed a type with no
MF_MT_FRAME_SIZE both fall back to the device's DEFAULT media type, and for a
UVC camera that is the first format it enumerates. On a BRIO offering MJPEG up
to 3840x2160 that default is 640x480. The DirectShow fallback had the same hole
from the other side: RenderStream's intelligent connect takes the capture pin's
default because nothing ever touched IAMStreamConfig. Measured with ffmpeg on
the same device: unconstrained gives `rawvideo (YUY2), 640x480`, while
`-video_size 1920x1080` gives 1080p.
Naming the size on the reader's OUTPUT type is not enough on its own. With
MF_SOURCE_READER_ENABLE_VIDEO_PROCESSING the reader answers S_OK by inserting a
converter in front of whichever native type is already selected, so the camera
keeps running at its default and the size asked for is quietly dropped.
Selecting the native type first is what reconfigures the camera.
`chooseWebcamFormat` decides which of the advertised modes to drive, shared by
both backends and unit-tested against the BRIO's real capability list: only
modes fitting inside the target in both dimensions, most pixels wins, ties go to
the lowest frame rate that still reaches the target, and a camera whose every
mode is oversized scales down from its smallest rather than pushing the largest
through the encoder.
The capture pixel format is now NV12 rather than RGB32 wherever the frame only
has to reach the webcam encoder. RGB32 made the source reader decode AND convert
every frame: instrumenting the capture loop put ReadSample at 92ms per 3840x2160
frame -- a ceiling near 11 fps -- against 32ms for NV12. The file still claimed
30 fps because the encoder padded the gap with duplicates, so the loss was
invisible from the outside. The encoder wanted NV12 anyway and was converting
RGB32 back to it, so the old path converted twice to arrive where it started.
BGRA is kept for the inline composite, which needs that layout.
Capture resolution becomes a user setting (1080p / 1440p / 2160p), persisted
beside the camera device and validated at the write boundary. The encoder
bitrate ladder gains 1080p, 1440p and 2160p tiers; the old one stopped at
8 Mbit/s for anything 720p or larger, which starved the frames the higher
resolutions now produce.
Measured on a BRIO (Snapdragon X, alongside a screen capture), 8s takes, frames
the camera actually delivered out of 240, four runs each:
1080p 244 14.2 Mbit/s 4/4
1440p 242-244 21.9 Mbit/s 4/4
2160p 243-244 38.4 Mbit/s 4/4
Under heavy competing CPU load a 2160p take can still drop frames; one run
overlapping a lint pass produced 174.
The capture loop now counts what it did with everything it read -- delivered,
empty samples, short buffers, buffer failures, read failures -- and reports the
tally when it ends. A camera that produced nothing used to surface only as
MF_E_SINK_NO_SAMPLES_PROCESSED from the encoder's Finalize, the one place that
cannot say which of the loop's five silent `continue` paths swallowed the
frames.
The DirectShow fallback keeps delivering BGRA and is unchanged by construction,
but it could not be exercised here: the test camera is visible to Media
Foundation, so that path never runs.
The gear is disabled while recording, but a settings panel already open stays mounted, so its resolution options stayed clickable. `webcamQuality` is a dependency of the webcam acquisition effect: changing it re-runs that effect's cleanup, which stops every track of the live stream -- the same stream the browser, macOS and Linux paths hand to the webcam MediaRecorder. The camera ended partway through the take, without a toast or any other word to the user. Guarded with `controlsLocked`, as every other control in this component already is. The test mock for `useCameraDevices` gains a configurable device list: the panel only renders its camera controls when a camera exists, so a test reaching for them could not previously do so. Reported by CodeRabbit on getopenscreen#875.
The warm-up probe exists to tell "the camera is running" from "the camera is
open but has not produced anything yet". Its NV12 variant judged the Y plane
raw, and a camera's NV12 is studio-range: an all-black frame is a plane of 16,
whose average clears the average threshold on its own. Every black frame
therefore counted as a picture, and the no-visible-frame warning could not fire
for the case it was written for. The comment claiming `maxLuma > 24` covered
this was wrong -- it overlooked the `||`.
Y is now normalised from studio range onto 0..255, so the thresholds mean the
same thing here as they do for BGRA.
Both probes move out of main.cpp into frame_visibility.{h,cpp} with a unit test,
alongside audio_sample_utils and webcam_format. Reverting the normalisation
makes that test fail on exactly the reported case, and the packaging build runs
it.
Reported by CodeRabbit on getopenscreen#875.
…tream Two of the three webcam recorders were switched to a camera-derived bitrate; the browser pipeline kept `Math.min(videoBitsPerSecond, BITRATE_BASE)`. That value is the SCREEN recording's rate, so the camera's quality tracked whichever monitor was being captured and was capped at 18 Mbit/s — well under what a 1440p or 2160p frame needs now that the capture resolution is a choice. The rule now lives in one `webcamRecorderOptions` helper that all three call sites share, because repeating it inline is exactly how one of them was missed.
Media Foundation may pad every row out to a stride wider than the frame. `ConvertToContiguousBuffer` joins multiple buffers but leaves that padding in place, and the flat `Lock()` view cannot express it -- so taking the first width*height*3/2 bytes folds the padding into the picture and shears it. The length check did not catch it either: a padded buffer is larger than the tight size it is compared against, so it passed. `IMF2DBuffer` reports the real pitch, so each Y and UV row is now copied on its own. A buffer offering no usable 2D view, or a pitch narrower than the frame, falls back to the flat copy -- which is correct for every tightly packed driver and is what this code did before. RGB32 keeps the flat path unchanged; only NV12, added in this branch, goes through the 2D view. Verified against a Logitech BRIO at 3840x2160: 245 frames delivered over 8s at 37 Mbit/s, and an extracted frame shows no shear or colour shift. This camera reports a tight pitch, so the padded case itself remains unexercised here. Reported by CodeRabbit on getopenscreen#875.
…ppen `IMFMediaBuffer::Lock` already guarantees what the reverted commit went to IMF2DBuffer for. From the IMF2DBuffer remarks: "The Lock method returns a buffer that is guaranteed to be contiguous. If the underlying buffer is not contiguous, the method copies the data into a new buffer, and the Unlock method copies it back into the original buffer." Row padding therefore never reaches the first width*height*3/2 bytes, and the flat copy was correct all along. The reverted commit added a second native-layout path, and its bounds and lifetime surface, for a corruption mode the documented contract excludes -- and said so itself by recording the padded case as unexercised on hardware. Lock2D would still avoid the copy Lock may perform, but that is an optimisation needing a measurement, and there is none: the capture loop's store step measured 2.3ms per 2160p frame against 92ms in ReadSample. The same page also advises against mixing IMF2DBuffer and IMFMediaBuffer calls on one buffer, which the reverted code did on its fallback. The contract is now quoted at the Lock site so the next reader does not have to rediscover it. Verified after the revert: 3840x2160 at 30 fps, 246 frames delivered over 8s. Raised by @My-Denia on getopenscreen#875.
`fps_` held the rate that was asked for, never the one negotiated. `chooseWebcamFormat` deliberately settles for less when a camera offers nothing faster -- its own tests cover a 30 fps target picking a 24 fps mode -- and a driver may choose its nearest supported rate after SetFormat regardless. `fps()` feeds three things that all have to agree with the frames actually arriving: the `webcam-format` event, the webcam encoder's nominal rate, and the constant-rate write interval in main.cpp. A 24 fps camera asked for 30 was therefore encoded and paced as 30, and reported as 30 to the app. `resolveConnectedFormat` now reads AvgTimePerFrame back off the connected type -- the format the graph actually holds, after any adjustment the driver made -- and logs it when it differs from the request. Rewriting AvgTimePerFrame in the capability pass stays: naming a rate inside the range a capability advertises is the documented way to ask for one. What was missing was reading the answer. Raised by @My-Denia on getopenscreen#875.
The webcam quality PR added camQuality as a required RecordingPreferences member, so the complete typed preference fixtures in RecStage and the prefsRace tests need the product default (DEFAULT_WEBCAM_QUALITY, 2160p). The new HudDeviceSettings quality test needed groupId on its CameraDevice fixture and showMicrophone on the panel, whose overrides spread over a partial can only typecheck when the base covers every required prop. Fixes the failing Typecheck (tests) CI job.
The picker offered two codecs where one was the right answer and the other was a trap the UI itself had to warn about. H.265 is software-only on Linux, slower than software on the measured Macs (452 fps software against 303 through VideoToolbox at 1080p60), and the files half the players cannot open. The CLI already exported H.264 only and nobody asked for more. Also drop the idle plate that said to pick a format: it was written for the UI that hid the format toggle under Advanced, and stayed up through the save picker where it was simply false.
editor-shell.md still described the named destinations under Advanced, which getopenscreen#883 already removed. export-pipeline.md and native-compositor.md now say the dialog always sends h264 while the pipeline still encodes h265 internally, and the manual checklist drops both its destination steps and its H.264/H.265 one.
The English pages stop offering a codec choice and stop describing the named destinations that PR 883 removed. The comparison bullets that sold codec choice over Cap and Screen Studio go with it, since they would be false now. Translated side: 140 mechanical collapses of the H.264-or-H.265 pairs across 84 files, then the rewritten passages per locale. i18n:sync --accept records the new English strings.
The checks were rewritten but the section introduction still sent testers to four destinations under Advanced. Reported by CodeRabbit on getopenscreen#890.
…ender Star Sprout's sculpted arrow and hand now keep the navy outline of the 2D art as a physical rim, as in the new 3D reference (design/cursors/star-sprout/3d-reference.png). Every piece is a navy tray, a rounded navy bead along the stroke and a puffy colour cushion inside it. The arrow and the palm are polygons fitted to the source PNG. The glove gets navy ridges between its fingers and a mint cuff in its own frame; the star, now with oval eyes, and its two leaves stand in front of the body. The model is written in the three shaders, and sculpt.rs gives Star Sprout its own boxes (the star and the thumb reach past the shared ones) and puts its hotspot on the rim, like its PNG. A Linux render test checks that both models keep a navy share around the body colour and show the yellow star. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019NRvCeDSc9MDhL3yrfQwZK
…w sheet The new reference sheet (design/cursors/3d-concept.png) keeps the drawn outline on three themes, so the Star Sprout construction becomes shared: every piece is a tray and a rounded bead in the outline colour, with a puffy colour cushion inside. - Studio Ink: the arrow is a black rim, a raised ivory band and a recessed satin black field; the hand is an ivory glove in a black rim. - Pop Coral: a coral arrow in a navy rim with the yellow layer behind it and two yellow click dashes; a yellow glove in a navy rim with two coral dashes. The three arrows share one polygon and the three gloves one construction, each glove measured on its own drawing (RIM_POLY, RIM_GLOVE). sculpt.rs gives each rimmed model its own box and its hotspot on the rim. The crystal now casts the plain Studio Ink shadow instead of Pop Coral's, which carries the dashes. Prism Glow and Pixel Candy already match the sheet and are unchanged. The Linux render test now checks all three rimmed themes: outline colour, body colour and what they carry in front. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019NRvCeDSc9MDhL3yrfQwZK
The crystal hand was still the old faceted glove (capsule fingers, a long index). It is now six convex gems on one slab, after 3d-concept.png: an index column with a cut tip, three fingers side by side, the thumb and a triangulated palm, the fingers standing over the palm so their crease is the knuckle line. Both Prism Glow models are now plane tables (the arrow keeps its exact cut), written into the three shaders by scripts/generate-prism-glow-gem.mjs; a test checks the three tables agree. The crystal glows on its own, facets take deeper cyan, blue and violet tones from their orientation, and sharp edges light up (s_gem_glint). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019NRvCeDSc9MDhL3yrfQwZK
… after the sheet The last three sculpted cursors that had not been remodelled from the new reference sheet (design/cursors/3d-concept.png). - Pixel Candy: both models are voxel grids drawn after the sheet, with its bigger voxels (about 13 rows): a pink face, mint voxels down the lower-left edge, pale pink step tops, and a purple ring one voxel wide set a little behind the face. The hand gets the sheet's layout: the index, notches between the fingers, the thumb, the mint diagonal. The grids live in scripts/generate-pixel-candy-voxels.mjs, which writes the tables into the three shaders; the shadow uses their outline as rectangles, so the old arrow polygon and hand spans are gone. - Prism Glow arrow: recut like the hand, as convex hulls of hand-placed points: a wide girdle, a crown fanning out from one point, a chunkier tail. sculpt.rs gives every theme its own boxes and mirrors the voxel size; a test checks the voxel tables agree across the three shaders. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019NRvCeDSc9MDhL3yrfQwZK
…p Coral's hand its coral layer Prism Glow read as a realistic glass render next to the other themes. It now follows their approach: the 2D art given volume. The cut gems keep their shape, shrunk a little inside the navy outline of the drawing (the tray and bead of the other outlined themes, following the pieces' outline), and every facet is a flat colour of the drawing chosen by where it faces (cyan, pale cyan, violet, blue), lit by the shared lamp. The glass shading and edge glint are gone. The generator also writes the pieces' outline and the stroke, the hotspot sits on the stroke's edge like the other outlined themes, and the shadow comes from that outline. Pop Coral's hand gets the coral layer of its drawing, the glove shifted down-left behind the tray, like the arrow's yellow layer. The glove is evaluated in a two-pass loop rather than called twice, so FXC keeps one copy, and the layer is skipped above the glove's own tray. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019NRvCeDSc9MDhL3yrfQwZK
The silhouette of a modelled cursor was already anti-aliased by the ray march, but every inner edge (outline against body, facet against facet, voxel seams, tight bevels) was shaded from one ray per pixel and stepped. After the first hit, four probes half a pixel away in the tangent plane look for a change of material or a surface leaving the plane. Where they find one, three more rays are marched through the pixel and shaded with the first ray's occlusion and shadows; the pixel is their average. Every sample goes through one STAGE_SHADE, so model_shade keeps a single call site (FXC copies each call). On a large cursor this costs about half the cursor's previous cost in the software renderer, and nothing elsewhere. The rimmed-model test counts clean outline pixels, which blended edges no longer are: Pop Coral's hand threshold drops from 0.2 to 0.18. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019NRvCeDSc9MDhL3yrfQwZK
Three components the installers ship had no entry: the web-demuxer WebAssembly module (MIT, with LGPL FFmpeg code inside), the crates compiled into the compositor addon, and icons and helper code taken from Recordly while it was MIT licensed.
The cursor icons it listed are renditions of Apple's macOS system cursors, not Recordly's own work.
Snapshot of feat/multi-camera-layouts at 47de44b without internal design notes, pushed only to run CI on Linux and macOS.
Snapshot of feat/multi-camera-layouts at 8b94cad without internal design notes.
Snapshot of feat/multi-camera-layouts at 7a50735 without internal design notes.
Snapshot of feat/multi-camera-layouts at 75ee2b0 without internal design notes.
Snapshot of feat/multi-camera-layouts at ce00b74 without internal design notes.
Snapshot of feat/multi-camera-layouts at ba3c08b without internal design notes.
Snapshot of feat/multi-camera-layouts at ce64d52 without internal design notes.
Owner
Author
|
CI-only snapshot, superseded by getopenscreen#1025 and getopenscreen#1026. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CI-only draft inside the fork to compile and test the compositor on Linux and macOS. Not for review, not for upstream. Will be closed.