Training/inference pipeline: tracked gaps from sleap-nn CLI/config audit

[gitttt-1234]

Checklist from an audit comparing sleap-app's training/inference pipeline against sleap-nn's actual CLI/config surface and the legacy ../sleap GUI. Same pass that caught and fixed the invalid --anchor_part inference flag (#... see recent history) and the "entire video" LabelsProvider bug (talmolab/sleap#2848 parity).

Training

• (@gitttt-1234 ) "Resume training" / "Reuse model" options are non-functional — training always runs from scratch regardless of selection. hp.trainingMode only locks UI fields (TrainingConfigDialog.tsx); resume_ckpt_path is never set anywhere in trainingStore.ts. Either wire it up for real or remove the misleading radio options. - feat(training): wire up Resume training and Fine-tune modes #305
• (@gitttt-1234) auto-update feature (3 channels: stable, latest, dev) - feat(updater): stable/latest/dev auto-update channels + Environment badge #314
• (@gitttt-1234) No LR scheduler control — sleap-nn supports reduce_lr_on_plateau, step_lr, cosine_annealing_warmup, linear_warmup_linear_decay; trainer_config.lr_scheduler is never set. - feat(training): add LR scheduler, checkpoint retention, and eval-metric controls #342
• (@gitttt-1234) No checkpoint retention control (save_top_k / save_last) — no way to also keep last.ckpt alongside the best checkpoint. - feat(training): add LR scheduler, checkpoint retention, and eval-metric controls #342
• (@alicup29 ) WandB integration is minimal — only use_wandb/entity/project exposed out of ~13 WandBConfig fields (no viz logging, offline mode, run grouping, etc.). — 🔧 feat(training): WandB offline mode + API key field #312 (draft): adds offline mode + a password-masked API Key field with mode-aware UX. (Several fields were already wired since the audit: save_viz_imgs_wandb/prv_runid/group.) Now also includes auth-status detection (native check_wandb_auth: env WANDB_API_KEY + cached ~/.netrc → green "Authenticated — API key optional", à la PyQt; desktop-only).
• (@gitttt-1234) No epoch-end evaluation config (EvalConfig) — no live OKS/PCK/centroid-distance metrics during training, only post-hoc. - feat(training): add LR scheduler, checkpoint retention, and eval-metric controls #342
• (@alicup29 ) No multi-GPU device/strategy control — trainer_device_indices (pick specific GPUs) and trainer_strategy (ddp/fsdp) aren't exposed. — ✅ feat(training): multi-GPU strategy dropdown (trainer_strategy) (#288) #335: added a trainer_strategy dropdown (auto/ddp/fsdp) in the Full Configuration Performance row (serialized to trainer_config.trainer_strategy, machine-specific reset-on-import). trainer_device_indices (GPU-index picker) deferred — needs a new GPU-enumeration command + is CUDA-only.
• No instance-size-distribution visualizer — legacy sleap/gui/learning/size.py equivalent; the receptive-field preview is already ported but this one isn't.
• (@alicup29 ) Online hard-keypoint-mining is half-ported — missing hard_to_easy_ratio / loss_scale, easy finish since the rest of the plumbing already exists.
• (@gitttt-1234 ) Auto-compute max_stride and crop size from the loaded project's instance sizes (like sleap-nn's config-picker's avgAnimalSize thresholds), instead of the current static default + manual medium/large-RF profile pick — crop size needs to account for rotation (and scale) augmentation padding so a rotated animal doesn't get clipped, and ModelStatsPreview's crop preview needs to reflect the auto-computed value. - feat(training): auto max_stride/crop_size recommendations + GPU/cache memory estimates #334
• (@gitttt-1234) Add better names for the baseline default training configs - feat(training): wire up Resume training and Fine-tune modes #305
• (@alicup29 ) Full training configuration: manually changing the crop size for centered-instance doesn't update the crop visualization at the top — ModelStatsPreview should react to the manual value, not only the auto-computed one (cf. the max_stride/crop auto-compute item above). — ✅ feat(video): desktop legacy-codec transcode fallback (Xvid/WMV/MPEG → H.264) #307 (draft): ModelStatsPreview now uses the manual hp.cropSize (falls back to auto only in Auto mode), so the crop number + drawn box track edits live.
• (@alicup29 ) The training tab's error output should have a Copy button (if not already) so users can grab the stack trace / diagnostics in one click. — ✅ feat(video): desktop legacy-codec transcode fallback (Xvid/WMV/MPEG → H.264) #307 (draft): Copy button added to ErrorOutput (with a brief "Copied" confirm).

Inference

• (@alicup29 ) No exported-model (ONNX/TensorRT) support anywhere — no sleap-nn export UI, and --runtime (onnx/tensorrt) is never emitted by buildInferenceArgs. — ✅ feat(inference): ONNX/TensorRT runtime selection (--runtime) (#288) #337 (inference Runtime dropdown auto/onnx/tensorrt + --runtime emission, TensorRT gated to CUDA) + feat(export): in-app ONNX/TensorRT model exporter + on-demand install (#288) #338 (in-app Export→ONNX/TensorRT dialog via sleap-nn export + on-demand [export] install + post-training entry point). End-to-end train→export→onnx-infer not yet desktop-verified.
• (@gitttt-1234) No inference-time confidence/visibility result filtering — --filter_min_visible_nodes, --filter_min_visible_node_fraction, --filter_min_mean_node_score, --filter_min_instance_score have no UI/CLI wiring (only the overlap filter is exposed). Not a regression — legacy sleap doesn't have these either, it's unclaimed new sleap-nn capability. - feat(inference): expose full sleap-nn tracking + post-processing filter params #309
• (@gitttt-1234) Add params for tracking (only "method" is exposed in the inference config window) - feat(inference): expose full sleap-nn tracking + post-processing filter params #309
• (@alicup29 ) "Existing predictions" Replace / Clear-all is inert — stale predictions are never removed. The existingPredictions setting (Training dialog, default replace) is threaded into startTraining but never read; all three merge-backs (inferenceStore random path, loadAndMergeResults, trainingStore.mergeOutputSlp) call MergePredictions with no strategy, so it defaults to "auto" (keeps non-overlapping old predictions). Re-running inference therefore stacks duplicates — repro sleap-app-tutorial/sleap-tutorial-data/labels.v001.slp has 23/50 frames with 4 predicted instances on a 2-fly video (old 2 + new 2). Also: regular inference has no existing-predictions control at all, and io's Labels.merge only visits frames present in the new output, so even replace_predictions leaves stale predictions on frames the new run didn't cover → Clear-all needs a project-/video-scoped DeleteAllPredictions before merge.

Data I/O

• (@alicup29 ) Support .avi / .mpeg / .mj2 (and .wmv) in Add Video. Collaborators can't select .avi in the Add-Video picker and have to pre-convert with sleap-io first (sio reencode in.avi -o out.mp4). Today SUPPORTED_VIDEO_EXTS (src/lib/resolveVideos.ts) is mp4/webm/mkv/mov/ogg/ogv/ts/seq and .avi is intentionally excluded ("no sleap-io.js backend decodes it") — so closing this needs transcode-on-import (desktop: shell out to sleap-io/ffmpeg on add), not just widening the file filter.
  • Draft PRs (io-gated) 🔧: in-app decode for .avi/.wmv (H.264/MJPEG, browser and desktop, no sio reencode pre-convert) via AviVideoBackend — feat(video): AviVideoBackend — in-browser .avi/.wmv decode (web-demuxer + WebCodecs/ImageDecoder) sleap-io.js#252 + app wiring feat(video): in-app .avi/.wmv labeling (sleap-io.js AviVideoBackend) #306 — plus a desktop ffmpeg-sidecar transcode for codecs WebCodecs can't decode (Xvid/DivX = MPEG-4 ASP, WMV3/VC-1, MPEG-1/2, 10-bit HEVC) and .mpeg/.mpg — feat(video): desktop legacy-codec transcode fallback (Xvid/WMV/MPEG → H.264) #307: a bundled ffmpeg converts once to a frame-exact H.264 MP4 cached in the app cache dir (frame count preserved 1:1 so labels stay aligned), with an opt-in "temporary copy for viewing — original file + .slp unchanged" prompt, live progress + cancel, and a Clear-cache menu. So legacy codecs convert automatically instead of a manual pre-convert. .mj2 (Motion JPEG 2000) still open — WebCodecs/web-demuxer don't decode JPEG 2000; route via the feat(video): desktop legacy-codec transcode fallback (Xvid/WMV/MPEG → H.264) #307 ffmpeg path or an opt-in @cornerstonejs/codec-openjpeg branch. (detail in comment below)

UI/UX

Noticed during a tutorial-session passthrough (@alicup29).

• (@alicup29 ) Overlay text (floating keypoint labels, seekbar-header frame markers, etc.) goes blurry after zoom — scale canvas text to devicePixelRatio and rescale reactively on window/zoom resize so it stays sharp (DPI-correct text). — header frame-markers ✅ fix(ux): tutorial-session UX batch (6 fixes) — magnifier, predicted marks, epoch count, suggestions scroll, header DPI, training freeze #295 (rescale on resize); keypoint-label blur still open (needs screen-space text rendering)
• (@alicup29 ) have multiple panel selection ON by default (or have draggable tabs ??) — ✅ feat(ux): OS-correct shortcut labels, multi-panel default, labels.vNNN.slp auto-versioning (#288) #302 (sidebarMultiPanel defaults on)
• (@alicup29 ) Skeleton builder: should clarify what the intent is (not detailed labeling yet), doesnt matter where you put the nodes; Make removing nodes action clear -- want to minimize user fear of clicking on the wrong things
• (@alicup29 ) Auto-name labels.vNNN.slp + add versioning — follow PyQt SLEAP as the ground truth for the naming convention (auto-increment the .vNNN suffix on save; see legacy sleap's save/SaveProjectAs behavior). — ✅ feat(ux): OS-correct shortcut labels, multi-panel default, labels.vNNN.slp auto-versioning (#288) #302 (getNewVersionFilename; Save As bumps, Save overwrites; unversioned names start at .v001)
• (@alicup29 ) System-dependent command labels — show OS-correct modifier keys / shortcut hints throughout the UI (menus, tooltips, hint text): e.g. Ctrl → ⌘ Cmd, Alt → ⌥ Option on macOS, so hints match the platform the user is actually on. — ✅ feat(ux): OS-correct shortcut labels, multi-panel default, labels.vNNN.slp auto-versioning (#288) #302 (formatShortcut choke-point across menus, palette, shortcuts dialog, context menu)
• (@alicup29 ) Menu shortcut hints still show Alt on macOS — Alt+modifier accelerators in the menus should render as ⌥ Option (gap in the feat(ux): OS-correct shortcut labels, multi-panel default, labels.vNNN.slp auto-versioning (#288) #302 formatShortcut pass — some menu Alt-based hints aren't being converted). — ✅ feat(video): desktop legacy-codec transcode fallback (Xvid/WMV/MPEG → H.264) #307 (draft): added altKey (⌥ on macOS) to platform.ts; the four hardcoded Alt+… menu items now use it.
• (@alicup29 ) The "SLEAP crashed last time" recovery message doesn't go away — it should clear once acknowledged / after a clean session (the crash-recovery sentinel from feat(diagnostics): "Collect Diagnostics" — traceable local bundle for desktop testers #293 isn't being reset). — ✅ feat(video): desktop legacy-codec transcode fallback (Xvid/WMV/MPEG → H.264) #307 (draft): the toast was duration: Infinity with only a "Send diagnostics" action; added a Dismiss button (and it now dismisses on "Send diagnostics" too). Desktop-only path.
• (@alicup29 ) Add a view filter to hide predicted instances (or make predictions visually more distinct from user labels) — currently there's no toggle to hide predictions, and predicted vs user-labeled can be hard to tell apart at a glance. — ✅ feat(video): desktop legacy-codec transcode fallback (Xvid/WMV/MPEG → H.264) #307 (draft): no filter (per decision) — predicted edges now render dashed (on top of the existing thinner + dimmer styling) so predictions read as clearly tentative. Open to a different cue (opacity/tint/badge) if preferred.
• (@alicup29 ) window.confirm / alert / prompt are broken in the Tauri (desktop) WebView — they silently no-op, and one of the bypasses is a data-loss risk. The Tauri dialog shim routes these to a nonexistent dialog|confirm command (@tauri-apps/plugin-dialog v2.7.1 only ships message/open/save), so the call rejects and sync code like if (!window.confirm()) treats the rejected Promise as truthy → the guarded action runs anyway with no dialog ever shown. Worst case: unsavedGuard.ts (confirmDiscardUnsavedWork) is bypassed, so New / Open / Import discard unsaved work without prompting (silent data loss on desktop). Other broken callers: App.tsx:286, MenuBar.tsx:894 (confirm); TrainingConfigDialog.tsx:391/397 (alert); navCommands.ts:241 (rename node) + VideoPlayer.tsx:2248 (go-to-frame) (prompt). Fix: route confirm-callers through the in-app confirmDialog() promise helper (@/stores/confirmStore + ConfirmDialog, added in feat(video): desktop legacy-codec transcode fallback (Xvid/WMV/MPEG → H.264) #307); alert → a message/info variant; prompt → needs a new in-app text-input dialog (bigger lift). confirmDiscardUnsavedWork becomes async (~13 callers await it — all already in async fns, so typecheck catches any miss). Surfaced during feat(video): desktop legacy-codec transcode fallback (Xvid/WMV/MPEG → H.264) #307: the Clear-cache confirm was silently wiping the transcode cache with no dialog until it was switched to the in-app confirmDialog.

[alicup29]

Draft PRs for the Data I/O item (.avi / .mpeg / .mj2 / .wmv in Add Video):

Approach ended up better than transcode-everything-on-import — native in-app decode (no install, browser + desktop), with a desktop transcode fallback only for codecs the browser genuinely can't decode:

• feat(video): AviVideoBackend — in-browser .avi/.wmv decode (web-demuxer + WebCodecs/ImageDecoder) sleap-io.js#252 — AviVideoBackend (data layer): demux .avi/.wmv via web-demuxer (ffmpeg-wasm) → decode H.264/HEVC/VP9/AV1 through WebCodecs, MJPEG via createImageBitmap. Frame-accurate, zero external install, works in the browser too. (MJPEG-in-AVI is the SLEAP NWB / JABS container, so it's a first-class path.)
• feat(video): in-app .avi/.wmv labeling (sleap-io.js AviVideoBackend) #306 (stacked on io#252) — app wiring: adds .avi/.wmv to SUPPORTED_VIDEO_EXTS so they appear in the Add-Video / Locate / New-Project pickers (fixes the original "can't select .avi" complaint), routes them to AviVideoBackend, vendors the web-demuxer wasm. Browser E2E confirmed (H.264 + MJPEG decode + frame-accurate seek).
• feat(video): desktop legacy-codec transcode fallback (Xvid/WMV/MPEG → H.264) #307 (stacked on feat(video): in-app .avi/.wmv labeling (sleap-io.js AviVideoBackend) #306) — desktop transcode fallback for codecs WebCodecs can't decode (Xvid/DivX = MPEG-4 ASP, WMV3/VC-1, MPEG-1/2, 10-bit HEVC) and adds .mpeg/.mpg: a bundled ffmpeg sidecar converts once to a frame-exact H.264 MP4 cached in the app cache dir, then hardware-decodes it (frame count preserved 1:1 so labels stay aligned). Opt-in prompt ("temporary copy for viewing — your original file and .slp are unchanged"), live progress + cancel, and a "Clear Video Transcode Cache" menu. Desktop E2E confirmed on Xvid / WMV / MPEG-1 / MPEG-2.

Coverage vs this item:

• .avi / .wmv — ✅ native in-app decode (H.264 / MJPEG) + desktop transcode for legacy codecs.
• .mpeg / .mpg — ✅ desktop transcode (MPEG-1/2 aren't WebCodecs-decodable; the browser shows a "convert to H.264" message).
• .mj2 (Motion JPEG 2000) — ❌ not yet. WebCodecs and web-demuxer don't decode JPEG 2000, so it's out of scope for these PRs. Path if we need it: an opt-in @cornerstonejs/codec-openjpeg (~250 KB, MIT) decode branch in AviVideoBackend, or just route .mj2 through the same desktop ffmpeg transcode path (feat(video): desktop legacy-codec transcode fallback (Xvid/WMV/MPEG → H.264) #307 — ffmpeg decodes mj2). Happy to file as a follow-up.

Notably this does not require the sio reencode in.avi -o out.mp4 pre-convert for the common H.264/MJPEG .avi case (decodes in-app), and the legacy-codec fallback is automatic + cached rather than a manual pre-convert step.

Status: all three are draft and io-gated — they turn green / are mergeable once sleap-io.js#252 is published and the app's @talmolab/sleap-io.js pin is bumped to that release.