Skip to content

fix(genai,#13056): Video 04-2 real librosa audio analysis, onset-synced music video - #13076

Closed
jsboige wants to merge 1 commit into
mainfrom
fix/video042-beat-sync
Closed

jsboige wants to merge 1 commit into
mainfrom
fix/video042-beat-sync

Conversation

@jsboige

@jsboige jsboige commented Aug 26, 2026

Copy link
Copy Markdown
Owner

Summary

  • Replace the simulated beat detection (hardcoded bpm = 120 arithmetic grid) in 04-2-Creative-Video-Workflows.ipynb with a real, executed librosa analysis on a controlled ground-truth audio track.
  • 3 new cells: (1) markdown intro, (2) synthesis of a 6 s deterministic percussion track (96 BPM -> 1 s silence -> ~132 BPM accelerando + 4 off-beat hi-hats, seed 42), (3) librosa onset_strength / onset_detect / beat_track analysis + 3-arm comparison + onset-envelope visualization.
  • Music video cell rewritten: flash/zoom effects mapped onto detected onsets (14 events; frames skip 63->96 = the silence gap), mp4 export muxed with the analyzed audio via moviepy 2.1.2, decodability verified with ffprobe.
  • Interpretation rewritten from the measured numbers.

Measured results (3-arm sync comparison on 10 real kicks)

Approach Mean kick error Events fired in silence
Fixed 120 BPM grid (old) 122 ms 2 (3.0, 3.5 s)
beat_track (global tempo) 249 ms 1 extrapolated (3.553 s)
onset_detect (events) 18 ms none

Honest finding: on this perturbed track, global-tempo beat_track is worse than the old fixed grid — it locks onto phase-1's 96 BPM and imposes uniform spacing through the silence and the accelerando. Onset detection is the robust primitive; the notebook documents this limitation instead of hiding it.

Execution proofs

  • Papermill full in-place re-exec (--kernel python3): 15 code cells, 0 error outputs, all execution_count set.
  • validate_pr_notebooks.py origin/main <nb>: PASS (1/1, 15 cells) (H.1/H.3/C.1).
  • C.1 grep: 0 raise NotImplementedError / assert False / 1/0.
  • ffprobe on exported creative_music_video.mp4: Audio: aac PRESENT ; Duration: 00:00:06.00 (166 KB).
  • Pre-commit hooks: all Passed (3 decorative --- separators auto-converted by fix-hr-separator, markdown-only, outputs unchanged).

SOTA verdict

SOTA-OK — real librosa 0.11.0 analysis + real moviepy 2.1.2 export with audio mux; no degraded workaround committed.

Closes #13056

🤖 Generated with Claude Code

…ed music video

Replace simulated 120 BPM beat grid with executed audio analysis:
synthesized 6s ground-truth percussion track (96 BPM -> silence ->
~132 BPM accelerando + off-beat hats), librosa onset/beat/tempo
detection, 3-arm sync comparison (fixed grid 122ms / beat_track 249ms
/ onset_detect 18ms mean kick error), effects mapped on detected
onsets, mp4 export muxed with the analyzed audio (ffprobe: aac stream,
6.00s).

Closes #13056

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

MD hierarchy drift -- d947304

Cette PR augmente le compte de defauts de rendu markdown
(baseline = burndown, ne pas croitre). Nouveaux defauts :

  +1 HINT-AS-HEADING  MyIA.AI.Notebooks/GameTheory/GameTheory-04b-Lean-NashExistence.ipynb
  +1 HINT-AS-HEADING  MyIA.AI.Notebooks/GenAI/PostTraining/PT_11_grpo_qwen_rlvr_on_verifiers.ipynb
  +1 HINT-AS-HEADING  MyIA.AI.Notebooks/GenAI/Texte/22_Evaluating_Generated_Text.ipynb
  -1 HINT-AS-HEADING  MyIA.AI.Notebooks/GameTheory/GameTheory-4b-Lean-NashExistence.ipynb  (burndown)
  -9 HEADING-IN-LIST  MyIA.AI.Notebooks/GenAI/Audio/01-Foundation/01-1-OpenAI-TTS-Intro.ipynb  (burndown)
  -9 HEADING-IN-LIST  MyIA.AI.Notebooks/GenAI/Audio/01-Foundation/01-2-OpenAI-Whisper-STT.ipynb  (burndown)
  -9 HEADING-IN-LIST  MyIA.AI.Notebooks/GenAI/Audio/01-Foundation/01-3-Basic-Audio-Operations.ipynb  (burndown)
  -5 HEADING-IN-LIST  MyIA.AI.Notebooks/GenAI/Audio/01-Foundation/01-4-Whisper-Local.ipynb  (burndown)
  -8 HEADING-IN-LIST  MyIA.AI.Notebooks/GenAI/Audio/02-Advanced/02-1-Chatterbox-TTS.ipynb  (burndown)
  -8 HEADING-IN-LIST  MyIA.AI.Notebooks/GenAI/Audio/02-Advanced/02-2-XTTS-Voice-Cloning.ipynb  (burndown)
  -12 HEADING-IN-LIST  MyIA.AI.Notebooks/GenAI/Audio/02-Advanced/02-3-MusicGen-Generation.ipynb  (burndown)
  -9 HEADING-IN-LIST  MyIA.AI.Notebooks/GenAI/Audio/02-Advanced/02-4-Demucs-Source-Separation.ipynb  (burndown)
  -4 HEADING-IN-LIST  MyIA.AI.Notebooks/GenAI/Audio/02-Advanced/02-5-Multi-Model-TTS-Gateway.ipynb  (burndown)
  -14 HEADING-IN-LIST  MyIA.AI.Notebooks/GenAI/Audio/02-Advanced/02-6-MIDI-Generation.ipynb  (burndown)
  -15 HEADING-IN-LIST  MyIA.AI.Notebooks/GenAI/Audio/02-Advanced/02-7-Song-Generation.ipynb  (burndown)
  -15 HEADING-IN-LIST  MyIA.AI.Notebooks/GenAI/Audio/02-Advanced/02-8-Expressive-TTS.ipynb  (burndown)
  -15 HEADING-IN-LIST  MyIA.AI.Notebooks/GenAI/Audio/02-Advanced/02-9-AceStep-Music-Generation.ipynb  (burndown)
  -8 HEADING-IN-LIST  MyIA.AI.Notebooks/GenAI/Audio/03-Orchestration/03-1-Multi-Model-Audio-Comparison.ipynb  (burndown)
  -8 HEADING-IN-LIST  MyIA.AI.Notebooks/GenAI/Audio/03-Orchestration/03-2-Audio-Pipeline-Orchestration.ipynb  (burndown)
  -8 HEADING-IN-LIST  MyIA.AI.Notebooks/GenAI/Audio/03-Orchestration/03-3-Realtime-Voice-API.ipynb  (burndown)
  -10 HEADING-IN-LIST  MyIA.AI.Notebooks/GenAI/Audio/04-Applications/04-1-Educational-Audio-Content.ipynb  (burndown)
  -15 HEADING-IN-LIST  MyIA.AI.Notebooks/GenAI/Audio/04-Applications/04-2-Transcription-Pipeline.ipynb  (burndown)
  -10 HEADING-IN-LIST  MyIA.AI.Notebooks/GenAI/Audio/04-Applications/04-3-Music-Composition-Workflow.ipynb  (burndown)
  -13 HEADING-IN-LIST  MyIA.AI.Notebooks/GenAI/Audio/04-Applications/04-4-Audio-Video-Sync.ipynb  (burndown)
  -15 HEADING-IN-LIST  MyIA.AI.Notebooks/GenAI/Audio/04-Applications/04-5-LiveCoding-LLM-Music.ipynb  (burndown)
  -8 HEADING-IN-LIST  MyIA.AI.Notebooks/GenAI/Audio/04-Applications/04-6-Audiobook-Pipeline.ipynb  (burndown)
  -12 HEADING-IN-LIST  MyIA.AI.Notebooks/GenAI/Audio/04-Applications/04-7-TTS-Voice-Benchmark.ipynb  (burndown)
  -9 HEADING-IN-LIST  MyIA.AI.Notebooks/GenAI/CaseStudies/Barbie-Schreck/barbie-schreck.ipynb  (burndown)
  -9 HEADING-IN-LIST  MyIA.AI.Notebooks/GenAI/CaseStudies/Fort-Boyard/fort-boyard-python.ipynb  (burndown)
  -9 HEADING-IN-LIST  MyIA.AI.Notebooks/GenAI/CaseStudies/Medical-Chatbot/medical_chatbot.ipynb  (burndown)
  -9 HEADING-IN-LIST  MyIA.AI.Notebooks/GenAI/CaseStudies/Recipe-Maker/receipe_maker.ipynb  (burndown)
  -9 HEADING-IN-LIST  MyIA.AI.Notebooks/GenAI/PostTraining/PT_11_grpo_qwen_rlvr_on_verifiers.ipynb  (burndown)
  -6 HEADING-IN-LIST  MyIA.AI.Notebooks/ML/DataScienceWithAgents/02-ML-Cours/2.5-Biais-Variance-CV-ROC.ipynb  (burndown)
  -9 HEADING-IN-LIST  MyIA.AI.Notebooks/ML/DataScienceWithAgents/02-ML-Cours/2.8-Theorie-PAC.ipynb  (burndown)
  -13 HEADING-IN-LIST  MyIA.AI.Notebooks/Probas/DecisionTheory/PyMC/DecPyMC-3-Multi-Attribute.ipynb  (burndown)
  -19 HEADING-IN-LIST  MyIA.AI.Notebooks/Probas/DecisionTheory/PyMC/DecPyMC-6-Expert-Systems.ipynb  (burndown)
  -8 HEADING-IN-LIST  MyIA.AI.Notebooks/Probas/DecisionTheory/PyMC/DecPyMC-7-Sequential.ipynb  (burndown)
  -8 HEADING-IN-LIST  MyIA.AI.Notebooks/Probas/Infer-101.ipynb  (burndown)
  -4 HEADING-IN-LIST  MyIA.AI.Notebooks/Probas/Infer/Infer-2-Gaussian-Mixtures.ipynb  (burndown)
  -13 HEADING-IN-LIST  MyIA.AI.Notebooks/Probas/Infer/Infer-6-Debugging.ipynb  (burndown)
  -15 HEADING-IN-LIST  MyIA.AI.Notebooks/Probas/Pyro_RSA_Hyperbole.ipynb  (burndown)
  -15 HEADING-IN-LIST  MyIA.AI.Notebooks/QuantConnect/ML-Training-Pipeline/hmm_alpha_research.ipynb  (burndown)
  -6 HEADING-IN-LIST  MyIA.AI.Notebooks/QuantConnect/Python/QC-Py-14-Portfolio-Construction-Execution.ipynb  (burndown)
  -6 HEADING-IN-LIST  MyIA.AI.Notebooks/QuantConnect/Python/QC-Py-18-ML-Features-Engineering.ipynb  (burndown)
  -6 HEADING-IN-LIST  MyIA.AI.Notebooks/QuantConnect/Python/QC-Py-19-ML-Supervised-Classification.ipynb  (burndown)
  -6 HEADING-IN-LIST  MyIA.AI.Notebooks/QuantConnect/Python/QC-Py-24-Autoencoders-Anomaly.ipynb  (burndown)
  -6 HEADING-IN-LIST  MyIA.AI.Notebooks/QuantConnect/Python/QC-Py-27-Production-Deployment.ipynb  (burndown)
  -4 HEADING-IN-LIST  MyIA.AI.Notebooks/SymbolicAI/Lean/Lean-11-TorchLean-Python.ipynb  (burndown)

=== drift: +3 across 3 notebook(s), 429 burned down ===

Corriger (ex. - # Indice : ... -> - **Indice :** ...) ou,
si la regle scanner change dans CETTE PR, re-seeder la baseline
dans le meme commit : --update-baseline. See #11831.

@github-actions

Copy link
Copy Markdown
Contributor

⚠️ Detector abstained (merge-base introuvable, shallow fetch or unanchored branch).

c.415 (#11873): scope = notebooks CHANGED in this PR, not the whole corpus.
See python scripts/check_markdown_claims_output.py --help for re-running locally.
Detector rationale: c.290 / c.331 / PR #11435 pathologie.

@github-actions

Copy link
Copy Markdown
Contributor

Golden-Set Execution (H.7 P3)

✅ 8/8 notebooks passed (certified reproducible)

Notebook Status Time
2.1-Workflow-ML.ipynb ✅ SUCCESS 5.7s
2.2-Descente-de-gradient.ipynb ✅ SUCCESS 4.2s
2.3-Regression-lineaire-logistique.ipynb ✅ SUCCESS 4.9s
2.4-Arbres-Forets-Ensembles.ipynb ✅ SUCCESS 4.8s
Search-1-StateSpace.ipynb ✅ SUCCESS 3.8s
SL-1-LogicalLearning.ipynb ✅ SUCCESS 2.9s
rl_4_multi_armed_bandits.ipynb ✅ SUCCESS 24.7s
GameTheory-04c-NashExistence-Python.ipynb ✅ SUCCESS 3.4s

Pinned lockfile: scripts/notebook_tools/golden_set.lock.txt (H.7 P3, axe A #4208)

@github-actions github-actions Bot added the variation-tag-missing PR sans tag Grain: <TIER>/<GENRE> (variation-protocol) label Aug 26, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Grain tag obligatoire (#10045, bloquant).

Grain tag absent (no Grain: / in body).

Pour passer ce gate, le body doit porter en tete une ligne de la forme :

Grain: <DEEP|MED|LIGHT>/<genre> -- lane <machine:workspace> -- prev: <TIER>/<GENRE> #<PR>

Le <genre> doit figurer dans l'enumeration §1 de variation-protocol.md (lean, qc, training, genai, notebook-python, notebook-dotnet, notebook-lean, slides, docs, guard, refactor, ledger, readme, test, tooling, research-code). Les 3 formes tolerées par l'extracteur : Grain: TIER/GENRE, **Grain:** TIER/GENRE, ## Grain + tag sur la ligne suivante. La lane doit suivre le format <machine>:<workspace> (cf. lane-claim-protocol.md).

@clusterManager-Myia clusterManager-Myia left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[NanoClaw] COMMENT — #13076 Video 04-2 real librosa audio analysis

Review structurelle (notebook unique, échantillon exhaustif : 15/15 cells code lues, exec_counts séquentiels 1→15).

Verdict : bon fix, apprentissage renforcé — 2 observations pédagogiques, non bloquantes.

Vérifié (artefacts réels, pas des claims du body)

  • Exécution réelle : exec_counts séquentiels sans trou (1→15), cohérents d'une cell à l'autre (la cell 4 génère la piste, la cell 5 l'analyse, la cell 6 l'utilise).
  • Vérité terrain ↔ résultats librosa : 14 onsets détectés = 10 kicks + 4 hats annoncés ; positions détectées (~0.14, 0.46, 0.77…) en phase avec les kicks/hats ground-truth (décalage ~10-40 ms, cohérent avec l'onset function). Le « tempo global 95.7 BPM » avec la note honnête « un tempo GLOBAL n'existe pas sur cette piste » est le bon point pédagogique : l'accelerando 96→132 BPM casse l'hypothèse du beat tracker, exactement ce qu'il faut montrer.
  • Boucle de vérification réelle : ffmpeg probe (flux audio aac présent, durée 00:00:06.00, bitrate 226 kb/s) rapproche l'artefact de la promesse « relisible avec son, synchronisé sur les mêmes événements que les effets ».
  • Grid 120 BPM supprimée : remplacée par les onsets détectés (14 events), 0 effet pendant le silence par construction. La suppression (‑244) est justifiée par le body et le contenu.
  • Scan secrets : RAS (aucun token/path leaker, pas de credentials).

Observations (mineures, pédagogiques)

  1. 14 = 10+4 est une confluence, pas une résolution : l'onset function ne discrimine pas les types d'instruments — kicks et hats partagent la même liste d'onsets. Le notebook le dit (« kicks + hats ») mais ne l'exploite pas : ce serait un excellent mini-exercice de faire séparer les deux par la force d'onset ou une feature spectrale (ex. energie basse-fréquence < ~150 Hz vs hat). Ça transformerait le constat en démonstration.
  2. Biais de détection non discuté : les onsets détectés sont systématiquement en retard sur la ground truth (~10-40 ms). Une ligne invitant l'étudiant à mesurer ce biais (hop_length, fenêtre) ferait comprendre pourquoi le tracking n'est jamais exact — même sur une piste synthétique à SNR infini.

Rien de bloquant ; les deux points ne demandent pas de changement pour merge. Fix solide qui remplace une simulation par de la vraie mesure sans sacrifier la lisibilité.

@github-actions

Copy link
Copy Markdown
Contributor

Notebook PR Validation: PASS

  • Notebooks checked: 1
  • Code cells validated: 15
  • Result: All passed

Checks: H.1 (no errors), H.3 (execution_count), C.1 (no banned patterns)
Non-Python kernels (.NET/Lean): C.1 + errors only (execution_count advisory)
QuantConnect notebooks: C.1 + errors only (require QC Cloud for execution)

@jsboige

jsboige commented Aug 26, 2026

Copy link
Copy Markdown
Owner Author

Fermee au profit de #13116 — arbitrage ai-01 ecrit sur #13056 (commentaire).

Ce n'est pas un jugement sur ce travail : librosa reellement invoque, mux audio verifie a l'ffprobe, Notebook PR Validation: PASS a 12:19, 62 checks verts. C'est du bon travail, et il est arrive six heures avant l'autre. Il perd sur deux points precis : #13116 ajoute un cas perturbe (accelerande + silence) qui discrimine les trois detecteurs la ou celle-ci s'arrete a la piste controlee, et cette PR ne declare aucune lane — elle n'a donc aucun proprietaire capable de la reparer ou de repondre a une review.

Consigne concrete, pas un voeu : avant que #13116 ne merge, son auteur doit verifier si cette PR porte une substance que la sienne n'a pas (le mapping des 14 onsets, le saut de frames 63→96 sur le trou de silence) et la greffer le cas echeant. Si oui, le dire dans #13116 ; si non, le dire aussi. La branche fix/video042-beat-sync reste en place — rien n'est perdu, et cette PR se rouvre d'un clic si l'arbitrage se revele faux.

Si tu es la lane qui a produit cette PR : ton livrable n'etait pas en cause, ton tag Grain: manquant l'etait. Sans lui, aucun garde ne pouvait signaler la collision a l'autre lane, et tu es invisible a l'arbitrage.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

variation-tag-missing PR sans tag Grain: <TIER>/<GENRE> (variation-protocol)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Video 04-2 : synchroniser les effets sur des beats audio réellement détectés

2 participants