Repository navigation
feat(cli): audio answers beats, downbeats, bpm, sections and describe through a provider ladder - #5632
Conversation
… through a provider ladder Beats and downbeats try api, apple-mu, beat-this, then signal; sections try api, apple-mu, then signal; describe tries api, then apple-sound. bpm is read off whichever provider placed the beats. Each provider says whether it can run here and why not, and the ladder warns when it passes one over. The signal rung is a TypeScript tracker: log-band spectral flux, autocorrelation tempo with a log-normal prior, Ellis dynamic-programming beats with a windowed local tempo, a low-band bar phase, and Foote novelty sections snapped to beats. It reports no steady beat when the onsets do not repeat. The api rung posts the audio to HYPERFRAMES_AUDIO_API_URL with HYPERFRAMES_AUDIO_API_KEY and moves each answered time onto the nearest signal beat within a quarter beat, reporting what moved and what did not. beat-this runs Beat This! from a configured Python and local weights and never downloads. apple-sound runs macOS's sound classifier through a Swift helper. apple-mu reports that Music Understanding needs macOS 27. words are no longer accepted: they come from hyperframes transcribe.
…r provider ladder
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Automations to automatically generate PRs for you. |
Edit accuracy: accurate 271 (base branch 271), smooth 174 of thoseThe gate passes. Quarantined, measured but not gated (0) |
jrusso1020
left a comment
jrusso1020
left a comment
There was a problem hiding this comment.
Requesting changes at 5b8a8d09 for one correctness bug in signal sections. The ladder, the providers and the rhythm math look right to me.
Blocking: signal sections splits silent or flat audio into a new section every 4 s
In boundaryBlocks (packages/core/src/audioSections.ts:92-103), a flat novelty curve makes mean and spread both 0. Then neither curve[i] < max(neighbourhood) nor curve[i] < mean + spread ever skips a block, so every block minBlocks apart becomes a boundary. labels then gives each span a new letter: unit() of a zero vector is zero, its dot with any earlier span is 0, which is below SAME_SECTION.
Running sections() on 30 s of digital silence gives A 0–4, B 4–8, C 8–12, D 12–16, E 16–20, F 20–24, G 24–30. A steady 440 Hz tone correctly gives one span. Silent or near-silent stretches are common in real input: intros, padding, voice-only tracks. So hyperframes audio --sections (and the fallback any provider fails over to) would report structure that isn't there.
A small fix: return no boundaries when spread is 0 (or within an epsilon), or require curve[i] > mean. Then add a silence case for sections to audioRhythm.test.ts, which has one for beats already.
Checked, looks right
- Ladder (
audioAnalysisLadder.ts):- An
available()that throws becomesunavailable, and a failedanalyzeis recorded and falls through. - Every skipped step is listed in one warning, and no failure comes back looking like a success.
perSignalcaches the rejected promise, so beats and downbeats share one failed call.
- An
- API provider:
- The key goes only in the
authorizationheader, and the URL is used exactly as configured. Errors carry only the status and the service's message. - The default timeout is 120 s and also covers reading the body.
apiruns only when both env vars are set. It is excluded by--quality local, andfastleaves onlysignal, so nothing makes a paid call implicitly.
- The key goes only in the
- Subprocesses:
execFilewith argument arrays and no shell.- Temp WAVs are removed in a
finally. - Off macOS,
apple-mureturns before spawning anything. beat-thisloads its weights from a local path and never downloads.
- Rhythm:
- Empty, sub-6 s and silent input return empty beats with a reason.
- The beat backtrack always points strictly backward.
(k - phase) % meteris correct for negative values.
- Back-compat:
- The default capabilities reach only
signal. shiftedTimesmaps the same way the removedtoFileTimedid.- The only importers of
@hyperframes/core/audio-analysisare the CLI's own audio files. - The
available()signature change isn't in a release tag yet.
- The default capabilities reach only
- PR-body claims checked in source:
- One request for beats and downbeats, separate ones for sections and describe (asserted in
api.test.ts). - The
"no steady beat"/"too short for a steady beat"reasons. - Snapping within a quarter of the answer's median interval.
- One request for beats and downbeats, separate ones for sections and describe (asserted in
Non-blocking
apianswers are only loosely validated (providers/api.ts:58-80):- Beat times aren't checked to be ascending and non-negative. A descending list makes the median interval negative, which sends every beat to
unsnappedand givestempoOfa negative BPM (tempoOf([2,1.5,1,0.5])→-120). - A section without
labelpasses, though the docs saylabel: string.api.test.ts:150,164does exactly this, and the text output then printsundefined 6.250–12.000. describeis passed through whole.
- Beat times aren't checked to be ascending and non-negative. A descending list makes the median interval negative, which sends every beat to
HYPERFRAMES_AUDIO_API_TIMEOUT_MSset to a fraction or above 2³²−1 makesAbortSignal.timeoutthrowERR_OUT_OF_RANGE. The rung fails over, but the warning is confusing. Round and clamp atapi.ts:54-55.http://URLs are accepted, so the bearer key can go out in cleartext. Worth a warning.--provider bpm=Xis accepted bypins(commands/audio.ts:84) but ignored: bpm comes from thebeatsladder (analyze.ts:131). Reject it, or treat it asbeats=.- The upload buffer is copied an extra time:
new Uint8Array(encodeWav(signal))(api.ts:109).new Blob([wav])takes the Buffer as is. - Error messages from the service reach the terminal as is. A JSON
error.messagefrom the service isn't truncated (api.ts:84-86).
Not run: I couldn't run vitest locally (no node_modules in my checkout). I ran the core rhythm/sections code directly under Node 22's type stripping. Some CI jobs were still pending at this head. The API and Beat This paths are covered by stand-ins, as the PR says.
— Rames
|
Addressed the blocking silence finding in Validation: core rhythm suite 11 passed; core rebuilt; CLI analysis/API suites 11 passed; lint, format, comment gates, typecheck and fallow pre-commit checks passed. API coverage remains a local HTTP fixture. The separately marked non-blocking API contract suggestions are not changed in this correction. |
miguel-heygen
left a comment
miguel-heygen
left a comment
There was a problem hiding this comment.
Review at 4e45ab310058e60a8b07f49eb31482f49ba7bfc8. The silent-section successor adds a strict threshold and passes its regression. The shared ladder catches availability/run failures; subprocess WAV cleanup is in finally. I independently ran the five affected core/provider/analysis suites: 50 passed, one native macOS test skipped.
Blocker — an accepted provider pin can still upload audio (packages/cli/src/audio/analysis/analyze.ts:119-131). As Rames noted, --provider bpm=signal is ignored, but the consequence warrants fixing before merge: --only bpm --provider bpm=signal delegates to ladderFor("beats"), which reads only provider.beats, so a configured API still receives the audio. I exercised the actual analyzeSignal and actual API adapter against a loopback HTTP server with an 8-second synthetic signal: one multipart POST containing 353,257 bytes, provenance bpm: api, despite provider: {bpm: "signal"}. The beats=signal control made no additional request. This is a transport/selection witness, not a real backend call.
Reject unsupported BPM pins before any analysis/upload, or normalize them to the beat provider with explicit handling for conflicting beats=/bpm= pins. Add a no-request regression. The current silent-section repair does not change this path.
CI on the new head has just started; no all-green claim. No real backend, Beat This model, Apple native analysis, subjective music evaluation or Desktop/Studio run was performed. The benchmark figures remain author evidence.
— Magi (desktop-review)
Verdict: REQUEST CHANGES
Reasoning: A user-selected local provider is silently ignored and the API receives the signal. Preserve or reject that pin before any network request.
|
Fixed the provider-pin blocker in The actual API adapter against a local HTTP server proves zero requests for BPM-only and both beats/BPM orderings. Both conflicting pin directions also reject with zero requests. All five new cases failed before the fix; all 38 analysis tests pass afterward. Built CLI + ffmpeg probes confirm zero API requests for CLI build, lint, format, comment checks and all commit hooks passed. No paid/backend/model calls; HTTP coverage is a loopback transport fixture. Merged into child |
miguel-heygen
left a comment
miguel-heygen
left a comment
There was a problem hiding this comment.
Re-review at 54dfc960c4bcfb003f2a9f09fcdc570bfe638868. Supersedes my provider-pin change request.
packages/cli/src/audio/analysis/analyze.ts:106-108,124 now normalizes BPM and beats onto the same provider before analysis, and rejects conflicting pins before any capability runs. I reran the actual API adapter against a loopback HTTP server on this head: BPM-only and both beats/BPM orderings with bpm=signal made zero requests; both conflicting pin directions, with describe first, rejected with zero requests. Matching API pins made exactly one shared request, proving the transport witness is non-vacuous. My prior-head witness made an unwanted upload; the successor fixes it.
Independent focused tests: 55 passed, one native macOS test skipped. The complete successor diff is the shared-ladder pin correction plus five regressions; prior silence/beat-file fixes remain intact. No new own-code blocker found. All required checks now pass, including Windows render, build, typecheck and runtime contract. I read the completed aggregate accuracy log: 271/271 accurate, matching the measured base. One non-required regression shard remains running; no failure is reported, so this does not claim every check is finished.
Limits: synthetic loopback/provider tests, not a real backend, installed Beat This model, Apple native inference, subjective corpus evaluation, or live Studio/Desktop certification. The author’s benchmark and browser captures remain author evidence. No merge, queue, release or deployment performed.
— Magi (desktop-review)
Verdict: APPROVE
Reasoning: The ignored provider pin is fixed on this exact head with discriminating transport controls; current-head validation supports the code verdict.
jrusso1020
left a comment
jrusso1020
left a comment
There was a problem hiding this comment.
Approving at 54dfc960. My silence blocker from 5b8a8d09 was fixed at 4e45ab31, Magi's bpm= blocker is fixed at this head, and required CI on this head is green.
4e45ab31..54dfc960 changes only analyze.ts and api.test.ts:
analyzeSignalreadsbeatsandbpmpins up front. If both are set and differ, it throwsbeats and bpm must use the same providerbefore any ladder runs, so nothing is uploaded.- Otherwise the beats ladder runs on
beats ?? bpm. Sincebpmis always computed bytempoFromBeats(ladderFor("beats")), abpm=signalpin now keeps the whole tempo path local.--only bpm,--only beats,bpmand--only bpm,beatsall share the one beats ladder. analyzeSignalis the only place that hands pins torunLadder, so there's no second path that ignores them.- The new loopback tests configure the API and assert zero requests for each of the three capability orders. They also assert zero requests for both conflicting-pin directions, including when
describecomes first.
The silence fix (<= in boundaryBlocks, plus the it.each([1, 4, 8]) silence case) is unchanged from 4e45ab31.
Non-blocking, still open from my first review: loose validation of api answers (beat order and sign, a missing section label), the timeout env value, and http:// URLs.
Not exercised: I didn't run vitest locally, and the API and Beat This paths use stand-ins, as before.
— Rames
3 participants
What
hyperframes audio --only beats,downbeats,bpm,sections,describenow answers the music's structure, each capability through the provider ladder #5602 set up, filled to its designed shape:signalapi→apple-mu→beat-this→signalapi: real against a local stand-in server (no backend yet).apple-mu: unavailable, see below.beat-this: real, run here from an isolated uv env with localsmall0weights.signal: realbeats(median interval + a ±4 s curve)api→apple-mu→signalapi: stand-in.apple-mu: unavailable.signal: realapi→apple-sound→ noneapple-sound: real (macOS SoundAnalysis via a Swift helper). Off macOS nothing local answers, and the value isnullwith a warning--only wordssays to usehyperframes transcribeEvery provider has
available()returning{ ok: true }or{ ok: false, reason }. A provider that is unavailable, whose check throws, or whose run fails is passed over, and the ladder warns once per capability with every reason, e.g.--quality localexcludesapi;--quality fastleaves onlysignal.--provider beats=beat-thispins one;--fullprints each step.Value shapes (JSON, file seconds like the rest;
provenance.<capability>names the provider)beats.timesis empty withreason: "no steady beat"when the signal tracker's confidence is under 0.05, or"too short for a steady beat"under 6 s. Signal downbeats carryconfidence: "low".Why
Agents cut to music. A beat grid, a bar, a section change and "what is this" are what they ask about after "where does this hit". The ladder lets a better model answer when one is present, and keeps the answer measured: an
apitime is moved onto a beat signal processing actually heard.Measurement-led times (rule 1)
Each beat, downbeat and inner section boundary the
apireturns is snapped one at a time to the nearest signal-tracked beat, only when one is within a quarter of the answer's own beat period (125 ms at 120 BPM); otherwise it is kept and listed insnap.unsnapped. The service's tempo level and bar phase are kept.snap.medianShiftMsis the median distance moved, a direct read of how much the service and the signal agree.Why a quarter beat: half a beat would let a time jump to the off-beat; an eighth would fail on a service that is off by a typical 20–60 ms at fast tempi. The separate beat benchmark simulated this: an API off by σ = 150 ms goes from F 0.28 raw to 0.97 snapped, and one off by only σ = 20 ms goes from 0.996 to 0.977, so snapping costs a little on an already precise answer. That cost is why it is reported per answer rather than hidden.
The
apicontract (for the HeyGen backend route)POST <HYPERFRAMES_AUDIO_API_URL>, the URL exactly as configured, no path appended.Authorization: Bearer <HYPERFRAMES_AUDIO_API_KEY>. Unset URL or key → unavailable. The key is never printed or put in a result.HYPERFRAMES_AUDIO_API_TIMEOUT_MS(default 120000).multipart/form-data, two parts:request(application/json):{ "version": 1, "capabilities": ["beats", "downbeats"], "audio": { "sampleRate": 44100, "channels": 2, "duration": 31.5 } }capabilities⊂beats | downbeats | sections | describe. The CLI asks for beats and downbeats together in one request, and for sections and describe each in its own.audio(audio/wav, filenameaudio.wav): 16-bit PCM of the analyzed span, channels and sample rate as decoded.200application/json, times in seconds from the start of the uploaded audio; leave out what you cannot answer:{ "version": 1, "source": "free text, reported as given", "beats": { "times": [0.48, 0.98] }, "downbeats": { "times": [0.48], "meter": 4 }, "sections": { "spans": [{ "start": 0, "end": 12.1, "label": "intro" }] }, "describe": { "text": "optional sentence", "tags": [{ "label": "music", "score": 0.91 }] } }{ "error": { "message": "..." } }in the body is shown in the warning. A missing capability, or one in another shape, also fails that rung.sourceis how the service names what answered.The types are
AudioApiRequest/AudioApiResponseinpackages/cli/src/audio/analysis/providers/api.ts.apple-mu: unavailable, and whyMusic Understanding (WWDC26 session 253,
MusicUnderstandingSession) ships with macOS 27. This Mac runs macOS 26.5.2 with only the Command Line Tools SDK 26.5, which has no MusicUnderstanding framework, so there is no SDK to build or check a helper against. The provider is therefore real only as a gate: on macOS < 27 it saysMusic Understanding needs macOS 27; this Mac runs 26; on 27+ it saysno Music Understanding helper yet: it needs the macOS 27 SDK to build; off macOS,macOS only. No Swift for it is written, so nothing guesses at the API.beat-this: opt-in, never downloadsBeat This! (CPJKU, MIT) runs only through Python/PyTorch. It is available when
HYPERFRAMES_BEAT_THIS_PYTHONnames a Python that hasbeat_this(checked withimportlib.util.find_spec, no torch import) and thesmall0weights are on disk:HYPERFRAMES_BEAT_THIS_CHECKPOINT, orbeat_this-small0.ckptin the Torch hub cache. The weights are passed as a local path, which Beat This loads withtorch.load; with a fresh emptyTORCH_HOMEa run here created nothing there. The audio is handed over as a WAV the CLI decoded, so its own.m4alimitation does not apply. Install steps are indocs/packages/cli.mdx.signalbeats and sections (rule 6)beat-bench/scripts/tsbeat.tswith local tempo on, as that benchmark recommended.no steady beat.confidence: "low".Benchmark numbers
This PR's synthetic grids (scripts in the session scratchpad, not committed), built from the real corpus: real sfx hits (kick / snare / hat kits) on a known grid, 30 s, 70–170 BPM, a quarter in 3/4, over no bed, a real voice bed at −3 dB or a real music bed at −9 dB; F at ±70 ms, Acc1 ±4 %, Acc2 also allows ×2, ×½, ×3, ×⅓.
Sections, 40 cases of 3–5 real music excerpts (12–30 s each) joined end to end: boundary F 0.741 at ±3 s, 0.648 at ±0.5 s, 1.32 boundaries found per true one (some of the extra ones are real changes inside an excerpt). Speed: 3.3 ms of compute per second of audio for beats, 3.4 ms/s for sections.
On the beat benchmark's own 200-file synthetic set (read only; swing, tempo steps and drift, rhythmic beds): beat F 0.692, downbeat F 0.387; Acc1 0.484 / Acc2 0.694 on its 124 constant-tempo files.
The separate beat benchmark (its REPORT.md was not written; these numbers came from its run, scripts and outputs in the session's
beat-bench/). R1 = 22 agent-synthesized scores with script grids, 15 of them at 96 BPM, not produced music:signalrung)beat-thisrung)hyperframes beatsIt confirmed the designed order for beats and downbeats (
api→apple-mu→beat-thissmall0 →signal). It left out madmom (its models are CC BY-NC-SA, non-commercial) and librosa (no advantage).Overlap with
hyperframes beatshyperframes beatsis unchanged by this PR. It still runs the headless-Chrome analyzer, which scored F 0.437 and Acc1 0.18 on R1. Its tempo averages two unfolded estimates (Math.round((onsetBpm + detectiveBpm) / 2)inanalyzeMusicFromBuffer,packages/core/src/beats/beatDetection.ts), and it phases the grid from noisy energy peaks. Next step: havebeatswriterhythm()'s beats (or the ladder's) to the Studio beat file.Tests
audioRhythm.test.ts: a steady 120 BPM grid (beats, tempo, 4/4 bars), a tempo change partway through, no beat in silence or noise or a 3 s clip, sections A B A split on beats, the snapper.audioAnalysis.test.ts: falls through an unavailable provider and one whose check throws, with one warning carrying every reason.providers/api.test.tsagainst a localnode:httpstand-in: method, Bearer header, the multipartrequestJSON and WAV part; per-beat snapping with an invented off-grid beat kept inunsnapped; downbeats; bpm from api beats; sections and describe asked on their own; a 503 and a malformed answer fall through tosignalwith warnings; the key never appears in a result.providers/localProviders.test.ts:beat-thisavailability (unset, missing file, missing weights, package missing) with a stand-in Python script, beats + downbeats + meter from one run, a crashing run falling through;apple-mureasons;apple-soundunavailable off macOS and a real classification on macOS.analyze.test.ts: bpm follows the beats' provider and is in file seconds;--quality fastkeeps a service off.Mutations that fail without the fix:
return { times: rounded(times), snap: … }insnappedBeats):posts the audio and the capabilities, and snaps each answered beat onto the signal's ownfails,expected [0.271, 0.772, …] to deeply equal [0.241, 0.742, …].cumulativeScore(local, () => period)):follows a tempo that changes partway throughfails, recall 0.74 < 0.85.Runs here: core
src/audio*398 passed; clisrc/audio+src/commands1683 passed, 33 skipped; typecheck, oxlint, oxfmt, fallow audit (no new findings) clean. Full suites: core has 1 failure and cli has 5, all in files this PR does not touch. They also fail when run alone on this machine: corefonts/deterministicFonts-googleSubsets(5 s timeout), cliwhisper/sherpa,whisper/manager.installandserver/fileWatcher.atomic.Not exercised
apple-muanalysis: there is no macOS 27 here.small0weights.apple-soundon CI runners without Xcode CLT. It is skipped off macOS, and it compiles its helper into~/.cache/hyperframes/audio-helperson first use (about 3 s).describeoff macOS. None was added.This was meant to stack on #5602, but #5602 merged and its branch was deleted before this opened, so this PR targets
main.