Skip to content

fix: align repository SenseVoice timestamps with FunASR - #354

Merged
LauraGPT merged 1 commit into
mainfrom
codex/sensevoice-remote-timestamps-20260910
Sep 10, 2026
Merged

fix: align repository SenseVoice timestamps with FunASR#354
LauraGPT merged 1 commit into
mainfrom
codex/sensevoice-remote-timestamps-20260910

Conversation

@LauraGPT

Copy link
Copy Markdown
Member

Summary

Related to #291. This does not close the issue: the reporter's original audio and pinned environment are unavailable.

  • Initialize the repository position encoder as an nn.Module so checkpoint state inspection and normal module calls work.
  • Bring the repository's timestamp output into FunASR's VAD contract: millisecond pairs plus a parallel words list, instead of token/seconds triples that fail at int(token).
  • Supply CTC log probabilities and re-encoded text targets; retain original-token ownership when one token expands into multiple IDs, and reject incomplete alignment instead of silently truncating it.
  • Keep batch lengths per item and normalize SentencePiece word boundaries.
  • Document the direct-call schema migration and update demo2.py; add CPU model contracts to CI.

Evidence

All development and real inference ran on ind-gpu8. Model weights were not changed.

  • Original repository implementation: actual checkpoint loading fails with missing nn.Module state hooks under PyTorch 2.11.0+cu128.
  • Module-only correction: both punctuation-on and punctuation-off pipelines fail with invalid literal for int() on a punctuation token, matching the timestamp type failure shown in the reporter's second screenshot.
  • A schema-only intermediate candidate was rejected: it returned only 3 aligned words for a full transcript. That evidence is retained; eliminating the exception alone was not acceptance.
  • A multi-ID real-aligner regression failed before the owner map, producing [[0,30],[30,90]] instead of [[0,90],[150,210]].
  • Final candidate: 44 repository tests passed with third-party pytest plugin autoload disabled; compile and whitespace checks passed.
  • Actual published FunASR 1.4.15 wheel was isolated and all 443 RECORD hashes checked. Fixed SenseVoice HF revision3847d57b6bdf2dd8875cb1508d2af43d80a16bf7, FSMN-VAD df20e6b, CAM++ e4b6ede7 and CT-punc d0e55e2 were used.
  • On the fixed public 51.66275-second sample, final repository source loaded successfully and returned 243 aligned words/timestamp pairs covering all decoded text with punctuation both off and on. The composed pipelines produced 3 and 25 sentence records respectively.
  • Actual unequal-length no-VAD batch: English7.18s and Chinese18s returned15/76 word pairs with complete text coverage and valid chronological intervals. Timestamp-disabled output retained its original schema.

Limits

This is functional/schema validation, not word-boundary accuracy, diarization accuracy, speaker identity recognition, hardware throughput, or a fix for the reporter's first empty-result screenshot. The model still emits Korean text on a one-second zero waveform without VAD; that existing hallucination is not hidden or claimed fixed. Punctuation duplication in the public sample is also not fixed here.

The tested runtime is a preserved PyTorch2.11/FunASR1.4.15 environment, not a clean installation of this repository's newer default requirements. CPU CI separately tests module/CTC contracts on PyTorch2.12.1; the normal container build validates installation. Original model card/Space permissions and GHCR visibility are unchanged.

Rollback

The original branch bundle and every candidate/source/log are backed up under the operations evidence store. Only the six explicitly reviewed files are included. No unrelated checkout or dependency changes, model publication, PyPI release or website deployment.

Signed-off-by: LauraGPT <18321252+LauraGPT@users.noreply.github.com>
@LauraGPT
LauraGPT merged commit ea15219 into main Sep 10, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant