400k steps into training, still heavy halucinating/unintelligible

[DedaDev]

Checks

• This template is only for usage issues encountered.
• I have thoroughly reviewed the project documentation but couldn't find information to solve my problem.
• I have searched for existing issues, including closed ones, and couldn't find a solution.
• I am using English to submit this issue to facilitate community communication.

Environment Details

runpod/pytorch:2.4.0-py3.11-cuda12.4.1-devel-ubuntu22.04

Steps to Reproduce

This is the training config that I use with my dataset of 130 hours of clean Serbian speech:

  exp_name:               F5TTS_v1_Base
  tokenizer:              char
  mixed_precision:        bf16
  learning_rate:          7.5e-05
  batch_size_per_gpu:     20189
  batch_size_type:        frame
  max_samples:            64
  grad_accumulation_steps: 1
  max_grad_norm:          1
  epochs:                 434
  num_warmup_updates:     3779
  save_per_updates:       5000
  keep_last_n_checkpoints: 1
  last_per_updates:       10000
  logger:                 tensorboard
  dataset:                serbian
  finetune:               false (training from scratch)
  dataset_size:           60,948 samples / 132.05 hours
  gpu:                    NVIDIA A40 (46GB)

after 3 days of training and ~400k steps, inferenced audio is still halucinating and repeating some parts of the word or the whole words, sometimes also unintelligible.

Loss curve

Learninig rate

✔️ Expected Behavior

referenced audio: https://voca.ro/1iSJaiUm5CHz
referenced text: u tom komitetu dobijamo vrlo vrlo opširne biografije kandidata, sa kojima vodimo razgovor i biramo ih, čak i ispitujemo.

❌ Actual Behavior

inferenced text: (same as referenced text)
inferenced audio: https://voca.ro/1mT0JkcugloJ
(this is with EMA enabled, without EMA is much worse)

[PoTaTo-Mika]

I trained the model on Emilia according to the repo's code, with 8xH200 and the same training config, the model begins to generate speech at about 150K steps, and performs well at 200K steps.