Finetuning facebook/wav2vec2-xls-r-2b

[Tirthankar-iiitb]

The requirement is to fine-tune the XLSR-2B pre-trained model for a new language/accent (Bhojpuri) using adapters. I have ~6 hrs of audio & transcriptions for the new language/accent. I want to use K2 for the same. Which recipe I can simulate? Any indicator to this will be very helpful.

I can do similar fine-tuning using Huggingface (HF). But I need to use Sherpa for inferences which I cannot do on the HF model.

[marcoyang1998]

You may look at https://github.com/marcoyang1998/icefall/tree/finetune_hubert/egs/librispeech/ASR/finetune_hubert_transducer, it's a recipe for fine-tuning a HuBERT model.

Also, if you want to deploy a wav2vec2 model with Sherpa, you may find this useful (k2-fsa/sherpa#198). Doing the fine-tuning in icefall is not necessary for deployment with Sherpa, as long as you have the model in the right format (torchscript, onnx etc.)

[Tirthankar-iiitb]

@marcoyang1998 - I was trying to run finetune_hubert_transducer (lhose version - '1.20.0.dev+git.0089643.clean') but faced with an error 'ImportError: cannot import name 'SingleCutSampler'.

from asr_datamodule import LibriSpeechAsrDataModule
File "/root/icefall/egs/librispeech/tirtho_ASR/finetune_hubert_transducer/asr_datamodule.py", line 37, in
from lhotse.dataset.sampling import SingleCutSampler
ImportError: cannot import name 'SingleCutSampler' from 'lhotse.dataset.sampling' (/root/.virtualenvs/newstream/lib/python3.8/site-packages/lhotse/dataset/sampling/init.py)

Any other library to be used with the latest Lhotse version?

Thanks.

[mohsen-goodarzi]

It seems that your icefall version is a bit old for your Lhotse.
More info:
lhotse-speech/lhotse#546

[Tirthankar-iiitb]

Oh...ok. Thanks. Let me consolidate the versions.

[marcoyang1998]

Note that the recipe is quite old and is just for a reference @Tirthankar-iiitb

[Tirthankar-iiitb]

Ok thanks. Let me try finetune.sh for the same task with necessary changes. Incase you have another recipe (latest), please do share. Thanks again.