Add fairseq as a submodule

Author: XiaoshanHsj

WavLLM/wavllm/scripts/inference_sft.sh

@@ -9,22 +9,21 @@ src_dir=${model_path%/*}
 cpt=${model_path##*/}
 cpt=${cpt%.*}
 
-beam_size=$2
-gen_set=$3
+gen_set=$2
 [ -z $gen_set ] && gen_set="?"
 [ -z $beam_size ] && beam_size=1
 
 
 FAIRSEQ_ROOT=${PWD}
-DATA_DIR=$FAIRSEQ_ROOT/../wavllm/test_data
+DATA_DIR=$FAIRSEQ_ROOT/examples/wavllm/test_data
 
 for subset in $gen_set; do
     results_path=$src_dir/decode_${cpt}_beam${beam_size}/${subset}
     [ ! -d $results_path ] && mkdir -p $results_path
 
-    python $FAIRSEQ_ROOT/../wavllm/inference/generate.py $DATA_DIR \
-    --user-dir ../wavllm \
-    --tokenizer-path $FAIRSEQ_ROOT/../wavllm/tokenizer/tokenizer.model \
+    python $FAIRSEQ_ROOT/examples/wavllm/inference/generate.py $DATA_DIR \
+    --user-dir examples/wavllm \
+    --tokenizer-path $FAIRSEQ_ROOT/examples/wavllm/tokenizer/tokenizer.model \
     --gen-subset ${subset} \
     \
     --task speechllm_task \
@@ -37,16 +36,4 @@ for subset in $gen_set; do
     --max-tokens 1600000 \
     --sampling --beam 1 --nbest 1 --temperature 0.5 \
     --max-len-a 0 --max-len-b 512
-    # --beam ${beam_size} \
-    #
-    
-
-    # echo $results_path
-    # tail -n 1 $results_path/generate-*.txt
-    # sleep 1s
-done
-
-    #--max-sample-size 2000000 \
-    # --max-len-a 0 --max-len-b 512 \
-    # --skip-invalid-size-inputs-valid-test \
-    # --max-sentences 20 \
+done

WavLLM/wavllm/test_data/CoT-task.tsv

@@ -0,0 +1,2 @@
+id	audio	n_frames	prompt	with_speech	tgt_text
+0	SpeechT5/WavLLM/fairseq/examples/wavllm/test_data/audio/CoT-task.wav	214437	First of all, transcribe the audio recording into text, capturing every spoken word; Additionally given this audio clip and text, can you condense it into a clear, concise summary, no more than 20 words?; Lastly disregarding the sound, translate this English summary into German.	True	Drei Filme aus dem asiatisch-pazifischen Raum im Rennen in Cannes

WavLLM/wavllm/test_data/II-task.tsv

@@ -0,0 +1,2 @@
+id	audio	n_frames	with_speech	prompt	tgt_text
+0	SpeechT5/WavLLM/fairseq/examples/wavllm/test_data/audio/II-task.wav	111111	True	To begin, Transcribe the audio recording into text, capturing every spoken word; Subsequently, How does the woman finally decide to go home? A. By bus; B. In the man’s car; C. In her father’s car.; Furthermore, ignore the audio clip, What is the capital of New Zealand?; Lastly, Continue the narrative of given audio clip in a coherent and engaging way	ASR+SQA+SFT+Continue

WavLLM/wavllm/test_data/SQA.tsv

@@ -0,0 +1,2 @@
+id	audio	n_frames	prompt	tgt_text	with_speech
+0	SpeechT5/WavLLM/fairseq/examples/wavllm/test_data/audio/sqa.wav	111111	What will the man do next? A. Start to take exercise; B. Do as he always does; C. Change his working time.	A	True

WavLLM/wavllm/test_data/SQQA.tsv

@@ -0,0 +1,2 @@
+id	audio	n_frames	prompt	tgt_text	with_speech
+0	SpeechT5/WavLLM/fairseq/examples/wavllm/test_data/audio/sqqa.wav	182574		The fundamental theorem of calculus is a theorem that links the concept of the derivative of a function with the concept of the integral .	True

WavLLM/wavllm/test_data/asr.tsv

@@ -0,0 +1,2 @@
+id	audio	n_frames	prompt	tgt_text	with_speech
+0	SpeechT5/WavLLM/fairseq/examples/wavllm/test_data/audio/asr.flac	166960	Based on the attached audio, generate a comprehensive text transcription of the spoken content.	he hoped there would be stew for dinner turnips and carrots and bruised potatoes and fat mutton pieces to be ladled out in thick peppered flour fattened sauce	True