Add XLSum evaluation / unify eval script

README.md

@@ -1,16 +0,0 @@
-### Previous Experiments 
-- `exp-001`: train gpt-2's tokenizer and finetune gpt-2's embedding layers `wte` and `wpe` on HF's OSCAR `unshuffled_deduplicated_fr` and `unshuffled_dudplicated_kr`.
-- `exp-002`: evaluate gpt-2 on FLUE's tasks (CLS, XNLI, PAWS)
-- `exp-003`: TODO: evaluate on multiatis 
-- `exp-004`: Does the embedding layer learn anything useful? Take a dataset in English for PAWS-X, finetune GPT-2 on this dataset, evaluate it on English test set T_e. Then, take the same test-set T_e translated in French (T_f), take GPT-2 parameters fine-tuned for the task X,  replace English embeddings with French embeddings and evaluate thus obtained model on French test set.
-
-# Experiment folders below after Conversation with Vassilina, Hady, Iz, and Maruf [Link](https://huggingface.slack.com/archives/C020G6A9KHQ/p1637023149074800) 
-- `exp-005`: cleaned from `exp-001` for finetuning GPT-2 embedding layers for DE and KO on Oscar.
-- `exp-006`: run zero-shot and finetuned evaluation setting for XNLI ✅, PAWS ❌, and XQuAD ❌. (❌ means not done. ✅ means done.)
-- `exp-007`: apply MAD-X adapter method. [Paper link](https://arxiv.org/abs/2005.00052)
-- `exp-008`: from exp-006, but using mBERT on the zero-shot and finetuning setting.
-
-
-# Carbon Tracking 
-Do not forget to log your experiments [in this spreadsheet](https://docs.google.com/spreadsheets/d/1Mk8mYCOF_WxMv-Uv5ThkFs5Ak5B9s9EnRUh1CpykEJ0/edit#gid=0)
-

jz/README.md

@@ -0,0 +1,64 @@
+# Run on JZ
+
+## Getting Started
+Clone the GitHub Repository and `cd` into it to run commands like `sbatch jz/emb.sh my 100000 24000 extend`.
+
+```
+git clone https://github.com/bigscience-workshop/multilingual-modeling.git
+cd multilingual-modeling/
+```
+
+## Change Configuration
+### SLURM Configuration
+We need to change the SLURM setting according to JZ to get the necessary compute.
+```
+# use a single V100 for each run
+#SBATCH --partition=gpu-he --gres=gpu:1  
+
+# output/error files for tracking pip installation
+#SBATCH -o /users/zyong2/data/zyong2/bigscience/logs/misc/lang-adapt-env_jz_lang_adapter.out
+#SBATCH -e /users/zyong2/data/zyong2/bigscience/logs/misc/lang-adapt-env_jz_lang_adapter.err
+```
+
+### Directory configuration (Line 22 - 28 in jz/emb.sh)
+Also, we need to change 6 lines of the directory configuration.
+```
+# virtual environment folder for `python3 -m venv $env_dir`
+env_dir="/users/zyong2/data/zyong2/bigscience/gh/multilingual-modeling/jz/env_jz_lang_adapter"
+
+# cache directory for HuggingFace datasets
+cache_dir="/users/zyong2/data/zyong2/huggingface"
+
+# cloned GitHub directory
+mm_dir="/users/zyong2/data/zyong2/bigscience/gh/multilingual-modeling"
+
+# directory to save adapted models and trained tokenizers
+output_dir="/users/zyong2/data/zyong2/bigscience/data/processed/misc/"  
+
+# folder for storing error and output logging text files
+logging_txt_dir="/users/zyong2/data/zyong2/bigscience/logs/misc"  
+
+# folder for storing all tensorboard logging
+logging_tb_dir="/users/zyong2/data/zyong2/bigscience/reports/misc/"
+```
+
+## Runs
+### 07/05/2022 (Language Adaptation - Embedding-only)
+Run the following commands for doing language adaptation for 4 languages varying along the the size of training samples. 
+```
+sbatch jz/emb.sh my 100000 24000 extend
+sbatch jz/emb.sh my 10000 5000 extend
+sbatch jz/emb.sh my 1000 5000 extend
+
+sbatch jz/emb.sh si 100000 24000 extend
+sbatch jz/emb.sh si 10000 5000 extend
+sbatch jz/emb.sh si 1000 5000 extend
+
+sbatch jz/emb.sh az 100000 24000 extend
+sbatch jz/emb.sh az 10000 5000 extend
+sbatch jz/emb.sh az 1000 5000 extend
+
+sbatch jz/emb.sh de 100000 24000 extend
+sbatch jz/emb.sh de 10000 5000 extend
+sbatch jz/emb.sh de 1000 5000 extend
+```

jz/emb.sh

@@ -0,0 +1,99 @@
+#!/bin/bash
+
+# Request half an hour of runtime:
+#SBATCH --time=2-23:59:00
+
+# Ask for the GPU partition and 1 GPU
+#SBATCH --partition=gpu-he --gres=gpu:1
+
+# Default resources are 1 core with 2.8GB of memory.
+#SBATCH --ntasks=8
+
+# Use more memory (10GB) (CPU RAM):
+#SBATCH --mem=200g
+
+# Specify a job name:
+#SBATCH -J lang-adapt-env_jz_lang_adapter
+
+# Specify an output file
+#SBATCH -o /users/zyong2/data/zyong2/bigscience/logs/misc/lang-adapt-env_jz_lang_adapter.out
+#SBATCH -e /users/zyong2/data/zyong2/bigscience/logs/misc/lang-adapt-env_jz_lang_adapter.err
+
+env_dir="/users/zyong2/data/zyong2/bigscience/gh/multilingual-modeling/jz/env_jz_lang_adapter"
+cache_dir="/users/zyong2/data/zyong2/huggingface"
+mm_dir="/users/zyong2/data/zyong2/bigscience/gh/multilingual-modeling"
+
+output_dir="/users/zyong2/data/zyong2/bigscience/data/processed/misc/"  # adapted model and trained tokenizer directory
+logging_txt_dir="/users/zyong2/data/zyong2/bigscience/logs/misc"  # error and output logging
+logging_tb_dir="/users/zyong2/data/zyong2/bigscience/reports/misc/"  # tensorboard logging
+
+mkdir -p $output_dir
+mkdir -p $logging_tb_dir
+mkdir -p $logging_txt_dir
+
+lang=$1  # language
+sample_size=$2  # training sample size
+vocab_size=$3  # vocab size of tokenizer
+tok_strategy=$4  # extend, replace, overlap-replace
+bigs_model="bigscience/bloom-1b3"
+adpt_strategy="emb"
+
+tokenizer_dir="${output_dir}/tok_$(basename $bigs_model)_${lang}_oscar_${sample_size}samples_${vocab_size}vocab_${tok_strategy}"
+logging_tb_dir="${logging_tb_dir}/$(basename $bigs_model)_${lang}_oscar_${sample_size}samples_${vocab_size}vocab_tok-${tok_strategy}_adpt-${adpt_strategy}"
+
+# setup environment
+module load python/3.7.4
+[ -d $env_dir ] || python3 -m venv $env_dir
+source "${env_dir}/bin/activate"
+pip3 install --upgrade pip
+pip3 install -r "${mm_dir}/requirements.txt"
+
+# train tokenizer
+python "${mm_dir}/scripts/lang_adapt/tokenized4clm_sampled.py" \
+--lang $lang \
+--model $bigs_model \
+--tokenizer_dir $tokenizer_dir \
+--hf_cache_dir $cache_dir \
+--vocab_size $vocab_size \
+--sample_size $sample_size \
+--use_auth_token \
+--tok_strategy $tok_strategy \
+> "${logging_txt_dir}/tok_$(basename $bigs_model)_${lang}_oscar_${sample_size}samples_${vocab_size}vocab_${tok_strategy}.txt" \
+2> "${logging_txt_dir}/tok_$(basename $bigs_model)_${lang}_oscar_${sample_size}samples_${vocab_size}vocab_${tok_strategy}.err"
+
+
+# finetune language model for langauge adaptation
+python "${mm_dir}/scripts/lang_adapt/madx_run_clm.py" \
+    --seed 0 \
+    --fp16 \
+    --model_name_or_path $bigs_model \
+    --tokenizer_name $tokenizer_dir \
+    --dataset_name oscar \
+    --cache_dir $cache_dir \
+    --dataset_config_name "unshuffled_deduplicated_${lang}" \
+    --logging_dir $logging_tb_dir \
+    --report_to "tensorboard" \
+    --learning_rate 0.001 \
+    --do_train \
+    --do_eval \
+    --output_dir $output_dir \
+    --preprocessing_num_workers 8 \
+    --overwrite_output_dir \
+    --per_device_train_batch_size 2 \
+    --gradient_accumulation_steps 4 \
+    --per_device_eval_batch_size 2 \
+    --eval_accumulation_steps 4 \
+    --eval_steps 1000 \
+    --evaluation_strategy "steps" \
+    --max_eval_samples 5000 \
+    --save_steps 5000 \
+    --save_strategy "steps" \
+    --max_train_samples $sample_size \
+    --max_steps 50000 \
+    --logging_steps 1000 \
+    --lang_adapt_strategies $adpt_strategy \
+    --embedding_strategies $tok_strategy \
+    --load_best_model_at_end \
+    --use_auth_token \
+    > "${logging_txt_dir}/$(basename $bigs_model)_${lang}_oscar_${sample_size}samples_${vocab_size}vocab_tok-${tok_strategy}_adpt-${adpt_strategy}.txt" \
+    2> "${logging_txt_dir}/$(basename $bigs_model)_${lang}_oscar_${sample_size}samples_${vocab_size}vocab_tok-${tok_strategy}_adpt-${adpt_strategy}.err"

requirements.txt

@@ -0,0 +1,5 @@
+git+https://github.com/yongzx/adapter-transformers.git@f55ab013599088a35c87a880ba13a6d912e27ef4
+--extra-index-url https://download.pytorch.org/whl/cu113
+torch
+datasets
+tensorboardX

scripts/README.md

@@ -0,0 +1,6 @@
+### README
+
+This folder contains everything we need for running BigScience language adaptation experiments.
+
+Google Doc: [BigScience - Extending BLOOM to New Languages](https://docs.google.com/document/d/1OEJq2max5kLPF4mnnb9nyoodqR_z_UVQlw4tVx9TvTc/edit#heading=h.kk1966kbedef)
+