diff --git a/graph/R-GAT/README.md b/graph/R-GAT/README.md index 7e37b8eeaa..6a44989fca 100644 --- a/graph/R-GAT/README.md +++ b/graph/R-GAT/README.md @@ -85,31 +85,12 @@ mlcr get,ml-model,rgat,_r2-downloader,_mlcommons --outdirname= ### Download the model using MLC R2 Downloader -Download the model using the MLCommons R2 Downloader: +Download the model using the MLCommons R2 Downloader (more information about the MLC R2 Downloader, including how to run it on Windows, can be found [here](https://inference.mlcommons-storage.org)): ```bash bash <(curl -s https://raw.githubusercontent.com/mlcommons/r2-downloader/refs/heads/main/mlc-r2-downloader.sh) https://inference.mlcommons-storage.org/metadata/rgat-model.uri ``` -### Download model using Rclone - -To run Rclone on Windows, you can download the executable [here](https://rclone.org/install/#windows). -To install Rclone on Linux/macOS/BSD systems, run: -``` -sudo -v ; curl https://rclone.org/install.sh | sudo bash -``` -Once Rclone is installed, run the following command to authenticate with the bucket: -``` -rclone config create mlc-inference s3 provider=Cloudflare access_key_id=f65ba5eef400db161ea49967de89f47b secret_access_key=fbea333914c292b854f14d3fe232bad6c5407bf0ab1bebf78833c2b359bdfd2b endpoint=https://c2686074cb2caf5cbaf6d134bdba8b47.r2.cloudflarestorage.com -``` -You can then navigate in the terminal to your desired download directory and run the following commands to download the checkpoints: - -**`fp32`** -``` -rclone copy mlc-inference:mlcommons-inference-wg-public/R-GAT/RGAT.pt $MODEL_PATH -P -``` - - ### Download and setup dataset #### Debug Dataset diff --git a/language/deepseek-r1/README.md b/language/deepseek-r1/README.md index a6c30a6155..3e9aa99a76 100644 --- a/language/deepseek-r1/README.md +++ b/language/deepseek-r1/README.md @@ -13,6 +13,14 @@ You can also do pip install mlc-scripts and then use `mlcr` commands for downloa - DeepSeek-R1 model is automatically downloaded as part of setup - Checkpoint conversion is done transparently when needed. +**Using MLCFlow Automation** + +Download the model using the MLCFlow Automation: + +``` +mlcr get,ml-model,deepseek-r1,_r2-downloader,_mlc --outdirname= -j +``` + **Using the MLC R2 Downloader** Download the model using the MLCommons R2 Downloader: diff --git a/language/gpt-j/README.md b/language/gpt-j/README.md index 9c952b65db..9538b3a686 100644 --- a/language/gpt-j/README.md +++ b/language/gpt-j/README.md @@ -9,7 +9,7 @@ You can also do `pip install mlc-scripts` and then use `mlcr` commands for downl ### Download model through MLCFlow Automation ``` -mlcr get,ml-model,gptj,_pytorch --outdirname= -j +mlcr get,ml-model,gptj,_pytorch,_fp32,_r2-downloader --outdirname= -j ``` ### Download dataset through MLCFlow Automation @@ -103,21 +103,14 @@ mlcr get,ml-model,gptj,_pytorch,_rclone ---outdirname =./model -P #### Manual method -The above command automatically runs a set of Rclone commands to download the data from a Cloudflare R2 bucket. However, if you'd like to run the Rclone commands manually, you can do so as follows: +The above command automatically runs a set of commands to download the data from a Cloudflare R2 bucket. However, if you'd like to run the commands manually, you can do so as follows: -To run Rclone on Windows, you can download the executable [here](https://rclone.org/install/#windows). -To install Rclone on Linux/macOS/BSD systems, run: -``` -sudo -v ; curl https://rclone.org/install.sh | sudo bash -``` -Once Rclone is installed, run the following command to authenticate with the bucket: -``` -rclone config create mlc-inference s3 provider=Cloudflare access_key_id=f65ba5eef400db161ea49967de89f47b secret_access_key=fbea333914c292b854f14d3fe232bad6c5407bf0ab1bebf78833c2b359bdfd2b endpoint=https://c2686074cb2caf5cbaf6d134bdba8b47.r2.cloudflarestorage.com -``` -You can then navigate in the terminal to your desired download directory and run the following command to download the model checkpoint: +(More information about the MLC R2 Downloader, including how to run it on Windows, can be found [here](https://inference.mlcommons-storage.org)) + +Navigate in the terminal to your desired download directory and run the following command to download the model checkpoint: ``` -rclone copy mlc-inference:mlcommons-inference-wg-public/gpt-j ./model -P +bash <(curl -s https://raw.githubusercontent.com/mlcommons/r2-downloader/refs/heads/main/mlc-r2-downloader.sh) -d model https://inference.mlcommons-storage.org/metadata/gpt-j-model-checkpoint.uri ``` diff --git a/language/llama2-70b/README.md b/language/llama2-70b/README.md index 8bec240469..5b3335b39d 100644 --- a/language/llama2-70b/README.md +++ b/language/llama2-70b/README.md @@ -66,7 +66,7 @@ CPU-only setup, as well as any GPU versions for applicable libraries like PyTorc ## Get Model ### MLCommons Members Download -MLCommons hosts the model and preprocessed dataset for download **exclusively by MLCommons Members**. You must first agree to the [confidentiality notice](https://llama2.mlcommons.org) using your organizational email address, then you will receive a link to a directory containing Rclone download instructions. _If you cannot access the form but you are part of a MLCommons Member organization, submit the [MLCommons subscription form](https://mlcommons.org/community/subscribe/) with your organizational email address and [associate a Google account](https://accounts.google.com/SignUpWithoutGmail) with your organizational email address._ +MLCommons hosts the model and preprocessed dataset for download **exclusively by MLCommons Members**. You must first agree to the [confidentiality notice](https://llama2.mlcommons.org) using your organizational email address, then you will receive a link to a page containing download instructions. _If you cannot access the form but you are part of a MLCommons Member organization, submit the [MLCommons subscription form](https://mlcommons.org/community/subscribe/) with your organizational email address and [associate a Google account](https://accounts.google.com/SignUpWithoutGmail) with your organizational email address._ ### Download model through MLCFlow Automation @@ -117,21 +117,12 @@ mlcr get,dataset,openorca,_calibration --outdirname= -j ### Preprocessed -You can use Rclone to download the preprocessed dataset from a Cloudflare R2 bucket. +You can use the MLCommons R2 Downloader to download the preprocessed dataset from a Cloudflare R2 bucket (more information about the MLC R2 Downloader, including how to run it on Windows, can be found [here](https://inference.mlcommons-storage.org)). -To run Rclone on Windows, you can download the executable [here](https://rclone.org/install/#windows). -To install Rclone on Linux/macOS/BSD systems, run: -``` -sudo -v ; curl https://rclone.org/install.sh | sudo bash -``` -Once Rclone is installed, run the following command to authenticate with the bucket: -``` -rclone config create mlc-inference s3 provider=Cloudflare access_key_id=f65ba5eef400db161ea49967de89f47b secret_access_key=fbea333914c292b854f14d3fe232bad6c5407bf0ab1bebf78833c2b359bdfd2b endpoint=https://c2686074cb2caf5cbaf6d134bdba8b47.r2.cloudflarestorage.com -``` -You can then navigate in the terminal to your desired download directory and run the following command to download the dataset: +Navigate in the terminal to your desired download directory and run the following command to download the dataset: ``` -rclone copy mlc-inference:mlcommons-inference-wg-public/open_orca ./open_orca -P +bash <(curl -s https://raw.githubusercontent.com/mlcommons/r2-downloader/refs/heads/main/mlc-r2-downloader.sh) https://inference.mlcommons-storage.org/metadata/llama-2-70b-open-orca-dataset.uri ``` ### Unprocessed diff --git a/language/llama3.1-405b/README.md b/language/llama3.1-405b/README.md index 52d986628f..8d2e8bb0e4 100644 --- a/language/llama3.1-405b/README.md +++ b/language/llama3.1-405b/README.md @@ -101,7 +101,7 @@ pip install -e ../../loadgen ### MLCommons Members Download (Recommended for official submission) -MLCommons hosts the model for download **exclusively by MLCommons Members**. You must first agree to the [confidentiality notice](https://llama3-1.mlcommons.org) using your organizational email address, then you will receive a link to a directory containing Rclone download instructions. _If you cannot access the form but you are part of a MLCommons Member organization, submit the [MLCommons subscription form](https://mlcommons.org/community/subscribe/) with your organizational email address and [associate a Google account](https://accounts.google.com/SignUpWithoutGmail) with your organizational email address._ +MLCommons hosts the model for download **exclusively by MLCommons Members**. You must first agree to the [confidentiality notice](https://llama3-1.mlcommons.org) using your organizational email address, then you will receive a link to a page with download instructions. _If you cannot access the form but you are part of a MLCommons Member organization, submit the [MLCommons subscription form](https://mlcommons.org/community/subscribe/) with your organizational email address and [associate a Google account](https://accounts.google.com/SignUpWithoutGmail) with your organizational email address._ ### Download model through MLCFlow Automation @@ -152,7 +152,7 @@ mlcr get,dataset,mlperf,inference,llama3,_calibration,_r2-downloader --outdirnam **Using R2-Downloader** -Download the model using the MLCommons R2 Downloader: +Download the model using the MLCommons R2 Downloader (more information about the MLC R2 Downloader, including how to run it on Windows, can be found [here](https://inference.mlcommons-storage.org)): Validation: @@ -166,30 +166,6 @@ Calibration: bash <(curl -s https://raw.githubusercontent.com/mlcommons/r2-downloader/refs/heads/main/mlc-r2-downloader.sh) https://inference.mlcommons-storage.org/metadata/llama3-1-405b-calibration-dataset-512.uri ``` -**Using RClone** - -You can use Rclone to download the preprocessed dataset from a Cloudflare R2 bucket. - -To run Rclone on Windows, you can download the executable [here](https://rclone.org/install/#windows). -To install Rclone on Linux/macOS/BSD systems, run: -``` -sudo -v ; curl https://rclone.org/install.sh | sudo bash -``` -Once Rclone is installed, run the following command to authenticate with the bucket: -``` -rclone config create mlc-inference s3 provider=Cloudflare access_key_id=f65ba5eef400db161ea49967de89f47b secret_access_key=fbea333914c292b854f14d3fe232bad6c5407bf0ab1bebf78833c2b359bdfd2b endpoint=https://c2686074cb2caf5cbaf6d134bdba8b47.r2.cloudflarestorage.com -``` -You can then navigate in the terminal to your desired download directory and run the following command to download the dataset: - -``` -rclone copy mlc-inference:mlcommons-inference-wg-public/llama3.1_405b/mlperf_llama3.1_405b_dataset_8313_processed_fp16_eval.pkl ./ -P -``` - -You can also download the calibration dataset from the Cloudflare R2 bucket by running the following command: - -``` -rclone copy mlc-inference:mlcommons-inference-wg-public/llama3.1_405b/mlperf_llama3.1_405b_calibration_dataset_512_processed_fp16_eval.pkl ./ -P -``` ## Run Performance Benchmarks diff --git a/language/mixtral-8x7b/README.md b/language/mixtral-8x7b/README.md index ed3f857ff7..14d4c8a59b 100644 --- a/language/mixtral-8x7b/README.md +++ b/language/mixtral-8x7b/README.md @@ -73,26 +73,19 @@ CPU-only setup, as well as any GPU versions for applicable libraries like PyTorc ### Download model through MLCFlow Automation ``` -mlcr get,ml-model,mixtral --outdirname= -j +mlcr get,ml-model,mixtral,_r2-downloader,_mlc --outdirname= -j ``` ### Get Checkpoint -#### Using Rclone +#### Using the MLCommons R2 Downloader -To run Rclone on Windows, you can download the executable [here](https://rclone.org/install/#windows). -To install Rclone on Linux/macOS/BSD systems, run: -``` -sudo -v ; curl https://rclone.org/install.sh | sudo bash -``` -Once Rclone is installed, run the following command to authenticate with the bucket: -``` -rclone config create mlc-inference s3 provider=Cloudflare access_key_id=f65ba5eef400db161ea49967de89f47b secret_access_key=fbea333914c292b854f14d3fe232bad6c5407bf0ab1bebf78833c2b359bdfd2b endpoint=https://c2686074cb2caf5cbaf6d134bdba8b47.r2.cloudflarestorage.com -``` -You can then navigate in the terminal to your desired download directory and run the following command to download the model checkpoint: +(More information about the MLC R2 Downloader, including how to run it on Windows, can be found [here](https://inference.mlcommons-storage.org)) -``` -rclone copy mlc-inference:mlcommons-inference-wg-public/mixtral_8x7b/mixtral-8x7b-instruct-v0.1 ./mixtral-8x7b-instruct-v0.1 -P +Navigate in the terminal to your desired download directory and run the following command to download the model checkpoint: + +``` bash +bash <(curl -s https://raw.githubusercontent.com/mlcommons/r2-downloader/refs/heads/main/mlc-r2-downloader.sh) https://inference.mlcommons-storage.org/metadata/mixtral-8x7b-model-checkpoint.uri ``` ## Get Dataset @@ -102,30 +95,23 @@ rclone copy mlc-inference:mlcommons-inference-wg-public/mixtral_8x7b/mixtral-8x7 **Validation** ``` -mlcr get,dataset-mixtral,openorca-mbxp-gsm8k-combined,_validation --outdirname= -j +mlcr get,dataset-mixtral,openorca-mbxp-gsm8k-combined,_validation,_r2-downloader --outdirname= -j ``` **Calibration** ``` -mlcr get,dataset-mixtral,openorca-mbxp-gsm8k-combined,_calibration --outdirname= -j +mlcr get,dataset-mixtral,openorca-mbxp-gsm8k-combined,_calibration,_r2-downloader --outdirname= -j ``` -- Adding `_wget` tag to the run command will change the download tool from `rclone` to `wget`. - ### Preprocessed -#### Using Rclone -We make many of the MLPerf infernce models and datasets available using Rclone. In order to keep compatibility, you can use Rclone to get the preprocessed dataset: +#### Using the MLCommons R2 Downloader +We make many of the MLPerf infernce models and datasets available using the MLC R2 Downloader (more information about the MLC R2 Downloader, including how to run it on Windows, can be found [here](https://inference.mlcommons-storage.org)). In order to keep compatibility, you can use the MLC R2 Downloader to get the preprocessed dataset: -To run Rclone on Windows, you can download the executable [here](https://rclone.org/install/#windows). -To install Rclone on Linux/macOS/BSD systems, run: +`cd` into the folder where you want to place the dataset and run: ```bash -sudo -v ; curl https://rclone.org/install.sh | sudo bash -``` -Once Rclone is installed, cd into the folder where you want to place the dataset and run: -```bash -rclone copyurl https://inference.mlcommons-storage.org/mixtral_8x7b/09292024_mixtral_15k_mintoken2_v1.pkl ./ -a -P +bash <(curl -s https://raw.githubusercontent.com/mlcommons/r2-downloader/refs/heads/main/mlc-r2-downloader.sh) https://inference.mlcommons-storage.org/metadata/mixtral-8x7b-validation-dataset.uri ``` #### Using wget @@ -138,10 +124,11 @@ wget https://inference.mlcommons-storage.org/mixtral_8x7b/09292024_mixtral_15k_m ### Calibration dataset -#### Using Rclone -Rclone is installed, cd into the folder where you want to place the dataset and run: +#### Using the MLCommons R2 Downloader + +`cd` into the folder where you want to place the dataset and run: ```bash -rclone copyurl https://inference.mlcommons-storage.org/mixtral_8x7b%2F2024.06.06_mixtral_15k_calibration_v4.pkl ./ -a -P +bash <(curl -s https://raw.githubusercontent.com/mlcommons/r2-downloader/refs/heads/main/mlc-r2-downloader.sh) https://inference.mlcommons-storage.org/metadata/mixtral-8x7b-calibration-dataset.uri ``` #### Using wget @@ -307,4 +294,4 @@ For official submissions, 99% of each reference score is enforced. Additionally, ## Automated command for submission generation via MLCFlow -Please see the [new docs site](https://docs.mlcommons.org/inference/submission/) for an automated way to generate submission through MLCFlow. \ No newline at end of file +Please see the [new docs site](https://docs.mlcommons.org/inference/submission/) for an automated way to generate submission through MLCFlow. diff --git a/recommendation/dlrm_v2/pytorch/README.md b/recommendation/dlrm_v2/pytorch/README.md index 9a0c523334..24a38a1ee0 100755 --- a/recommendation/dlrm_v2/pytorch/README.md +++ b/recommendation/dlrm_v2/pytorch/README.md @@ -78,30 +78,21 @@ CFLAGS="-std=c++14" python setup.py develop --user #### Download dataset through MLCFlow Automation ``` -mlcr get,preprocessed,dataset,criteo,_validation --outdirname= -j +mlcr get,preprocessed,dataset,criteo,_r2-downloader,_mlc,_validation --outdirname= -j ``` -#### Download the preprocessed dataset using Rclone. +#### Download the preprocessed dataset using the MLCommons R2 Downloader (more information about the MLC R2 Downloader, including how to run it on Windows, can be found [here](https://inference.mlcommons-storage.org)). -To run Rclone on Windows, you can download the executable [here](https://rclone.org/install/#windows). -To install Rclone on Linux/macOS/BSD systems, run: -``` -sudo -v ; curl https://rclone.org/install.sh | sudo bash -``` -Once Rclone is installed, run the following command to authenticate with the bucket: -``` -rclone config create mlc-inference s3 provider=Cloudflare access_key_id=f65ba5eef400db161ea49967de89f47b secret_access_key=fbea333914c292b854f14d3fe232bad6c5407bf0ab1bebf78833c2b359bdfd2b endpoint=https://c2686074cb2caf5cbaf6d134bdba8b47.r2.cloudflarestorage.com -``` Prepare your dataset destination: -``` +``` shell cd $HOME/mlcommons/inference/recommendation/dlrm_v2/pytorch/ mkdir ./dataset && cd ./dataset mv ./ export DATA_DIR=./dataset ``` Download the dataset -``` -rclone copy mlc-inference:mlcommons-inference-wg-public/dlrm_preprocessed ./dataset -P +``` bash +bash <(curl -s https://raw.githubusercontent.com/mlcommons/r2-downloader/refs/heads/main/mlc-r2-downloader.sh) -d ./dataset https://inference.mlcommons-storage.org/metadata/dlrm-v2-preprocessed-dataset.uri ``` @@ -115,26 +106,19 @@ N/A | pytorch | <2GB | - #### Download model through MLCFlow Automation ``` -mlcr get,ml-model,get,ml-model,dlrm,_pytorch,_weight_sharded,_rclone --outdirname= -j +mlcr get,ml-model,get,ml-model,dlrm,_pytorch,_fp32,_weight_sharded,_r2-downloader --outdirname= -j ``` #### Manual method -The above command automatically runs a set of Rclone commands to download the data from a Cloudflare R2 bucket. However, if you'd like to run the Rclone commands manually, you can do so as follows: +The above command automatically runs a set of commands to download the data from a Cloudflare R2 bucket. However, if you'd like to run the commands manually, you can do so as follows: + +(More information about the MLC R2 Downloader, including how to run it on Windows, can be found [here](https://inference.mlcommons-storage.org)) -To run Rclone on Windows, you can download the executable [here](https://rclone.org/install/#windows). -To install Rclone on Linux/macOS/BSD systems, run: -``` -sudo -v ; curl https://rclone.org/install.sh | sudo bash -``` -Once Rclone is installed, run the following command to authenticate with the bucket: -``` -rclone config create mlc-inference s3 provider=Cloudflare access_key_id=f65ba5eef400db161ea49967de89f47b secret_access_key=fbea333914c292b854f14d3fe232bad6c5407bf0ab1bebf78833c2b359bdfd2b endpoint=https://c2686074cb2caf5cbaf6d134bdba8b47.r2.cloudflarestorage.com -``` You can then navigate in the terminal to your desired download directory and run the following command to download the model weights: -``` -rclone copy mlc-inference:mlcommons-inference-wg-public/model_weights ./model_weights -P +``` bash +bash <(curl -s https://raw.githubusercontent.com/mlcommons/r2-downloader/refs/heads/main/mlc-r2-downloader.sh) https://inference.mlcommons-storage.org/metadata/dlrm-v2-model-weights.uri ``` #### (optional) diff --git a/speech2text/accuracy_eval.py b/speech2text/accuracy_eval.py index eb6cc53299..c356ab6398 100644 --- a/speech2text/accuracy_eval.py +++ b/speech2text/accuracy_eval.py @@ -57,12 +57,12 @@ "x", "y", "z", - "'", + "'", "0", "1", "2", "3", - "4", + "4", "5", "6", "7", diff --git a/speech2text/reference_SUT.py b/speech2text/reference_SUT.py index 63d491a00f..0b2f02c490 100644 --- a/speech2text/reference_SUT.py +++ b/speech2text/reference_SUT.py @@ -90,12 +90,12 @@ def get_start_cores(start_cores="0"): "x", "y", "z", - "'", + "'", "0", "1", "2", "3", - "4", + "4", "5", "6", "7", diff --git a/text_to_image/README.md b/text_to_image/README.md index e4e4d75af5..4c723c49c3 100644 --- a/text_to_image/README.md +++ b/text_to_image/README.md @@ -62,29 +62,22 @@ mlcr get,ml-model,sdxl,_fp32,_rclone --outdirname=$MODEL_PATH ``` #### Manual method -The above command automatically runs a set of Rclone commands to download the data from a Cloudflare R2 bucket. However, if you'd like to run the Rclone commands manually, you can do so as follows: +The above command automatically runs a set of commands to download the data from a Cloudflare R2 bucket. However, if you'd like to run the commands manually, you can do so as follows: -To run Rclone on Windows, you can download the executable [here](https://rclone.org/install/#windows). -To install Rclone on Linux/macOS/BSD systems, run: -``` -sudo -v ; curl https://rclone.org/install.sh | sudo bash -``` -Once Rclone is installed, run the following command to authenticate with the bucket: -``` -rclone config create mlc-inference s3 provider=Cloudflare access_key_id=f65ba5eef400db161ea49967de89f47b secret_access_key=fbea333914c292b854f14d3fe232bad6c5407bf0ab1bebf78833c2b359bdfd2b endpoint=https://c2686074cb2caf5cbaf6d134bdba8b47.r2.cloudflarestorage.com -``` -You can then navigate in the terminal to your desired download directory and run the following commands to download the checkpoints: +(More information about the MLC R2 Downloader, including how to run it on Windows, can be found [here](https://inference.mlcommons-storage.org)) + +Navigate in the terminal to your desired download directory and run the following commands to download the checkpoints: ``` cd $MODEL_PATH ``` **`fp32`** ``` -rclone copy mlc-inference:mlcommons-inference-wg-public/stable_diffusion_fp32 $MODEL_PATH -P +bash <(curl -s https://raw.githubusercontent.com/mlcommons/r2-downloader/refs/heads/main/mlc-r2-downloader.sh) -d $MODEL_PATH https://inference.mlcommons-storage.org/metadata/stable-diffusion-xl-1-0-fp32-checkpoint.uri ``` **`fp16`** ``` -rclone copy mlc-inference:mlcommons-inference-wg-public/stable_diffusion_fp16 $MODEL_PATH -P +bash <(curl -s https://raw.githubusercontent.com/mlcommons/r2-downloader/refs/heads/main/mlc-r2-downloader.sh) -d $MODEL_PATH https://inference.mlcommons-storage.org/metadata/stable-diffusion-xl-1-0-fp16-checkpoint.uri ``` ### Download validation dataset