Remove rclone references and update download instructions for DeepSeek-R1, Llama 3.1 8b, and Whisper

- Replace rclone-based download instructions with new MLCommons downloader infrastructure
- Update DeepSeek-R1, Llama 3.1 8b, and Whisper READMEs to use https://inference.mlcommons-storage.org
- Maintain MLCFlow automation commands alongside native download methods
- Add file size information for each download
- Include -d flag documentation for custom download directories

Fixes #2265

Author: anivar

language/deepseek-r1/README.md

@@ -33,21 +33,20 @@ mlcr get,dataset,whisper,_preprocessed,_mlc,_rclone --outdirname=<path to downlo
 
 **Using Native method**
 
-You can use Rclone to download the preprocessed dataset from a Cloudflare R2 bucket.
+Download the preprocessed dataset using the MLCommons downloader:
 
-To run Rclone on Windows, you can download the executable [here](https://rclone.org/install/#windows).
-To install Rclone on Linux/macOS/BSD systems, run:
-```
-sudo -v ; curl https://rclone.org/install.sh | sudo bash
-```
-Once Rclone is installed, run the following command to authenticate with the bucket:
-```
-rclone config create mlc-inference s3 provider=Cloudflare access_key_id=f65ba5eef400db161ea49967de89f47b secret_access_key=fbea333914c292b854f14d3fe232bad6c5407bf0ab1bebf78833c2b359bdfd2b endpoint=https://c2686074cb2caf5cbaf6d134bdba8b47.r2.cloudflarestorage.com
+```bash
+bash <(curl -s https://raw.githubusercontent.com/mlcommons/r2-downloader/refs/heads/main/mlc-r2-downloader.sh) \
+  https://inference.mlcommons-storage.org/deepseek-r1%2Fdataset.json
 ```
-You can then navigate in the terminal to your desired download directory and run the following command to download the dataset:
 
-```
-rclone copy mlc-inference:mlcommons-inference-wg-public/deepseek_r1/datasets/mlperf_deepseek_r1_dataset_4388_fp8_eval.pkl ./ -P
+This will download the dataset file `mlperf_deepseek_r1_dataset_4388_fp8_eval.pkl` (~163MB).
+
+To specify a custom download directory, use the `-d` flag:
+```bash
+bash <(curl -s https://raw.githubusercontent.com/mlcommons/r2-downloader/refs/heads/main/mlc-r2-downloader.sh) \
+  -d /path/to/download/directory \
+  https://inference.mlcommons-storage.org/deepseek-r1%2Fdataset.json
 ```
 
 ### Calibration
@@ -60,12 +59,20 @@ mlcr get,preprocessed,dataset,deepseek-r1,_calibration,_mlc,_rclone --outdirname
 
 **Using Native method**
 
-Download and install Rclone as described in the previous section.
-
-Then navigate in the terminal to your desired download directory and run the following command to download the dataset:
+Download the calibration dataset using the MLCommons downloader:
 
+```bash
+bash <(curl -s https://raw.githubusercontent.com/mlcommons/r2-downloader/refs/heads/main/mlc-r2-downloader.sh) \
+  https://inference.mlcommons-storage.org/deepseek-r1%2Fcalibration.json
 ```
-rclone copy mlc-inference:mlcommons-inference-wg-public/deepseek_r1/datasets/mlperf_deepseek_r1_calibration_dataset_500_fp8_eval.pkl ./ -P
+
+This will download the calibration dataset file `mlperf_deepseek_r1_calibration_dataset_500_fp8_eval.pkl`.
+
+To specify a custom download directory, use the `-d` flag:
+```bash
+bash <(curl -s https://raw.githubusercontent.com/mlcommons/r2-downloader/refs/heads/main/mlc-r2-downloader.sh) \
+  -d /path/to/download/directory \
+  https://inference.mlcommons-storage.org/deepseek-r1%2Fcalibration.json
 ```
 
 ## Docker

language/llama3.1-8b/README.md

@@ -137,18 +137,7 @@ Downloading llama3.1-8b model from Hugging Face will require an [**access token*
 
 ### Preprocessed
 
-You can use Rclone to download the preprocessed dataset from a Cloudflare R2 bucket.
-
-To run Rclone on Windows, you can download the executable [here](https://rclone.org/install/#windows).
-To install Rclone on Linux/macOS/BSD systems, run:
-```
-sudo -v ; curl https://rclone.org/install.sh | sudo bash
-```
-Once Rclone is installed, run the following command to authenticate with the bucket:
-```
-rclone config create mlc-inference s3 provider=Cloudflare access_key_id=f65ba5eef400db161ea49967de89f47b secret_access_key=fbea333914c292b854f14d3fe232bad6c5407bf0ab1bebf78833c2b359bdfd2b endpoint=https://c2686074cb2caf5cbaf6d134bdba8b47.r2.cloudflarestorage.com
-```
-You can then navigate in the terminal to your desired download directory and run the following command to download the dataset:
+Download the preprocessed datasets using the MLCommons downloader:
 
 #### Full dataset (datacenter) 
 
@@ -158,9 +147,11 @@ mlcr get,dataset,cnndm,_validation,_datacenter,_llama3,_mlc,_rclone --outdirname
 ```
 
 **Native method**
+```bash
+bash <(curl -s https://raw.githubusercontent.com/mlcommons/r2-downloader/refs/heads/main/mlc-r2-downloader.sh) \
+  https://inference.mlcommons-storage.org/llama3.1-8b%2Fcnn-eval.json
 ```
-rclone copy mlc-inference:mlcommons-inference-wg-public/llama3.1_8b/datasets/cnn_eval.json ./ -P
-```
+This will download `cnn_eval.json` (~267MB).
 
 #### 5000 samples (edge)
 
@@ -170,9 +161,11 @@ mlcr get,dataset,cnndm,_validation,_edge,_llama3,_mlc,_rclone --outdirname=<path
 ```
 
 **Native method**
+```bash
+bash <(curl -s https://raw.githubusercontent.com/mlcommons/r2-downloader/refs/heads/main/mlc-r2-downloader.sh) \
+  https://inference.mlcommons-storage.org/llama3.1-8b%2Fsample-cnn-eval-5000.json
 ```
-rclone copy mlc-inference:mlcommons-inference-wg-public/llama3.1_8b/datasets/sample_cnn_eval_5000.json ./ -P
-```
+This will download `sample_cnn_eval_5000.json` (~95MB).
 
 #### Calibration
 
@@ -182,14 +175,17 @@ mlcr get,dataset,cnndm,_calibration,_llama3,_mlc,_rclone --outdirname=<path to d
 ```
 
 **Native method**
+```bash
+bash <(curl -s https://raw.githubusercontent.com/mlcommons/r2-downloader/refs/heads/main/mlc-r2-downloader.sh) \
+  https://inference.mlcommons-storage.org/llama3.1-8b%2Fcnn-dailymail-calibration.json
 ```
-rclone copy mlc-inference:mlcommons-inference-wg-public/llama3.1_8b/datasets/cnn_dailymail_calibration.json ./ -P
-```
+This will download `cnn_dailymail_calibration.json` (~21MB).
 
-You can also download the calibration dataset from the Cloudflare R2 bucket by running the following command:
-
-```
-rclone copy mlc-inference:mlcommons-inference-wg-public/llama3.1_8b/cnn_eval.json ./ -P
+To specify a custom download directory for any of these, use the `-d` flag:
+```bash
+bash <(curl -s https://raw.githubusercontent.com/mlcommons/r2-downloader/refs/heads/main/mlc-r2-downloader.sh) \
+  -d /path/to/download/directory \
+  <URI>
 ```
 
 

speech2text/README.md

@@ -107,21 +107,20 @@ mlcr get,ml-model,whisper,_rclone,_mlc --outdirname=<path_to_download> -j
 
 **Official Model download using native method**
 
-You can use Rclone to download the preprocessed dataset from a Cloudflare R2 bucket.
+Download the Whisper model using the MLCommons downloader:
 
-To run Rclone on Windows, you can download the executable [here](https://rclone.org/install/#windows).
-To install Rclone on Linux/macOS/BSD systems, run:
-```
-sudo -v ; curl https://rclone.org/install.sh | sudo bash
-```
-Once Rclone is installed, run the following command to authenticate with the bucket:
-```
-rclone config create mlc-inference s3 provider=Cloudflare access_key_id=f65ba5eef400db161ea49967de89f47b secret_access_key=fbea333914c292b854f14d3fe232bad6c5407bf0ab1bebf78833c2b359bdfd2b endpoint=https://c2686074cb2caf5cbaf6d134bdba8b47.r2.cloudflarestorage.com
+```bash
+bash <(curl -s https://raw.githubusercontent.com/mlcommons/r2-downloader/refs/heads/main/mlc-r2-downloader.sh) \
+  https://inference.mlcommons-storage.org/whisper%2Fmodel.json
 ```
-You can then navigate in the terminal to your desired download directory and run the following command to download the model:
 
-```
-rclone copy mlc-inference:mlcommons-inference-wg-public/Whisper/model/ ./ -P
+This will download the Whisper model files (~25GB).
+
+To specify a custom download directory, use the `-d` flag:
+```bash
+bash <(curl -s https://raw.githubusercontent.com/mlcommons/r2-downloader/refs/heads/main/mlc-r2-downloader.sh) \
+  -d /path/to/download/directory \
+  https://inference.mlcommons-storage.org/whisper%2Fmodel.json
 ```
 
 ### External Download (Not recommended for official submission)
@@ -158,11 +157,20 @@ mlcr get,dataset,whisper,_preprocessed,_mlc,_rclone --outdirname=<path to downlo
 
 **Native method**
 
-Download and install rclone as decribed in the [MLCommons Download section](#mlcommons-download)
+Download the preprocessed dataset using the MLCommons downloader:
 
-You can then navigate in the terminal to your desired download directory and run the following command to download the dataset:
+```bash
+bash <(curl -s https://raw.githubusercontent.com/mlcommons/r2-downloader/refs/heads/main/mlc-r2-downloader.sh) \
+  https://inference.mlcommons-storage.org/whisper%2Fdataset.json
 ```
-rclone copy mlc-inference:mlcommons-inference-wg-public/Whisper/dataset/ ./ -P
+
+This will download the LibriSpeech dataset files (~4.6GB).
+
+To specify a custom download directory, use the `-d` flag:
+```bash
+bash <(curl -s https://raw.githubusercontent.com/mlcommons/r2-downloader/refs/heads/main/mlc-r2-downloader.sh) \
+  -d /path/to/download/directory \
+  https://inference.mlcommons-storage.org/whisper%2Fdataset.json
 ```
 
 ### Unprocessed