Skip to content

README.md

Latest commit

 

History

History
155 lines (118 loc) · 3.9 KB

README.md

File metadata and controls

155 lines (118 loc) · 3.9 KB

Data Preprocess in USB

This document provides the instructions for downloading and processing the datasets used in USB. Part of the datasets used in USB are allowed for re-distribution, and we provide download link directly for processed datasets of this part. The remaining datasets need to be downloaded from the original website and use the process code provided to be converted into the format used in USB.

Download Datasets

Most of the datasets used in USB can be download and used directly:

Download the data from this onedrive link: https://1drv.ms/u/c/af426f3217f6565a/ERsn-5wCWE1Bj-Rz4UyJEwABW4K10laA4cKG1xyqJmzpmg

cd ../
mkdir data & cd data
tar -xvf usbdata.tar.gz

The tar.gz file contains:

  • CV datasets: CIFAR-10, CIFAR-100, STL-10, SVHN, EuroSAT, TissueMNIST
  • NLP: Amazon Review, Yahoo Answers, Yelp Review, AG News
  • Audio: FSDNoisy18k, GTZAN, ESC50

You can now directly use these datasets by setting the data_dir argument in configuration files as "./data/"

The data structure should be like:

Semi-supervised-learning
├── semilearn
├── configs
├── train.py
├── data
│   ├── cifar10
│   │   ├── cifar-10-batches-py
│   ├── cifar100
│   │   ├── cifar-100-python
│   ├── stl10
│   │   ├── stl10_binary
│   ├── svhn
│   │   ├── train_32x32.mat
│   │   ├── test_32x32.mat
│   │   ├── extra_32x32.mat
│   ├── eurosat
│   │   ├── AnnualCrop
│   │   ├── Forest
│   │   ├── .....
│   ├── medmnist
│   │   ├── tissuemnist
│   ├── amazon_review
│   │   ├── train.json
│   │   ├── dev.json
│   │   ├── test.json
│   ├── ag_news
│   │   ├── train.json
│   │   ├── dev.json
│   │   ├── test.json
│   ├── yahoo_answers
│   │   ├── train.json
│   │   ├── dev.json
│   │   ├── test.json
│   ├── yelp_review
│   │   ├── train.json
│   │   ├── dev.json
│   │   ├── test.json
│   ├── fsdnoisy
│   │   ├── train.pkl
│   │   ├── dev.pkl
│   │   ├── test.pkl
│   ├── gtzan
│   │   ├── train.pkl
│   │   ├── dev.pkl
│   │   ├── test.pkl
│   ├── esc50
│   │   ├── train.pkl
│   │   ├── dev.pkl
│   │   ├── test.pkl

Process Raw Datasets

For the remaining part of the datasets, you need to download the raw data and process them using the provided scripts.

Semi-Aves

Download the raw data from "https://github.com/cvl-umass/semi-inat-2020#data-and-annotations" and "https://github.com/cvl-umass/ssl-evaluation/tree/main/data" into "./data/semi_fgvc"

Make sure the semi_fgvc folder in data follows:

├── data
│   ├── semi_fgvc
│   │   ├── annotation
│   │   ├── trainval_images
│   │   ├── test
│   │   ├── u_train_in
│   │   ├── u_train_out
│   │   ├── cub
│   │   ├── semi_aves

AclIMDB

Download the raw dataset from "https://ai.stanford.edu/~amaas/data/sentiment/"

Run

python preprocess/preprocess_aclimdb.py

Check the processed data follows:

├── data
│   ├── aclImdb
│   │   ├── train.json
│   │   ├── dev.json
│   │   ├── test.json

UrbanSound8k

Download the raw dataset from "https://urbansounddataset.weebly.com/urbansound8k.html"

Run

python preprocess/preprocess_urbansound.py

Check the processed data follows:

├── data
│   ├── urbansound8k
│   │   ├── train.pkl
│   │   ├── dev.pkl
│   │   ├── test.pkl
│   │   ├── id2label.json
│   │   ├── info.txt

SuperbKS

Run

python preprocess/preprocess_superb.py