datasets

[casperdcl]

something like this¹ (for use in notebook demos such as TomographicImaging/CIL-Demos#144):

data = cil.datasets.walnut.load()

btw internal logic could be roughly:

class Dataset:
    cache = os.getenv("CIL_DATASETS", "~/.cache/cil")
    @classmethod
    def load(cls):
        filename = brainweb.get_file(
            cls.source.rsplit("/", 1)[-1],
            cls.source,
            cache_dir=cls.cache)
        if filename.endswith(".zip"):
            return cls.load_from_zip(filename)
        raise ValueError(f"{filename}: unknown extension")

class walnut(Dataset):
    source = "https://zenodo.org/record/4822516/files/walnut.zip"

Footnotes

1. loosely inspired by https://keras.io/api/datasets approach ↩

[paskino]

We have a structure for data which could be used.

CIL/Wrappers/Python/cil/utilities/dataexample.py

Lines 39 to 47 in db5a2a6

	class DATA(object):
	@classmethod
	def dfile(cls):
	return None
	@classmethod
	def get(cls, size=None, scale=(0,1), **kwargs):
	ddir = kwargs.get('data_dir', data_dir)
	loader = TestData(data_dir=ddir)
	return loader.load(cls.dfile(), size, scale, **kwargs)