clarify where to put file paths (e.g ml-25m/ratings.csv)

[pdurbin]

During the 2024-03-20 Crossaint Task Force meeting I asked where to put file paths (e.g. "ml-25m" for "ml-25m/ratings.csv" and @benjelloun said to go ahead and create an issue to clarify the spec.

I understand that the spec is pretty clear in the case where a zip file is available and contentUrl can be used to show the paths to files within the zip. Here's an example from https://github.com/mlcommons/croissant/blob/v1.0.5/datasets/1.0/movielens/metadata.json that shows a file path of "ml-25m/ratings.csv":

  "distribution": [
    {
      "@type": "cr:FileObject",
      "@id": "ml-25m-archive",
      "name": "ml-25m-archive",
      "contentUrl": "https://files.grouplens.org/datasets/movielens/ml-25m.zip",
      "encodingFormat": "application/zip",
      "sha256": "8b21cfb7eb1706b4ec0aac894368d90acf26ebdfb6aced3ebd4ad5bd1eb9c6aa"
    },
    {
      "@type": "cr:FileObject",
      "@id": "ratings-table",
      "name": "ratings-table",
      "containedIn": {
        "@id": "ml-25m-archive"
      },
      "contentUrl": "ml-25m/ratings.csv",
      "encodingFormat": "text/csv"
    },

However, while Dataverse often can provide a zip of all files in a dataset, increasingly files are large and zipping is expensive, so we plan to continue using contentUrl for direct links to the files. (Besides, why download an entire zip if you only need one file?) I say continue because to support Google Dataset Search we already provide the following, for example, in our Schema.org output:

{
  "@type": "DataDownload",
  "name": "2023-01-03.tab",
  "fileFormat": "text/tab-separated-values",
  "contentSize": 21865,
  "description": "Information on known Harvard repositories on GitHub, such as the number of stars, programming language, day last updated, number of open issues, size, number of forks, repository URL, create date, and description.",
  "contentUrl": "https://dataverse.harvard.edu/api/access/datafile/6867331"
}

So, if not contentUrl, which field should I use for the file path? Thanks!

[benjelloun]

Just to make sure I understand the issue correctly: You have an organized set of files, with directories, etc. You would like to provide them individually as FileObjects instead of a zip file, and the contentURL of each file does not contain their path, but uses a flat structure with an identifier instead, so you need a way to represent path to the file in the original directory structure. Is that right?

Assuming my understanding is correct, how about encoding the path in the @id of each fileObject? e.g.,
"@id": "path/to/2023-01-03.tab"? This also helps ensure that each @id is unique, which is a requirement.

[pdurbin]

@benjelloun yes, you understand our situation perfectly.

Thanks for the suggestion. I implemented it here: gdcc/dataverse-exporters@52c9e72

Should I go ahead and close this issue or should we use it to add something to the spec about file paths?

Thanks again! ❤️

[benjelloun]

I think we can close this issue now, and reopen if you run into more issues with file paths. Thanks!

[pdurbin]

@benjelloun hi! Can you please reopen this issue? I don't have permission.

Unfortunately we didn't think about spaces in file names. 😅

I'm getting this error from the validator (mlcroissant==1.0.22):

mlcroissant._src.core.issues.ValidationError: Found the following 1 error(s) during the validation:
- The dataset contains a wrong @id: 'Online Appendix.pdf'. Note that currently we do not support @ids containing whitespaces (not even if URL-escaped).

This was originally reported here:

• support id containing whitespaces gdcc/exporter-croissant#22

Here's the offending id mentioned in the error:

{
  "@type": "cr:FileObject",
  "@id": "Online Appendix.pdf",
  "name": "Online Appendix.pdf",
  "encodingFormat": "application/pdf",
  "md5": "4b2e636d327948e1fa8114a88dc5a390",
  "contentSize": "318081",
  "description": "",
  "contentUrl": "https://dataverse.harvard.edu/api/access/datafile/4818878"
}

Here's the complete Croissant JSON: Z9UB1H-v2.0.json. It comes from https://dataverse.harvard.edu/api/v1/datasets/export?exporter=croissant&persistentId=doi:10.7910/DVN/Z9UB1H . The dataset is at version 2.0 currently. I only mention this because we for published versions we only show the latest. Here's the dataset landing page: https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/Z9UB1H

Any suggestions for me? Thanks!

[benjelloun]

Re-opening the issue.

I'm not crazy about my earlier suggestion to encode the path in the id...

Can you say a bit more about the purpose of providing the path to the file?

If it is general enough, maybe we can add a dedicated property to FileObject.

[pdurbin]

@benjelloun certainly! In Dataverse, researchers can optionally organize their files into folders like "code", "data", and "docs". They can use whatever names they want and nest as deeply as they want. It's a very popular feature. Here's a simply example from my dataset:

When these files are downloaded as a zip, the folder hierarchy is preserved:

Even when files are not downloaded a zip, such as by integrations, we encourage services to pay attention to file paths (advertised in Croissant and other metadata formats) and place files in the appropriate folders.

Often, code in the dataset is looking for a specific path to data, which is important for reproducibility.

A dedicated FileObject property sounds lovely!

[pdurbin]

@benjelloun et al. I'm working on adding descriptive statistics (#640) by following https://docs.mlcommons.org/croissant/docs/croissant-spec-1.1.html#application-representing-descriptive-statistics which was added in #990 based on work by @kulnor.

I'm afraid the same @id problem will happen with descriptive statistics. That is, in the following example...

"annotation": [
  {
    "@id": "person/age/mean",
    "value": 42.51,
    "dataType": {
      "@type": "sc:DefinedTerm",
      "termCode": "ArithmeticMean",
      "name": "Arithmetic Mean",
      "@id": "ddi-stats:7975ed0",
      "inDefinedTermSet": "http://rdf-vocabulary.ddialliance.org/cv/SummaryStatisticType/2.1.2/"
    }
  },

... "person" is the name of the file, which could have a space in it. And "age" in the name of a column/variable, which could have a space in it. With whitespace in @ids like this, they'll fail validation.