Types in mlcroissant

[marcenacp]

Current problem:

In Croissant JSON-LD, you can write:

{
    "dataType": [
      "sc:URL",
      "wd:Q3985153"
  ]
}

So types in Croissant mix:

• "Programmatic" data types (str, bool, etc).
• "Semantic" data types (URL, image, bounding box, etc).

What we've implemented so far:

So far, sc:Field have two attributes:

• field.data_types is a mirror of the JSON syntax. So in the previous example:

data_types = [URIRef("https://schema.org/URL"), URIRef("https://www.wikidata.org/wiki/Q3985153")]

• field.data_type is the actual programmatic type for scalars, or the semantic type for more complex features (image). Programmatic types use built-in Python types (str, bool, int). So in the previous example:

data_type = str

Proposal:

We propose to define mlc.dtypes based on np-dtypes:

np.int64
np.str_
np.bool_

class Image:
  dtype: np.dtype

class BoundingBox:
  dtype: np.dtype

On client-side, this allows to write:

import mlcroissant as mlc

if field.data_type is np.bool_:
  return bool
elif field.data_type is np.str_:
  return str
elif isinstance(field.data_type, mlc.dtypes.BoundingBox):
  return tfds.features.BBoxFeature(dtype=field.data_type.dtype)

[ccl-core]

+1 to the proposal!