VOParquet metadata for DP1 object catalog

[delucchi-cmu]

Generate VOParquet metadata for DP1 object catalog and check the size. It will help us to decide if we put the metadata to every parquet file or not.

[hombit]

Blocked by #568

[delucchi-cmu]

Prototyping #568, I was able to generate a _common_metadata file with "basic" VO parquet metadata (just names and types), and then also some expanded metadata that contains the units, UCD, and descriptions found here: https://sdm-schemas.lsst.io/dp1.html

• original file size: 141K
• with "basic" votable metadata: 386K
• expanded votable metadata: 800K

[hombit]

Thank you, @delucchi-cmu! I checked with our DP1 object_lc catalog and loaded RA, Dec, and healpix29 columns from the largest file we have, and the result table is ~300 kB in memory. So if we put the metadata in each Parquet file, then it may significantly contribute to the download size in this edge case.

[delucchi-cmu]

A little more detail on how I got these number, and how to generate the VOParquet metadata for DP1 object table:

attrs = pd.read_csv("object_table_attrs.csv")
field_ucds = {row["Column Name"]: row["UCD"] for _, row in attrs.iterrows() if pd.notna(row["UCD"])}
field_descriptions = {
    row["Column Name"]: row["Description"] for _, row in attrs.iterrows() if pd.notna(row["Description"])
}
field_units = {row["Column Name"]: row["Unit"] for _, row in attrs.iterrows() if pd.notna(row["Unit"])}
write_voparquet_in_common_metadata(
    "lsdb/benchmarks/data/object_collection/object_lc",
    field_units=field_units,
    field_ucds=field_ucds,
    field_descriptions=field_descriptions,
)

Using attached CSV file, that has fields from object and forced source tables here: https://sdm-schemas.lsst.io/dp1.html

object_table_attrs.csv

Using the _common_metadata from the object lightcurve catalog we have inside the LSDB benchmarks directory.

And using the write_voparquet_in_common_metadata method implemented in #591