Skip to content

Validate non-PUDL data source metadata #1122

Description

@zaneselvans

Overview

In #1105 (see comment) we removed this repository's dependence on the main PUDL repo for data source metadata, which makes a lot of things simpler. However, it also meant that we lost access to the PUDL metadata Pydantic classes for data validation. This isn't such a big deal for the metadata we're pulling from the PUDL datapackage.json since we know it was validated on the other end, but there are dozens of data sources whose metadata is only defined in the pudl-archiver repository, and we should do at least some lightweight validation / explicit structuring of them. This could be either Python's @dataclass or possibly Pydantic, but we are not trying to duplicate the PUDL classes -- just set structural expectations and make sure that they're respected.

Data structures to define / validate

  • data sources (sources.py, nrelcambium.py, pudl.py)
  • licenses (constants.py)
  • contributors (constants.py, pudl.py)

Metadata

Metadata

Assignees

No one assigned

    Labels

    metadataManaging data about our data

    Type

    No type

    Projects

    Status
    Icebox

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions