Overview
In #1105 (see comment) we removed this repository's dependence on the main PUDL repo for data source metadata, which makes a lot of things simpler. However, it also meant that we lost access to the PUDL metadata Pydantic classes for data validation. This isn't such a big deal for the metadata we're pulling from the PUDL datapackage.json since we know it was validated on the other end, but there are dozens of data sources whose metadata is only defined in the pudl-archiver repository, and we should do at least some lightweight validation / explicit structuring of them. This could be either Python's @dataclass or possibly Pydantic, but we are not trying to duplicate the PUDL classes -- just set structural expectations and make sure that they're respected.
Data structures to define / validate
- data sources (
sources.py, nrelcambium.py, pudl.py)
- licenses (
constants.py)
- contributors (
constants.py, pudl.py)
Overview
In #1105 (see comment) we removed this repository's dependence on the main PUDL repo for data source metadata, which makes a lot of things simpler. However, it also meant that we lost access to the PUDL metadata Pydantic classes for data validation. This isn't such a big deal for the metadata we're pulling from the PUDL
datapackage.jsonsince we know it was validated on the other end, but there are dozens of data sources whose metadata is only defined in thepudl-archiverrepository, and we should do at least some lightweight validation / explicit structuring of them. This could be either Python's@dataclassor possibly Pydantic, but we are not trying to duplicate the PUDL classes -- just set structural expectations and make sure that they're respected.Data structures to define / validate
sources.py,nrelcambium.py,pudl.py)constants.py)constants.py,pudl.py)