Skip to content

refactor(datalake): redesign clean result layout, move to columnar crawl format, and support many crawlers per bucket #364

Description

@thecaffiend

Background

The result-clean bucket layout and table-naming scheme were built quickly during the bactopia-only era, when there was a single result source. The crawler config prefix becomes the Glue table-name prefix and one crawler walks the entire bucket, so every top-level folder becomes a result_<folder> table sharing one crawl scope, schedule, classifiers, and excludes.

Adding a second result source (the caerbannog consumer ETL in #363) makes this coupling a problem: distinct datasets are forced under one result_ namespace and one crawler with no isolation. The current approach works for the demo but will not scale as result sources multiply.

Scope

  • Redesign the clean result bucket layout and table-naming so multiple independent result sources coexist without a shared flat result_ namespace (per-source prefixes/namespacing).
  • Move crawled output to a columnar format (Parquet, and evaluate Iceberg) instead of CSV. This is intended as a single coordinated switch across the whole datalake, not a per-source change. (No pre-existing tracking issue was found for the columnar move; folding it in here.)
  • Support many crawlers per crawled bucket. Today configure_bucket builds exactly one DataCrawler per bucket from a single crawler: config block; extend the config schema, configure_bucket, and DataCrawler to allow a list of crawlers per bucket so each result source can have its own schedule, classifiers, excludes, and table prefix.

Notes

Refs #363

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1Almost highest priority

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions