Background
The result-clean bucket layout and table-naming scheme were built quickly during the bactopia-only era, when there was a single result source. The crawler config prefix becomes the Glue table-name prefix and one crawler walks the entire bucket, so every top-level folder becomes a result_<folder> table sharing one crawl scope, schedule, classifiers, and excludes.
Adding a second result source (the caerbannog consumer ETL in #363) makes this coupling a problem: distinct datasets are forced under one result_ namespace and one crawler with no isolation. The current approach works for the demo but will not scale as result sources multiply.
Scope
- Redesign the clean result bucket layout and table-naming so multiple independent result sources coexist without a shared flat
result_ namespace (per-source prefixes/namespacing).
- Move crawled output to a columnar format (Parquet, and evaluate Iceberg) instead of CSV. This is intended as a single coordinated switch across the whole datalake, not a per-source change. (No pre-existing tracking issue was found for the columnar move; folding it in here.)
- Support many crawlers per crawled bucket. Today
configure_bucket builds exactly one DataCrawler per bucket from a single crawler: config block; extend the config schema, configure_bucket, and DataCrawler to allow a list of crawlers per bucket so each result source can have its own schedule, classifiers, excludes, and table prefix.
Notes
Refs #363
Background
The
result-cleanbucket layout and table-naming scheme were built quickly during the bactopia-only era, when there was a single result source. The crawler configprefixbecomes the Glue table-name prefix and one crawler walks the entire bucket, so every top-level folder becomes aresult_<folder>table sharing one crawl scope, schedule, classifiers, and excludes.Adding a second result source (the caerbannog consumer ETL in #363) makes this coupling a problem: distinct datasets are forced under one
result_namespace and one crawler with no isolation. The current approach works for the demo but will not scale as result sources multiply.Scope
result_namespace (per-source prefixes/namespacing).configure_bucketbuilds exactly oneDataCrawlerper bucket from a singlecrawler:config block; extend the config schema,configure_bucket, andDataCrawlerto allow a list of crawlers per bucket so each result source can have its own schedule, classifiers, excludes, and table prefix.Notes
result-cleancrawler with distinct top-level output folders as a low-churn demo path, deferring this redesign here.Refs #363