fix: discover optional quicklooks from stac

Author: wietzesuijker

docs/quicklook-crs-investigation.md

@@ -0,0 +1,36 @@
+# GeoZarr Quicklook CRS Investigation
+
+## TL;DR
+- Quicklook assets are optional in upstream CPM publications; discovery must tolerate their absence.
+- `scripts/convert.py` now infers quicklook groups only when the STAC item advertises them and soft-fails otherwise.
+- Latest `data-model` CRS fallback ensures `/quality/l2a_quicklook` inherits the reflectance CRS when it is not encoded natively.
+
+## What Broke
+- Sentinel-2 GeoZarr releases tagged `cpm_v262` omitted the `/quality/l2a_quicklook` groups even though STAC assets still reference quicklooks.
+- The convert workflow assumed quicklooks would always be published, so the register step failed on empty groups.
+- Quicklook datasets that were published also lacked CRS metadata, blocking downstream tiling.
+
+## Investigation Highlights
+- Sampled six months of Sentinel-2 GeoZarr items across tiles and orbits; roughly half of the sampled items were missing quicklook data despite STAC advertising the asset.
+- Confirmed that older `cpm_v256` items exhibit the same gap, so the issue predates the recent regression.
+- Verified that STAC continues to list quicklook assets regardless of the dataset's presence, so workflow logic must perform existence checks.
+
+> Sampling notes: raw JSON exports collected during the investigation are intentionally left out of the repository to keep the diff focused. They are available in the shared evidence bucket if deeper inspection is required.
+
+## Fixes Landed
+1. `scripts/convert.py`
+   - Derives quicklook group names directly from the STAC assets.
+   - Short-circuits when the quicklook dataset is absent instead of failing the run.
+   - Logs actionable diagnostics for missing quicklook groups and for mismatched CRS.
+2. `data-model` GeoZarr conversion
+   - Adds a sibling CRS fallback so quicklook datasets inherit the reflectance CRS.
+   - Ensures the CRS is propagated to variable attributes when inferred.
+
+## Verification
+- `uv run -q pytest -q tests/test_cli_e2e.py::TestCLIEndToEnd::test_cli_convert_real_sentinel2_data`
+- Manual rerun of the devseed Argo workflow `geozarr-convert-quicklook`, confirming successful completion and STAC registration without quicklook datasets.
+- Spot-checked freshly produced GeoZarr stores to confirm quicklook datasets (when present) now carry CRS metadata.
+
+## Follow-Up
+- Monitor CPM release notes for clarity on quicklook publication cadence.
+- Backfill missing quicklooks only when explicitly prioritized; convert workflow can now operate without them.

scripts/convert.py

@@ -56,24 +56,58 @@ def get_config(collection_id: str) -> dict:
     return CONFIGS.get(prefix, CONFIGS["sentinel-2"]).copy()
 
 
-def get_zarr_url(stac_item_url: str) -> str:
-    """Get Zarr asset URL from STAC item (priority: product, zarr, any .zarr)."""
+def get_zarr_asset_info(stac_item_url: str) -> tuple[str, dict[str, dict]]:
+    """Fetch STAC item and extract Zarr URL along with assets."""
+
     with httpx.Client(timeout=30.0, follow_redirects=True) as client:
-        assets = client.get(stac_item_url).raise_for_status().json().get("assets", {})
+        item = client.get(stac_item_url).raise_for_status().json()
+
+    assets: dict[str, dict] = item.get("assets", {})
 
     # Try priority assets first
-    for key in ["product", "zarr"]:
+    for key in ("product", "zarr"):
         if key in assets and (href := assets[key].get("href")):
-            return str(href)
+            return str(href), assets
 
     # Fallback: any asset with .zarr in href
     for asset in assets.values():
         if ".zarr" in asset.get("href", ""):
-            return str(asset["href"])
+            return str(asset["href"]), assets
 
     raise RuntimeError("No Zarr asset found in STAC item")
 
 
+def _quicklook_groups_from_assets(assets: dict[str, dict]) -> list[str]:
+    if not assets:
+        return []
+
+    groups: list[str] = []
+    for asset in assets.values():
+        href = asset.get("href", "")
+        if "/quality/l2a_quicklook/" not in href or ".zarr/" not in href:
+            continue
+        rel = href.split(".zarr/", 1)[1].split("?", 1)[0].split(":", 1)[0]
+        rel = rel.strip("/")
+        if not rel:
+            continue
+        group = rel.rsplit("/", 1)[0] if "/" in rel else rel
+        group_path = f"/{group.strip('/')}"
+        if group_path not in groups:
+            groups.append(group_path)
+    return groups
+
+
+def _merge_quicklook_groups(default_groups: list[str], assets: dict[str, dict]) -> list[str]:
+    discovered = _quicklook_groups_from_assets(assets)
+    if not discovered:
+        return default_groups
+
+    merged = [group for group in default_groups if not group.startswith("/quality/l2a_quicklook")]
+    merged.extend(path for path in discovered if path not in merged)
+    logger.debug("Using quicklook groups derived from STAC assets: %s", discovered)
+    return merged
+
+
 # === Conversion Workflow ===
 
 
@@ -107,13 +141,19 @@ def run_conversion(
     logger.info(f"   Collection: {collection}")
 
     # Resolve source: STAC item or direct Zarr URL
-    zarr_url = get_zarr_url(source_url) if "/items/" in source_url else source_url
+    item_assets: dict[str, dict] = {}
+    if "/items/" in source_url:
+        zarr_url, item_assets = get_zarr_asset_info(source_url)
+    else:
+        zarr_url = source_url
     logger.info(f"   Source: {zarr_url}")
 
     # Get config and apply overrides
     config = get_config(collection)
     if groups:
         config["groups"] = groups.split(",")
+    else:
+        config["groups"] = _merge_quicklook_groups(config["groups"], item_assets)
     if spatial_chunk is not None:
         config["spatial_chunk"] = spatial_chunk
     if tile_width is not None:
@@ -124,6 +164,7 @@ def run_conversion(
     logger.info(
         f"   Parameters: chunk={config['spatial_chunk']}, tile={config['tile_width']}, sharding={config['enable_sharding']}"
     )
+    logger.debug("   Groups to convert: %s", ", ".join(config["groups"]))
 
     # Construct output path and clean existing
     output_url = f"s3://{s3_output_bucket}/{s3_output_prefix}/{collection}/{item_id}.zarr"