docs(readme): restore comprehensive usage and API sections while keeping concise quick start

Author: wietzesuijker

README.md

@@ -1,137 +1,219 @@
 # EOPF GeoZarr
 
-Turn EOPF datasets into a GeoZarr-style Zarr v3 store. Keep the data values intact and add standard geospatial metadata, multiscale overviews, and per-variable dimensions.
+GeoZarr compliant data model for EOPF (Earth Observation Processing Framework) datasets.
 
-## Quick Start
+Turn EOPF datasets into a GeoZarr-style Zarr v3 store while:
+- Preserving native CRS (no forced TMS reprojection)
+- Adding CF + GeoZarr compliant metadata
+- Building /2 multiscale overviews
+- Writing robust, retry-aware band data with validation
 
-Install (uv):
+## Overview
 
+This library converts EOPF datatrees into GeoZarr-spec 0.4 aligned Zarr v3 stores without forcing web-mercator style tiling. It focuses on scientific fidelity (native CRS), robust metadata (CF + GeoZarr), and operational resilience (retry + completeness auditing) while supporting multiscale /2 overviews.
+
+## Key Features
+
+- **GeoZarr Specification Compliance** (0.4 features implemented)
+- **Native CRS Preservation** (UTM, polar, arbitrary projections)
+- **Multiscale /2 Overviews** (COG-style hierarchy as child groups)
+- **CF Conventions** (`standard_name`, `grid_mapping`, `_ARRAY_DIMENSIONS`)
+- **Resilient Writing** (band-by-band with retries & auditing)
+- **S3 & S3-Compatible Support** (AWS, OVH, MinIO, custom endpoints)
+- **Optional Parallel Processing** (local Dask cluster)
+- **Automatic Chunk Alignment** (prevents overlapping Dask/Zarr chunks)
+- **HTML Summary & Validation Tools**
+- **STAC & Benchmark Commands**
+- **Consolidated Metadata** (faster open)
+
+## GeoZarr Compliance Features
+
+- `_ARRAY_DIMENSIONS` attributes on all arrays
+- CF grid mapping variables with `GeoTransform`
+- Per-variable `grid_mapping` references
+- Multiscales metadata structure on parent groups
+- Native CRS tile matrix logic (no forced EPSG:3857)
+
+## Installation
+
+Stable:
+```bash
+pip install eopf-geozarr
+```
+
+Development (uv):
 ```bash
 uv sync --frozen
 uv run eopf-geozarr --help
 ```
 
-Or pip:
-
+Editable (pip):
 ```bash
-pip install -e .
+pip install -e .[dev]
 ```
 
-## Workflows
-
-For Argo / batch orchestration use: https://github.com/EOPF-Explorer/data-model-pipeline
+## Quick Start (CLI)
 
-## Convert
+Convert local → local:
+```bash
+eopf-geozarr convert input.zarr output_geozarr.zarr --groups /measurements/r10m /measurements/r20m
+```
 
 Remote → local:
-
 ```bash
-uv run eopf-geozarr convert \
+eopf-geozarr convert \
   "https://.../S2B_MSIL2A_... .zarr" \
   "/tmp/S2B_MSIL2A_..._geozarr.zarr" \
-  --groups /measurements/reflectance \
-  --verbose
+  --groups /measurements/reflectance --verbose
 ```
 
 Notes:
-- Parent groups auto-expand to leaf datasets.
-- Overviews use /2 coarsening; multiscales live on parent groups.
-- Defaults: Blosc Zstd, conservative chunking, metadata consolidation after write.
+- Parent groups auto-expand to leaf datasets
+- Overviews: /2 coarsening, attached at parent multiscales
+- Defaults: Blosc Zstd level 3, conservative chunking, metadata consolidation
 
-## S3
+Info / HTML / Validate:
+```bash
+eopf-geozarr info /tmp/..._geozarr.zarr --html report.html
+eopf-geozarr validate /tmp/..._geozarr.zarr
+```
 
-Env for S3/S3-compatible storage:
+## S3 Support
 
+Environment vars:
 ```bash
 export AWS_ACCESS_KEY_ID=...
 export AWS_SECRET_ACCESS_KEY=...
-export AWS_REGION=eu-west-1
-# Custom endpoint (OVH, MinIO, etc.)
-export AWS_ENDPOINT_URL=https://s3.your-endpoint.example
+export AWS_DEFAULT_REGION=eu-west-1
+export AWS_ENDPOINT_URL=https://s3.your-endpoint.example   # optional custom endpoint
 ```
 
-Write to S3:
-
+Write directly:
 ```bash
-uv run eopf-geozarr convert \
-  "https://.../S2B_MSIL2A_... .zarr" \
-  "s3://your-bucket/path/S2B_MSIL2A_..._geozarr.zarr" \
-  --groups /measurements/reflectance \
-  --verbose
+eopf-geozarr convert input.zarr s3://my-bucket/path/output_geozarr.zarr --groups /measurements/r10m
 ```
 
-## Info & Validate
+Features:
+- Credential validation before write
+- Custom endpoints (OVH, MinIO, etc.)
+- Retry logic around object writes
 
-Summary:
+## Parallel Processing with Dask
 
 ```bash
-uv run eopf-geozarr info "/tmp/S2B_MSIL2A_..._geozarr.zarr"
+eopf-geozarr convert input.zarr out.zarr --dask-cluster --verbose
 ```
+Benefits:
+- Local cluster auto-start & cleanup
+- Chunk alignment to prevent overlapping writes
+- Better memory distribution for large scenes
 
-HTML report:
+## Python API
 
-```bash
-uv run eopf-geozarr info "/tmp/S2B_MSIL2A_..._geozarr.zarr" --html /tmp/summary.html
+High-level dataset conversion:
+```python
+import xarray as xr
+from eopf_geozarr import create_geozarr_dataset
+
+dt = xr.open_datatree("path/to/eopf.zarr", engine="zarr")
+out = create_geozarr_dataset(
+    dt_input=dt,
+    groups=["/measurements/r10m", "/measurements/r20m"],
+    output_path="/tmp/out_geozarr.zarr",
+    spatial_chunk=4096,
+    min_dimension=256,
+    tile_width=256,
+)
 ```
 
-Validate (counts only real data vars, skips `spatial_ref`/`crs`):
+Selective writer usage (advanced):
+```python
+from eopf_geozarr.conversion.geozarr import GeoZarrWriter
+writer = GeoZarrWriter(output_path="/tmp/out.zarr", spatial_chunk=4096)
+# writer.write_group(...)
+```
+
+## API Reference
+
+`create_geozarr_dataset(dt_input, groups, output_path, spatial_chunk=4096, ...) -> xr.DataTree`
+: Produce a GeoZarr-compliant hierarchy.
+
+`setup_datatree_metadata_geozarr_spec_compliant(dt, groups) -> dict[str, xr.Dataset]`
+: Apply CF + GeoZarr metadata to selected groups.
+
+`downsample_2d_array(source_data, target_h, target_w) -> np.ndarray`
+: Block-average /2 overview generation primitive.
+
+`calculate_aligned_chunk_size(dimension_size, target_chunk_size) -> int`
+: Returns evenly dividing chunk to avoid overlap.
+
+## Architecture
 
-```bash
-uv run eopf-geozarr validate "/tmp/S2B_MSIL2A_..._geozarr.zarr"
 ```
+eopf_geozarr/
+  commands/        # CLI subcommands (convert, validate, info, stac, benchmark)
+  conversion/      # Core geozarr pipeline, helpers, multiscales, encodings
+  metrics.py       # Lightweight metrics hooks (optional)
+```
+
+## Contributing to GeoZarr Specification
 
-## Benchmark (optional)
+Upstream issue discussions influenced:
+- Arbitrary CRS preservation
+- Chunking performance & strategies
+- Multiscale hierarchy clarity
 
+## Benchmark & STAC Commands
+
+Benchmark:
 ```bash
-uv run eopf-geozarr benchmark "/tmp/..._geozarr.zarr" --samples 8 --window 1024 1024
+eopf-geozarr benchmark /tmp/out_geozarr.zarr --samples 8 --window 1024 1024
 ```
 
-## STAC
-
+STAC draft artifacts:
 ```bash
-uv run eopf-geozarr stac \
-  "/tmp/..._geozarr.zarr" \
-  "/tmp/..._collection.json" \
-  --bbox "minx miny maxx maxy" \
-  --start "YYYY-MM-DDTHH:MM:SSZ" \
-  --end "YYYY-MM-DDTHH:MM:SSZ"
+eopf-geozarr stac /tmp/out_geozarr.zarr /tmp/collection.json \
+  --bbox "minx miny maxx maxy" --start 2025-01-01T00:00:00Z --end 2025-01-31T23:59:59Z
 ```
 
-## Python API
+## What Gets Written
 
-```python
-from eopf_geozarr.conversion.geozarr import GeoZarrWriter
-from eopf_geozarr.validation.validate import validate_store
-from eopf_geozarr.info.summary import summarize
+- `_ARRAY_DIMENSIONS` per variable (deterministic axis order)
+- Per-variable `grid_mapping` referencing `spatial_ref`
+- Multiscales metadata on parent groups; /2 overviews
+- Blosc Zstd compression, conservative chunking
+- Consolidated metadata index
+- Band attribute propagation across levels
 
-src = "https://.../S2B_MSIL2A_... .zarr"
-dst = "/tmp/S2B_MSIL2A_..._geozarr.zarr"
+## Consolidated Metadata
 
-writer = GeoZarrWriter(src, dst, storage_options={})
-writer.write(groups=["/measurements/reflectance"], verbose=True)
+Improves open performance. Spec discussion ongoing; toggle by disabling consolidation if strict minimalism required.
 
-report = validate_store(dst)
-print(report.ok)
+## Troubleshooting
 
-tree = summarize(dst)
-print(tree["summary"])  # or write HTML via CLI
-```
+| Symptom | Cause | Fix |
+|---------|-------|-----|
+| Parent group empty | Only leaf groups hold arrays | Use `--groups` or rely on auto-expansion |
+| Overlapping chunk error | Misaligned dask vs encoding chunks | Allow auto chunk alignment or reduce spatial_chunk |
+| S3 auth failure | Missing env vars or endpoint | Export AWS_* vars / set AWS_ENDPOINT_URL |
+| HTML path is a directory | Provided path not file | A default filename is created inside |
 
-## What it writes
+## Development & Contributing
 
-- `_ARRAY_DIMENSIONS` per variable (correct axis order).
-- `grid_mapping = "spatial_ref"` per variable; `spatial_ref` holds CRS/georeferencing.
-- Multiscales on parent groups; /2 overviews.
-- Blosc Zstd compression; conservative chunking; consolidated metadata.
-- Overviews keep per-band attributes (grid_mapping reattached across levels).
+```bash
+git clone <repo-url>
+cd eopf-geozarr
+pip install -e '.[dev]'
+pre-commit install
+pytest
+```
 
-## Consolidated metadata
+Quality stack: Black, isort, Ruff, Mypy, Pytest, Coverage.
 
-Speeds up reads. Some tools note it isn’t in the core Zarr v3 spec yet; data stays valid. You can disable consolidation during writes or remove the index if preferred.
+## License & Acknowledgments
 
-## Troubleshooting
+Apache 2.0. Built atop xarray, zarr, dask; follows evolving GeoZarr specification.
 
-- Parent group shows no data vars: select leaves (CLI auto-expands).
-- S3 errors: check env vars and `AWS_ENDPOINT_URL` for custom endpoints.
-- HTML path is a directory: a default filename is created inside.
+---
+For questions or issues open a GitHub issue.