Skip to content

Commit bfa3e25

Browse files
smrgeoinfoclaude
andcommitted
Update README, AGENTS, user guide, memory for TAPP pipeline + inference helper
README: - Tool listing now mentions --pub filter on the first two pipeline scripts. - New "Publication migration helper" subsection pointing at tools/interpret_pub_analytes.py and the side xlsx + JSON review outputs at docs/TAPP_EPMA_filled-interp.xlsx and build/interp-review/. AGENTS.md: - Pipeline section's CLI synopsis now shows the --pub flag with a one-line explanation: schema/catalog artifacts always rebuild; --pub limits which per-publication examples regen. Useful when migrating pubs one at a time. - New "Heuristic analyte-axis inference (publication migration helper)" subsection covering what interpret_pub_analytes.py parses (rows 32/48/59/64), what it writes (layout A side xlsx with adjacent <pub>-interp columns + paired JSON review files), and when to run it (publication-style free-text rows that need cleanup before regenerating production examples). docs/TAPP_TEMPLATE_GUIDE.md: - New "Migrating an existing publication-style spreadsheet" section before the Reference. Walks through running interpret_pub_analytes.py, what each output is, and the merge- back-then-rerun workflow. Notes that pubs with too-generic calibration text produce empty interp columns and need manual Target Element entry. memory: - New project_tapp_pipeline.md memory consolidating the pipeline conventions: 4 scripts + inference helper, --pub filter, | per-analyte delimiter, parameter routing by readOnly, parser-bug fix to per-tag readOnly extraction, share_or_write_catalog reuse-detection by $id ownership. - MEMORY.md index updated. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent 4187895 commit bfa3e25

4 files changed

Lines changed: 73 additions & 8 deletions

File tree

README.md

Lines changed: 18 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -93,15 +93,29 @@ Browse the building blocks at: https://usgin.github.io/geochemBuildingBlocks/
9393
The end-to-end pipeline for adding a new technique profile from a filled-in TAPP template spreadsheet. See [docs/TAPP_TEMPLATE_GUIDE.md](docs/TAPP_TEMPLATE_GUIDE.md) for what to put in the spreadsheet.
9494

9595
```
96-
python tools/build_TAPP_from_spreadsheet.py [TAPP_NAME] [XLSX_PATH] # 1. TAPP BB + catalogs
97-
python tools/build_detail_BB.py [TAPP_NAME] [XLSX_PATH] # 2. detail BB + parameterValues
98-
python tools/build_profile_BB.py [TAPP_NAME] # 3. profile BB scaffold
99-
python tools/build_dataset_template.py <tapp-instance.json> # 4. xlsx data-entry template
96+
python tools/build_TAPP_from_spreadsheet.py [TAPP_NAME] [XLSX_PATH] [--pub Pn]… # 1. TAPP BB + catalogs
97+
python tools/build_detail_BB.py [TAPP_NAME] [XLSX_PATH] [--pub Pn]… # 2. detail BB + parameterValues
98+
python tools/build_profile_BB.py [TAPP_NAME] # 3. profile BB scaffold
99+
python tools/build_dataset_template.py <tapp-instance.json> # 4. xlsx data-entry template
100100
[<out.xlsx>]
101101
```
102102

103103
All four scripts default to `empaTAPP` / `docs/TAPP_EPMA_filled.xlsx` for back-compat. The shared library at `tools/_tapp_lib.py` does the heavy lifting (parser, catalog emit helpers, scaffolders); the four drivers are thin wrappers.
104104

105+
`--pub <code>` (repeatable) on scripts 1 and 2 limits which publication-derived examples get regenerated — useful when migrating pub columns one at a time.
106+
107+
### Publication migration helper
108+
109+
```
110+
python tools/interpret_pub_analytes.py # heuristic analyte-axis inference
111+
```
112+
113+
Reads publication columns whose analyte axis isn't explicitly populated and infers it from rows 48 / 59 / 64 (Halogen Correction / Primary Calibration Standard / Typical Detection Limit). Writes:
114+
- `docs/TAPP_EPMA_filled-interp.xlsx` — side workbook with each `<pub>-interp` column inserted right after its source pub for side-by-side review.
115+
- `build/interp-review/example<empaTAPP|detailEMPA>-<pub>-interp.json` — paired review JSON instances built from the inferred data.
116+
117+
Review, merge interp values back into the source xlsx, then run the regular pipeline to produce production examples.
118+
105119
### Schema generation and resolution
106120

107121
- `tools/generate_profiles.py` — generates technique-specific profile building blocks from configuration data

agents.md

Lines changed: 21 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -145,13 +145,15 @@ Lives at `_sources/techniqueProtocols/tappDefinition/`. Was previously `geochemP
145145
The pipeline for going from a filled-in TAPP template spreadsheet to a published technique-specific profile. The shared library is `tools/_tapp_lib.py`; the four driver scripts are thin wrappers around `configure(tapp_name, xlsx_path)` + a single library entry point.
146146

147147
```
148-
build_TAPP_from_spreadsheet.py [TAPP_NAME] [XLSX] # → TAPP BB + analyteColumns/parameterTemplates/vocab catalogs
149-
build_detail_BB.py [TAPP_NAME] [XLSX] # → detail BB scaffold + parameterValues catalog + parametersConstraint.yaml
150-
build_profile_BB.py [TAPP_NAME] # → profiles/geochemProfiles/<short>Profile/ scaffold
151-
build_dataset_template.py <tapp-instance.json> # → xlsx with columns from analyteColumns, rows from defaultAnalytes
148+
build_TAPP_from_spreadsheet.py [TAPP_NAME] [XLSX] [--pub P0]… # → TAPP BB + analyteColumns/parameterTemplates/vocab catalogs
149+
build_detail_BB.py [TAPP_NAME] [XLSX] [--pub P0]… # → detail BB scaffold + parameterValues catalog + parametersConstraint.yaml
150+
build_profile_BB.py [TAPP_NAME] # → profiles/geochemProfiles/<short>Profile/ scaffold
151+
build_dataset_template.py <tapp-instance.json> # → xlsx with columns from analyteColumns, rows from defaultAnalytes
152152
[<out.xlsx>]
153153
```
154154

155+
`--pub <code>` (repeatable) on the first two scripts restricts which publication-derived examples get regenerated. Schema and shared-catalog artifacts always rebuild regardless. Use this when only some publication columns have been migrated to the pipe-delimited per-analyte convention and you don't want to overwrite the others. Example: `--pub P0 --pub P1`.
156+
155157
All four default to `empaTAPP` / `docs/TAPP_EPMA_filled.xlsx`. Templates and the user-facing guide live in `docs/`:
156158
- `docs/TAPP_EPMA_filled.xlsx` — canonical filled template (clone for new techniques).
157159
- `docs/TAPP_TEMPLATE_GUIDE.md` — explains the worksheet structure (column-by-column, impl-notes tag conventions, the readOnly:true → empaTAPP / readOnly:false → detailEMPA routing, hasPart additionalType pattern, vocab handling).
@@ -166,6 +168,21 @@ For each `analyteColumn:<name>` impl-notes tag → `techniqueProtocols/analyteCo
166168

167169
The parser fix on 2026-04-28 (`parse_impl()``tag_records`) extracts per-tag `readOnly` so a row carrying both a `property:` (readOnly:true) and a `parameter:` (readOnly:false) tag routes each correctly.
168170

171+
### Heuristic analyte-axis inference (publication migration helper)
172+
173+
`tools/interpret_pub_analytes.py` is a one-shot helper for migrating publication columns whose analyte axis isn't explicitly populated. It walks each pub column and tries to recover the analyte list from the cells that DO have signal:
174+
175+
- **Row 59 (Primary Calibration Standard Name)** — entries like `"Anorthite (Si Kα, Al Kα, Ca Kα); Albite (Na Kα); ..."` parsed for `(element, x-ray line, standard)` tuples.
176+
- **Row 64 (Typical Detection Limit)** — both `"Compound: value"` and `"<value> for X, Y, Z"` shapes; oxide compounds (`SiO2`, `Cr2O3`, `FeO`) have the element extracted.
177+
- **Row 48 (Halogen Correction on Oxygen)**`F`, `Cl`, `OH`, `CO2`, `S`, `H2O` mentions are treated as additional analytes.
178+
- **Row 32 (Target Element)** — explicit pipe- or comma-delimited list takes precedence over inference when present.
179+
180+
Outputs:
181+
- `docs/TAPP_EPMA_filled-interp.xlsx` — side workbook with each `<pub>-interp` column inserted right after its source pub for side-by-side review. Structural columns (Level / cdif-path / matchComment / impl notes) push to the right edge.
182+
- `build/interp-review/example<empaTAPP|detailEMPA>-<pub>-interp.json` — paired review JSON instances generated by feeding the interp column data through the existing `example_for_pub` builder.
183+
184+
Use case: after editing a pub column in the source xlsx with publication-style free text in row 59 / 64 / 48, run the interpreter to see what analyte axis is extractable, review the side xlsx and the JSON examples, then merge the cleaned-up interp column values back into the source pub. Pubs with no extractable signal (calibration text like "natural and synthetic materials") are skipped — fill in their Target Element row manually.
185+
169186
### Reuse-detection (`share_or_write_catalog`)
170187

171188
All catalog writes go through `share_or_write_catalog(path, data)` in `_tapp_lib.py`:

docs/TAPP_EPMA_filled-interp.xlsx

12.3 KB
Binary file not shown.

docs/TAPP_TEMPLATE_GUIDE.md

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -274,6 +274,40 @@ For a brand-new technique TAPP (let's say XRD):
274274

275275
---
276276

277+
## Migrating an existing publication-style spreadsheet
278+
279+
If you're starting from an xlsx where the publication columns hold free-text descriptions instead of clean pipe-delimited data, the inference helper saves a lot of hand work:
280+
281+
```bash
282+
python tools/interpret_pub_analytes.py
283+
```
284+
285+
It scans every pub column and tries to recover the analyte axis from:
286+
287+
- **Row 32 (Target Element)** — used directly when explicitly populated (pipe- or comma-delim).
288+
- **Row 59 (Primary Calibration Standard Name)** — entries like `"Anorthite (Si Kα, Al Kα, Ca Kα); Albite (Na Kα); …"` are parsed for `(element, x-ray line, standard)` triples.
289+
- **Row 64 (Typical Detection Limit)**`"Compound: value"` and `"<value> for X, Y, Z"` shapes are both recognised. Oxide formulas (`SiO2`, `Cr2O3`, `FeO`, `P2O5`, …) have the element extracted (the non-O part).
290+
- **Row 48 (Halogen Correction on Oxygen)** — mentions of `F`, `Cl`, `OH`, `CO2`, `S`, `H2O` get treated as additional analytes.
291+
292+
Outputs:
293+
294+
- `docs/TAPP_EPMA_filled-interp.xlsx` — a side workbook where each pub column is followed immediately by a `<pub>-interp` column carrying:
295+
- Row 32: inferred analyte list (pipe-delim).
296+
- Row 40: x-ray line per analyte.
297+
- Row 59: calibration standard per analyte.
298+
- Row 64: detection limit per analyte.
299+
- All other rows: verbatim copy of the source pub.
300+
- `build/interp-review/exampleempaTAPP-<pub>-interp.json` and `exampledetailEMPA-<pub>-interp.json` — paired JSON instances produced by feeding the interp column data through the regular `example_for_pub` builder. Easier to spot-check the inferred analyteTemplate / defaultAnalytes than to read the spreadsheet rows directly.
301+
302+
Workflow:
303+
304+
1. Open the side xlsx in Excel — each `<pub>-interp` column sits next to its source pub for side-by-side review.
305+
2. Open the corresponding JSON files to see how the inference produces the analyteTemplate.
306+
3. When satisfied, copy the `<pub>-interp` column's values into the source pub column in `TAPP_EPMA_filled.xlsx` (replacing the original free-text rows).
307+
4. Re-run the regular pipeline (`build_TAPP_from_spreadsheet.py` / `build_detail_BB.py`) to produce production examples.
308+
309+
Pubs whose calibration-standard text is too generic (e.g. "natural and synthetic materials") to extract an element list will produce an empty interp column — fill in their Target Element row manually before re-running.
310+
277311
## Reference
278312

279313
- `tools/_tapp_lib.py` — the shared library; everything here is parameterized on `TAPP_NAME` set by `configure(tapp_name, xlsx)`.

0 commit comments

Comments
 (0)