paper: refresh experiments and submission artifacts

Author: mo-tunn

submission/README.md

@@ -14,9 +14,9 @@ In this study, semantic chunks produced from long documents are modeled as knaps
 
 The selection task is formulated as a 0/1 Knapsack Problem. The exact dynamic programming solution is compared with heuristic or retrieval-oriented approaches such as top-k, budget-aware greedy selection, MMR, and prompt-compression baselines such as LLMLingua-2 in the QASPER experiments.
 
-The current scoring taxonomy is intentionally separated into baseline profiles (`cosine`, `hybrid`), the default evidence-selection profile (`evidence-hybrid`), a budget-stressed profile (`knapsack-aware`), a related-work proxy profile (`budgetmem-style`), and experimental profiles (`query-support`, `decision-aware`, `instruction-ami`). The `budgetmem-style` profile is an approximate feature-family baseline, not a reproduction of BudgetMem's learned policy. The `query-support` profile is general query/instruction-aware scoring and does not use LongBench-specific answer-option features. The `decision-aware` profile is for tasks with explicit candidate answers or decisions, and scores whether a chunk helps discriminate among candidates.
+The default chunker is structure-aware and semantic-aware: it preserves file, section, content-type, and code-symbol boundaries, while using adjacent block embedding similarity to split topic shifts inside compatible document sections. The production scoring path is intentionally narrow: `evidence-hybrid` is the only selectable scoring profile. Older profiles (`cosine`, `hybrid`, `knapsack-aware`, `budgetmem-style`, `query-support`, `decision-aware`) remain in `tokenpack.scoring_experimental` for ablation and historical comparison, not for default runs.
 
-The selector taxonomy is similarly conservative: `knapsack-redundancy` remains the main TokenPack selector used in the paper, while `knapsack-coverage` is an experimental grounded-generation ablation that favors complementary query coverage after redundancy adjustment.
+The selector taxonomy is similarly conservative: `budget-top-k` is now the main TokenPack selector used in the paper. In the paper text this is called hybrid-greedy because it combines evidence-hybrid scoring with a budget-feasible greedy fill. `knapsack`, `knapsack-redundancy`, and `knapsack-coverage` remain as algorithmic baselines and historical ablations.
 
 ## Folder Structure
 
@@ -36,8 +36,6 @@ The selector taxonomy is similarly conservative: `knapsack-redundancy` remains t
 - `results/knapsack_summary_table.tex`: LaTeX table used in the paper.
 - `results/knapsack_timeout_table.tex`: DP scalability table used in the paper.
 - `results/figures/*.png`: Runtime scaling, approximation gap, and DP scalability plots generated by the experiment script.
-- `gold/candidate_gold.jsonl`: Auto-proposed evidence dataset that is ready for human review, but should not be treated as fully validated gold evidence.
-- `gold/README.md`: Human review instructions for the candidate evidence dataset.
 - `source_code_manifest.md`: Explains where the main source code is located and how to run it.
 
 ## Quick Experiment Run
@@ -61,15 +59,15 @@ python -m pytest -p no:cacheprovider
 QASPER selector ablation, using a locally downloaded converted parquet file:
 
 ```powershell
-python submission\experiments\qasper_selector_eval.py --data-file C:\tmp\qasper-validation.parquet --backend hash --max-papers 40 --max-questions 200 --budget-ratios 0.05,0.10,0.20 --output-dir submission\results\qasper
+python submission\experiments\qasper_selector_eval.py --data-file C:\tmp\qasper-validation.parquet --max-papers 40 --max-questions 200 --budget-ratios 0.05,0.10,0.20 --output-dir submission\results\qasper
 ```
 
 If `--data-file` is omitted, the script tries to read the converted QASPER parquet URL from Hugging Face. A local parquet file is more reliable on restricted networks.
 
 QASPER cost-quality proxy:
 
 ```powershell
-python submission\experiments\qasper_cost_quality_curve.py --data-file C:\tmp\qasper-validation.parquet --backend hash --chunkers semantic-threshold --scorings hybrid --strategies budget-top-k,greedy-density,knapsack --budget-ratios 0.20,0.40,0.60,0.80,1.00 --candidate-pool 10000 --output-dir submission\results\qasper_cost_quality_widepool
+python submission\experiments\qasper_cost_quality_curve.py --data-file C:\tmp\qasper-validation.parquet --chunkers structure-aware --scorings evidence-hybrid --strategies production-rag,budget-top-k,greedy-density,knapsack,knapsack-redundancy --budget-ratios 0.50,0.70,0.80,1.00 --candidate-pool 300 --chunk-size-preset low-budget --output-dir submission\results\qasper_cost_quality_hybrid_greedy_rerun
 ```
 
 ## Building the LaTeX PDF

submission/TokenPack-paper.pdf

@@ -15,7 +15,7 @@
 if str(SRC) not in sys.path:
     sys.path.insert(0, str(SRC))
 
-from tokenpack.chunking import ParagraphGroupChunker, SemanticThresholdChunker, StructureAwareChunker
+from tokenpack.chunking import SemanticThresholdChunker, StructureAwareChunker
 from tokenpack.embeddings import EmbeddingCache, make_embedder
 from tokenpack.index import ChunkIndex, save_index
 from tokenpack.loaders import iter_supported_files, load_text_blocks
@@ -28,7 +28,7 @@
 DEFAULT_SOURCE = ROOT / "resources"
 DEFAULT_OUTPUT_DIR = ROOT / "submission" / "results"
 DEFAULT_WORK_DIR = ROOT / ".tokenpack" / "chunking-ablation"
-CHUNKERS = ["paragraph", "semantic-threshold", "structure-aware"]
+CHUNKERS = ["semantic-threshold", "structure-aware"]
 TOKEN_RE = re.compile(r"[A-Za-z][A-Za-z-]{3,}")
 SENTENCE_RE = re.compile(r"(?<=[.!?])\s+")
 STOPWORDS = {
@@ -67,7 +67,6 @@ class EvidenceTemplate:
 def main() -> int:
     parser = argparse.ArgumentParser(description="Compare chunking strategies under the same knapsack solver.")
     parser.add_argument("--source", default=str(DEFAULT_SOURCE))
-    parser.add_argument("--backend", default="hash", choices=["auto", "hash", "sentence-transformers"])
     parser.add_argument("--model", default="sentence-transformers/all-MiniLM-L6-v2")
     parser.add_argument("--budget-ratios", default="0.01,0.03,0.05")
     parser.add_argument("--budgets", help="Optional comma-separated absolute budgets; overrides --budget-ratios.")
@@ -95,7 +94,7 @@ def main() -> int:
     work_dir.mkdir(parents=True, exist_ok=True)
 
     token_counter = TokenCounter()
-    embedder = make_embedder(backend=args.backend, model_name=args.model, local_files_only=True)
+    embedder = make_embedder(model_name=args.model, local_files_only=True)
     blocks = _load_experiment_blocks(
         Path(args.source),
         max_documents=args.max_documents,
@@ -239,10 +238,16 @@ def _build_index(
     token_counter: TokenCounter,
 ) -> ChunkIndex:
     cache = EmbeddingCache(work_dir / f"{chunker_name}.embeddings.json")
-    if chunker_name == "paragraph":
-        chunker = ParagraphGroupChunker(target_tokens, min_tokens, max_tokens, token_counter=token_counter)
-    elif chunker_name == "structure-aware":
-        chunker = StructureAwareChunker(target_tokens, min_tokens, max_tokens, token_counter=token_counter)
+    if chunker_name == "structure-aware":
+        block_embeddings = cache.get_or_embed([block.text for block in blocks], embedder)
+        chunker = StructureAwareChunker(
+            target_tokens,
+            min_tokens,
+            max_tokens,
+            token_counter=token_counter,
+            block_embeddings=block_embeddings,
+            semantic_threshold=semantic_threshold,
+        )
     elif chunker_name == "semantic-threshold":
         block_embeddings = cache.get_or_embed([block.text for block in blocks], embedder)
         chunker = SemanticThresholdChunker(
@@ -299,6 +304,7 @@ def _evaluate_index(
     budget: int,
     candidate_pool: int,
     scoring: str,
+    scorer=score_chunks,
 ) -> dict[str, float]:
     totals = {
         "evidence_term_recall": 0.0,
@@ -310,7 +316,7 @@ def _evaluate_index(
     }
     for template in templates:
         query_embedding = embedder.embed([template.query])[0]
-        scored = score_chunks(
+        scored = scorer(
             query_embedding,
             index.chunks,
             index.embeddings,

submission/experiments/build_qa_review_packet.py

@@ -41,9 +41,8 @@ def main() -> int:
     parser.add_argument("--dataset", choices=["hotpotqa"], default="hotpotqa")
     parser.add_argument("--data-file", help="Optional local HotpotQA JSON/JSONL/Arrow file.")
     parser.add_argument("--split", default="validation", choices=["validation", "train"])
-    parser.add_argument("--backend", default="hash", choices=["auto", "hash", "sentence-transformers"])
     parser.add_argument("--model", default="sentence-transformers/all-MiniLM-L6-v2")
-    parser.add_argument("--chunker", choices=["paragraph", "semantic-threshold", "structure-aware"], default="paragraph")
+    parser.add_argument("--chunker", choices=["semantic-threshold", "structure-aware"], default="structure-aware")
     parser.add_argument(
         "--scoring",
         choices=list(SCORING_PROFILES),
@@ -76,7 +75,7 @@ def main() -> int:
     work_dir.mkdir(parents=True, exist_ok=True)
 
     token_counter = TokenCounter()
-    embedder = make_embedder(backend=args.backend, model_name=args.model, local_files_only=True)
+    embedder = make_embedder(model_name=args.model, local_files_only=True)
     chunk_size = resolve_chunk_size_config(
         args.chunk_size_preset,
         args.target_tokens,

submission/experiments/chunking_ablation.py

@@ -34,7 +34,7 @@
 
 DEFAULT_OUTPUT_DIR = ROOT / "submission" / "results" / "hotpotqa"
 DEFAULT_WORK_DIR = ROOT / ".tokenpack" / "hotpotqa"
-STRATEGIES = ["budget-top-k", "greedy-density", "knapsack", "knapsack-redundancy", "knapsack-augment"]
+STRATEGIES = ["production-rag", "budget-top-k", "greedy-density", "knapsack", "knapsack-redundancy", "knapsack-augment"]
 TOKEN_RE = re.compile(r"[A-Za-z][A-Za-z-]{2,}")
 STOPWORDS = {
     "and",
@@ -74,9 +74,8 @@ def main() -> int:
     parser = argparse.ArgumentParser(description="Evaluate selectors on HotpotQA supporting facts.")
     parser.add_argument("--data-file", help="Optional local HotpotQA JSON/JSONL file.")
     parser.add_argument("--split", default="validation", choices=["validation", "train"])
-    parser.add_argument("--backend", default="hash", choices=["auto", "hash", "sentence-transformers"])
     parser.add_argument("--model", default="sentence-transformers/all-MiniLM-L6-v2")
-    parser.add_argument("--chunker", choices=["paragraph", "semantic-threshold", "structure-aware"], default="paragraph")
+    parser.add_argument("--chunker", choices=["semantic-threshold", "structure-aware"], default="structure-aware")
     parser.add_argument(
         "--scoring",
         choices=list(SCORING_PROFILES),
@@ -108,7 +107,7 @@ def main() -> int:
     work_dir.mkdir(parents=True, exist_ok=True)
 
     token_counter = TokenCounter()
-    embedder = make_embedder(backend=args.backend, model_name=args.model, local_files_only=True)
+    embedder = make_embedder(model_name=args.model, local_files_only=True)
     chunk_size = resolve_chunk_size_config(
         args.chunk_size_preset,
         args.target_tokens,