Manifesto Argument Mining (UK Pilot)

This repository implements a transfer argument mining pipeline for UK party manifestos.

Layout

• 01_src/: source code, evaluators, and runnable helper scripts.
• 02_data/interim/: transfer-training tables and dataset-build reports.
• 02_data/processed/: canonical annotation files, maintained markdown summaries, and LLM comparison outputs.
• 02_data/processed/experiments/: saved run-specific predictions, metrics, analyses, and local logs.
• 02_data/reference/: checked-in lookup tables and example mappings.

Core Files

The current pipeline relies on these main files in 02_data/processed/:

• manifestos_uk_qs.csv
  • Columns: manifesto_id, party_id, party_name, election_date, qs_id, qs_text, cmp_code, country
  • Generated locally from the Manifesto Project export script.
• party_status_uk.csv
  • Columns: party_id, election_date, gov_status, coalition_flag, source
  • Generated locally from the manifesto export plus the checked-in incumbency reference table.
• annotation_human_labour2001.csv
  • Columns: manifesto_id, qs_id, component_label, claim_type, annotator_id
• analysis_report.md
  • The maintained markdown summary for the evaluation.
• LLM_evaluation.md and LLM_evaluation.json
  • Markdown and machine-readable comparisons between the human Labour 2001 labels and the saved GPT5.2-thinking outputs.

Setup

1. R dependencies:

install.packages(c("manifestoR", "dplyr", "readr", "stringr", "lubridate"))

2. Python dependencies:

python3 -m venv .venv
source .venv/bin/activate
pip install -r 01_src/python/requirements.txt

3. Manifesto API key file:

• Needs an account on https://manifestoproject.wzb.eu/signup
• Default path expected by script: 01_src/R/manifesto_apikey.txt
• Or set env var:

export MANIFESTO_API_KEY_PATH=/absolute/path/to/manifesto_apikey.txt

Supplementary UK files

The repo already includes the generator for the UK supplementary files:

• 01_src/R/export_uk_data.R: main R export script
• 01_src/scripts/generate_uk_supplementary_files.sh: thin wrapper with default paths

The wrapper uses your own Manifesto Project API key and will stop if the key or the incumbency reference file is missing.

Use it when:

• setting up the repo on a new machine,
• regenerating 02_data/processed/manifestos_uk_qs.csv,
• regenerating 02_data/processed/party_status_uk.csv,
• replacing local copies after cleaning 02_data/processed/.

Keep these two generated files local. The checked-in source input for party-status matching is:

• 02_data/reference/uk_incumbent_governments.csv

Generate the supplementary files with:

bash 01_src/scripts/generate_uk_supplementary_files.sh

Optional custom output paths:

bash 01_src/scripts/generate_uk_supplementary_files.sh \
  path/to/manifestos_uk_qs.csv \
  path/to/party_status_uk.csv \
  02_data/reference/uk_incumbent_governments.csv

Reproducible Commands

Phase A: Export UK data

bash 01_src/scripts/generate_uk_supplementary_files.sh

Direct R entrypoint:

Rscript 01_src/R/export_uk_data.R \
  02_data/processed/manifestos_uk_qs.csv \
  02_data/processed/party_status_uk.csv \
  02_data/reference/uk_incumbent_governments.csv

Phase B: Use the canonical Labour 2001 human annotation file

The main evaluation file is:

• 02_data/processed/annotation_human_labour2001.csv

It contains the 400-row Labour 2001 human annotations used by the evaluation scripts.

Phase C0: Build transfer_train.csv from source corpus export

python3 01_src/python/build_transfer_train.py \
  --input path/to/source_corpus.csv \
  --output 02_data/interim/transfer_train.csv \
  --report 02_data/interim/transfer_train_report.json \
  --source-name uselecdeb60to16 \
  --mapping-json 02_data/reference/transfer_label_map_example.json

Notes:

• Supported input formats: .csv, .jsonl, .json, .parquet.
• Auto-detects text and component columns when possible.
• Normalized component labels are claim, premise, and none (non-argument).
• USElec-style o labels are mapped to none during normalization.

Use this helper to prepare transfer data directly from the Multimodal-USElecDeb60To16 corpus:

bash 01_src/scripts/prepare_uselecdeb60to16_transfer.sh

Use this helper to prepare the context-window transfer file used by the ManifestoBERTa-context report run:

bash 01_src/scripts/prepare_uselecdeb60to16_transfer_context.sh

This writes:

• 02_data/interim/transfer_train_context.csv
• 02_data/interim/transfer_train_report_uselecdeb60to16_context.json

Phase C1: Fine-tune the current two-stage component models

Current BERT two-stage run:

bash 01_src/scripts/run_finetune_bert_mixed_2stage_epoch1.sh

Current ManifestoBERTa two-stage run:

bash 01_src/scripts/run_finetune_manifestoberta_mixed_2stage_epoch1.sh

Optional context-aware ManifestoBERTa run:

bash 01_src/scripts/run_finetune_manifestoberta_context_mixed_2stage_epoch1.sh

Notes:

• Produces contract-compatible predictions (model_id, manifesto_id, qs_id, pred_component, confidence).
• Saves fine-tuned checkpoints to 02_data/models.
• For checkpoints with different classifier heads, add --ignore-mismatched-sizes; for offline reruns, add --local-files-only.
• For Apple Silicon, use --device auto (or --device mps) and consider --max-length 128 to speed up runs.
• Add --component-mode two-stage to train argument vs none first, then claim vs premise only on argument-unit rows.

Phase C2: Fine-tune the claim-type model

bash 01_src/scripts/run_finetune_roberta_claim_types.sh

This helper builds the claim-type transfer set and writes these claim-type training artifacts:

• 02_data/interim/transfer_train_claim_types_klaebel_stede_gurevych.csv
• 02_data/interim/transfer_train_report_claim_types_klaebel_stede_gurevych.json
• 02_data/processed/experiments/predictions/predictions_roberta_claim_types_klaebel_stede_gurevych_claim_type_epoch10.csv
• 02_data/processed/experiments/metrics/finetune_metrics_roberta_claim_types_klaebel_stede_gurevych_claim_type_epoch10.json
• 02_data/processed/experiments/validation/validation_roberta_claim_types_klaebel_stede_gurevych_claim_type_epoch10.csv

Phase D: Focused Current Analysis

bash 01_src/scripts/run_focused_analysis.sh

This is the only maintained evaluation/report path for the current project scope. It reruns only:

• two-stage component evaluations
• claim-type evaluations
• the focused markdown summary in 02_data/processed/analysis_report.md

Notes:

• Governing-versus-opposition analysis is intentionally excluded from this workflow.
• The script only evaluates artifacts that already exist on disk; it does not retrain missing models.
• The script looks for these saved prediction files when present: predictions_finetune_bert_mixed_2stage_epoch1.csv, predictions_finetune_manifestoberta_mixed_2stage_epoch1.csv, predictions_finetune_manifestoberta_context_mixed_2stage_epoch1.csv, and predictions_roberta_claim_types_human400_infer.csv.
• The script clears stale analysis outputs before regenerating the current human-only reports.
• Per-run reports are written to 02_data/processed/experiments/analysis/.

Full run helper

bash 01_src/scripts/run_all.sh

This helper exports UK data and then runs the focused analysis workflow above.

Optional: Mini-IAA

If two annotators fill annotator_id in annotation_human_labour2001.csv, run:

python3 01_src/python/compute_iaa.py \
  --gold 02_data/processed/annotation_human_labour2001.csv \
  --output 02_data/processed/iaa_report.json

Optional: LLM_evaluation

To compare the current manual Labour 2001 component annotations against the two GPT5.2-thinking label files:

python3 01_src/python/run_LLM_evaluation.py \
  --human 02_data/processed/annotation_human_labour2001.csv \
  --llm 02_data/processed/labour_2001_first400_GPT5.2-thinking_few-shot.csv \
  --llm 02_data/processed/labour_2001_first400_GPT5.2-thinking_zero-shot.csv \
  --output-json 02_data/processed/LLM_evaluation.json \
  --report 02_data/processed/LLM_evaluation.md

Annotation resources

• Annotation UI notes: 01_src/annotation_ui/README.md
• Visual annotation UI: 01_src/annotation_ui/annotator.html (bash 01_src/annotation_ui/run_annotation_ui.sh)