#(Github) Beyond Token Generation: Inter-Token Structure in the Voynich Manuscript

Author: Youngsan Chang
Date: April 2026

This repository contains the reproduction package for Paper 5 of the Voynich Manuscript structural analysis series.

Paper

The final paper is located at:

paper/Chang_2026e_Voynich_InterToken.pdf

Research Question

Do Voynich tokens exhibit non-random local sequential organization beyond token-level PCS generation?

Summary

This study extends previous prefix-core-suffix (PCS) token-generation analysis to the inter-token level. It tests whether Voynich tokens are arranged non-randomly across local sequences using token transition entropy, next-token prediction, PCS component transitions, line-boundary analysis, recurrent local patterns, and family clustering.

Key Results

• Corpus: 23,753 tokens, 12,688 types, 8,510 lines
• Bigram entropy: real 11.7001 vs line-internal shuffle 12.7229
• Next-token top-1 accuracy: real 0.6917 vs line-internal shuffle 0.5505
• Next-token top-3 accuracy: real 0.8168 vs line-internal shuffle 0.6912
• suffix(token_i) -> prefix(token_i+1): real conditional entropy 4.4008 vs baseline 4.5087
• Line-boundary chi-square: 46151.0531, p < .001
• same_core mean distance: real 235.93 vs shuffled 442.77
• same_prefix_core mean distance: real 319.98 vs shuffled 632.90

Repository Structure

• paper/ - final PDF and manuscript source
• scripts/ - analysis scripts
• data/ - input transcription or data source notes
• outputs/ - generated analysis outputs
• figures/ - final figures
• tables/ - final tables
• docs/ - reproducibility notes, captions, and AI disclosure

Reproducing the Analysis

Install dependencies:

pip install -r requirements.txt

Run:

python scripts/voynich_intertoken_pipeline.py   --input data/ZL3b-n.txt   --segmentations outputs/pcs_pipeline/best_segmentations.csv   --families outputs/pcs_pipeline/morphological_families.csv   --out outputs/intertoken_pipeline   --figures figures/intertoken   --n-iter 1000   --seed 42

If the PCS segmentation and family files are not included, see docs/REPRODUCIBILITY.md.

AI Use Disclosure

AI-based tools (Claude, ChatGPT, Gemini) assisted with code implementation and manuscript editing. Research design, analytical decisions, and interpretation were conducted by the author.

Related Work

This paper builds on:

• Chang (2026a): https://doi.org/10.5281/zenodo.19687875
• Chang (2026b): https://doi.org/10.5281/zenodo.19737888
• Chang (2026c): https://doi.org/10.5281/zenodo.19765707
• Chang (2026d): https://doi.org/10.5281/zenodo.19858094

License

• Text/manuscript: CC BY 4.0
• Code: MIT License
• Data: follows the original Zandbergen-Landini transcription source terms