Does `ttok` need an internet connection?

[NightMachinery]

ttok either does not work or is very very slow when the internet connection is bad/non-existent. Is this a known design behavior or a bug?

Related:

• ttok adds a 20 second delay when using llm-gpt4all when offline #10

[NightMachinery]

@simonw Bumping this; I am considering disabling token counting, but I am not sure if the culprit is really this.

[NightMachinery]

Bumping this again.

[NightMachinery]

ttok seems to use HuggingFace, which is most probably the culprit. See this issue. I suspect some similar env flag must be set for offline usage.

[Manamama]

Do introduce the offline switch after caching the file.

ChatGPT invented and wrote this fix, which works:

🐛 Offline Failure: ttok silently attempts net access instead of bundling cl100k_base.tiktoken

Summary:
ttok fails or becomes extremely slow in offline/bad-network environments due to lazy runtime downloading of cl100k_base.tiktoken. This is a fragile and unnecessary design choice, as the required file is <2 MB and could easily be bundled.

---

🔍 Repro Steps

1. Ensure no internet access (e.g., disable network, or run in an airgapped VM)

2. Clear any previous .tiktoken cache:

   rm -rf ~/.cache/tiktoken

3. Run:

   ttok "Test input"

💥 Observed Behavior

• 10–20 second delay (or permanent hang if net blocked via firewall rules)
• Fails silently if CDN unreachable
• Fails even though the tokenization logic and encoding map are deterministic and static

---

📦 Root Cause

tiktoken (used under the hood) attempts to lazily fetch cl100k_base.tiktoken from:

https://openaipublic.blob.core.windows.net/encodings/cl100k_base.tiktoken

This is done on first invocation, with the file not shipped in any part of:

• ttok pip package
• tiktoken pip package
• openai SDK
• Local pre-caching setup

---

🤯 Why This Is Broken

• cl100k_base.tiktoken is only ~1.6 MB and compressible
• It does not change frequently
• It is required for ttok to function at all
• It's core logic — not optional telemetry or docs
• Many real-world environments are intentionally offline (e.g., secure airgapped systems)

---

✅ Proposed Fix

1. Bundle cl100k_base.tiktoken directly inside the ttok or tiktoken PyPI package

   • Place under tiktoken/resources/ or similar
   • Use that as fallback if the cache is empty

2. Respect an env flag like TIKTOKEN_OFFLINE=1 to prevent any net attempt

3. (Optional) On install, download and cache the file once, rather than at runtime

---

🧱 Workaround for Users

Until fixed, users must manually:

mkdir -p ~/.cache/tiktoken
curl -o ~/.cache/tiktoken/cl100k_base.tiktoken \
  https://openaipublic.blob.core.windows.net/encodings/cl100k_base.tiktoken

Or bake this into Dockerfiles, config mgmt, etc.

---