Fetches the nine Table 1 tokenizers and records their revisions and vocabulary sizes.

https://github.com/stw2/tokenizer-science-tax @ 1855bbfabc832bcb964a9b74b90c4c52a7ce50ca

Reference checked 2026-09-14 13:08 UTC. Later commits, branches or plan changes do not retarget this attempt.

• experiments/E01-fertility-atlas/scripts/02_fetch_tokenizers.py

Command

python scripts/02_fetch_tokenizers.py

Working directory

experiments/E01-fertility-atlas

Configuration paths

None

Parameters

none

Environment

Python 3.12; Apple silicon CPU, no accelerator

Output directory

Not recorded

Materials the registrant named when registering this attempt, by content identity; obtainability is derived from their location reports. Nothing is fetched or verified.

• tokenizer bielik-v3-original · tokenizer · M2 Mistral-derived tokenizer (Bielik-11B-v3.0-Instruct) · tokenizer · Ask the reporter
• tokenizer apt4 · tokenizer · M3 APT4 tokenizer (Bielik-PL-11B-v3.0-Instruct) · tokenizer · Ask the reporter
• tokenizer apt3 · tokenizer · M4 APT3 tokenizer · tokenizer · Download
• tokenizer qwen3 · tokenizer · M5 Qwen3 tokenizer · tokenizer · Download
• tokenizer llama-3.2-smollm3 · tokenizer · M6 SmolLM3 tokenizer · tokenizer · Download
• tokenizer gemma3 · tokenizer · M7 Gemma 3 tokenizer · tokenizer · Ask the reporter
• tokenizer eurollm · tokenizer · M8 EuroLLM tokenizer · tokenizer · Download
• tokenizer mistral-small-3.1 · tokenizer · M9 Mistral Small 3.1 tokenizer · tokenizer · Download
• tokenizer apertus · tokenizer · M10 Apertus tokenizer · tokenizer · Download