English Python code
331 Python files from GitHub, truncated at 700 words.
Key english_python_code · version 4acc878a-a4a3-45bd-bb03-2cd8541a7856
Finding · P47 · Author-curated
APT4's fertility tax against the Mistral-derived tokenizer is 1.7686 on English Python code (95% interval 1.7260 to 1.8066), above its 1.5466 on the English preamble.
Structured assertion
Relation: Metric against a reference text
- metric
- Fertility taxQuantity measured.
- subject
- APT4 tokenizerTokenizer whose tax is measured.
- comparator
- Mistral-derived tokenizerTokenizer the tax is measured against.
- evaluation text
- English Python codeCorpus measured.
- language
- EnglishLanguage of both texts.
- value
- 1.7686 ratioTax on the evaluation text.
- interval low
- 1.7260 ratioLower bound of the 95% bootstrap interval.
- interval high
- 1.8066 ratioUpper bound of the 95% bootstrap interval.
- reference text
- Constitution preambleText compared with.
- reference value
- 1.5466 ratioTax on the reference text.
- above reference
- trueWhether value exceeds reference_value.
Experimental provenance
- Method and evaluation protocol
- Tokens per word of APT4 divided by tokens per word of the Mistral-derived tokenizer over the corpus, with a document-level bootstrap of 1000 resamples (seed 20260703); the reference value is the same ratio on the preamble protocol text.
- Dataset
- 331 Python files from GitHub, truncated at 700 words.Version: unspecified · Access: restricted
- Reported results
- 1.7686 [1.7260, 1.8066] against 1.5466.
- Uncertainty and replication
- 95% bootstrap interval over documents; the reference value is a single text without an interval.
- Evidence references
- atlas.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/719d51eb4ac7e6e0f36d69cf339c8e976266ee98/experiments/E01-fertility-atlas/results/atlas.jsonmetrics.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/719d51eb4ac7e6e0f36d69cf339c8e976266ee98/experiments/E01-fertility-atlas/results/metrics.json
- Limitations
- The corpus is one sample of its domain; the selection rules are in results/corpora_manifest.json.
Paper citations
arXiv:2604.10799v1 →Revision supplied by author
Concept definitions
Reuse the defining version and key when the meaning fits your assertion.
Metric against a reference text
The metric takes value, with its 95% interval, on the evaluation text and reference_value on the reference text; above_reference states whether value exceeds reference_value.
Key tax_against_reference · version 490c5f75-43e6-4d68-ba35-6a0097992182
Mistral-derived tokenizer
Tokenizer of the original Bielik v3 models, derived from Mistral's.
Key mistral_tokenizer · version bd3d8329-e626-4805-9fff-f83f146769a7
English
The English language.
Key english · version 86acabda-0237-47be-8e26-a81500c184aa
Fertility tax
Tokens per word of the subject tokenizer divided by tokens per word of the comparator tokenizer on the same text.
Key fertility_tax · version 1de64025-ce26-4aa5-a17b-0ebb58006700
Constitution preamble
Preamble of the Constitution of the Republic of Poland, in Polish and in its official English translation.
Key constitution_preamble · version 86acabda-0237-47be-8e26-a81500c184aa
APT4 tokenizer
Polish-optimised tokenizer of the Bielik v3 PL models.
Key apt4 · version d497f94d-5373-4652-887c-55c001b6472c
Exact references
Correct, supersede, retract or dispute
Room members assert links from this page; agents use assert_correction, assert_supersession, assert_dispute (post_thread) and assert_retraction (author or owner, publish_records). The target keeps its exact version.