APT4's fertility tax against the Mistral-derived tokenizer is 1.3640 on GSM8K problems (95% interval 1.3616 to 1.3665), below its 1.5466 on the English preamble.

Relation: Metric against a reference text

metric

Fertility taxQuantity measured.

subject

APT4 tokenizerTokenizer whose tax is measured.

comparator

Mistral-derived tokenizerTokenizer the tax is measured against.

evaluation text

GSM8K problemsCorpus measured.

language

EnglishLanguage of both texts.

value

1.3640 ratioTax on the evaluation text.

interval low

1.3616 ratioLower bound of the 95% bootstrap interval.

interval high

1.3665 ratioUpper bound of the 95% bootstrap interval.

reference text

Constitution preambleText compared with.

reference value

1.5466 ratioTax on the reference text.

above reference

falseWhether value exceeds reference_value.

Method and evaluation protocol

Tokens per word of APT4 divided by tokens per word of the Mistral-derived tokenizer over the corpus, with a document-level bootstrap of 1000 resamples (seed 20260703); the reference value is the same ratio on the preamble protocol text.

Dataset

The 7,473 questions of the GSM8K training split.Version: unspecified · Access: public

Reported results

1.3640 [1.3616, 1.3665] against 1.5466.

Uncertainty and replication

95% bootstrap interval over documents; the reference value is a single text without an interval.

Evidence references

atlas.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/719d51eb4ac7e6e0f36d69cf339c8e976266ee98/experiments/E01-fertility-atlas/results/atlas.jsonmetrics.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/719d51eb4ac7e6e0f36d69cf339c8e976266ee98/experiments/E01-fertility-atlas/results/metrics.json

Limitations

The corpus is one sample of its domain; the selection rules are in results/corpora_manifest.json.