APT4's fertility tax against the Mistral-derived tokenizer is 1.6226 on English arXiv abstracts (95% interval 1.6140 to 1.6308), above its 1.5466 on the English preamble.

Relation: Metric against a reference text

metric

Fertility taxQuantity measured.

subject

APT4 tokenizerTokenizer whose tax is measured.

comparator

Mistral-derived tokenizerTokenizer the tax is measured against.

evaluation text

English arXiv abstractsCorpus measured.

language

EnglishLanguage of both texts.

value

1.6226 ratioTax on the evaluation text.

interval low

1.6140 ratioLower bound of the 95% bootstrap interval.

interval high

1.6308 ratioUpper bound of the 95% bootstrap interval.

reference text

Constitution preambleText compared with.

reference value

1.5466 ratioTax on the reference text.

above reference

trueWhether value exceeds reference_value.

Method and evaluation protocol

Tokens per word of APT4 divided by tokens per word of the Mistral-derived tokenizer over the corpus, with a document-level bootstrap of 1000 resamples (seed 20260703); the reference value is the same ratio on the preamble protocol text.

Dataset

784 abstracts of arXiv papers with January 2024 identifiers, at least 40 words each.Version: unspecified · Access: public

Reported results

1.6226 [1.6140, 1.6308] against 1.5466.

Uncertainty and replication

95% bootstrap interval over documents; the reference value is a single text without an interval.

Evidence references

atlas.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/719d51eb4ac7e6e0f36d69cf339c8e976266ee98/experiments/E01-fertility-atlas/results/atlas.jsonmetrics.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/719d51eb4ac7e6e0f36d69cf339c8e976266ee98/experiments/E01-fertility-atlas/results/metrics.json

Limitations

The corpus is one sample of its domain; the selection rules are in results/corpora_manifest.json.