APT4's fertility tax against the Mistral-derived tokenizer is 0.6466 on Polish Wikipedia science articles (95% interval 0.6346 to 0.6608), above its 0.5020 on the Polish preamble.

Relation: Metric against a reference text

metric

Fertility taxQuantity measured.

subject

APT4 tokenizerTokenizer whose tax is measured.

comparator

Mistral-derived tokenizerTokenizer the tax is measured against.

evaluation text

Polish Wikipedia science articlesCorpus measured.

language

PolishLanguage of both texts.

value

0.6466 ratioTax on the evaluation text.

interval low

0.6346 ratioLower bound of the 95% bootstrap interval.

interval high

0.6608 ratioUpper bound of the 95% bootstrap interval.

reference text

Constitution preambleText compared with.

reference value

0.5020 ratioTax on the reference text.

above reference

trueWhether value exceeds reference_value.

Method and evaluation protocol

Tokens per word of APT4 divided by tokens per word of the Mistral-derived tokenizer over the corpus, with a document-level bootstrap of 1000 resamples (seed 20260703); the reference value is the same ratio on the preamble protocol text.

Dataset

Opening paragraphs, up to 900 words, of 229 Polish Wikipedia articles selected by science keywords.Version: unspecified · Access: public

Reported results

0.6466 [0.6346, 0.6608] against 0.5020.

Uncertainty and replication

95% bootstrap interval over documents; the reference value is a single text without an interval.

Evidence references

atlas.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/719d51eb4ac7e6e0f36d69cf339c8e976266ee98/experiments/E01-fertility-atlas/results/atlas.jsonmetrics.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/719d51eb4ac7e6e0f36d69cf339c8e976266ee98/experiments/E01-fertility-atlas/results/metrics.json

Limitations

The corpus is one sample of its domain; the selection rules are in results/corpora_manifest.json.