APT4's fertility tax against the Mistral-derived tokenizer is 0.6461 on Polish PES examination questions (95% interval 0.6420 to 0.6501), above its 0.5020 on the Polish preamble.

Relation: Metric against a reference text

metric

Fertility taxQuantity measured.

subject

APT4 tokenizerTokenizer whose tax is measured.

comparator

Mistral-derived tokenizerTokenizer the tax is measured against.

evaluation text

Polish PES examination questionsCorpus measured.

language

PolishLanguage of both texts.

value

0.6461 ratioTax on the evaluation text.

interval low

0.6420 ratioLower bound of the 95% bootstrap interval.

interval high

0.6501 ratioUpper bound of the 95% bootstrap interval.

reference text

Constitution preambleText compared with.

reference value

0.5020 ratioTax on the reference text.

above reference

trueWhether value exceeds reference_value.

Method and evaluation protocol

Tokens per word of APT4 divided by tokens per word of the Mistral-derived tokenizer over the corpus, with a document-level bootstrap of 1000 resamples (seed 20260703); the reference value is the same ratio on the preamble protocol text.

Dataset

1,725 questions of the Polish State Specialization Examination, 2018 to 2022, at least 25 words each.Version: unspecified · Access: restricted

Reported results

0.6461 [0.6420, 0.6501] against 0.5020.

Uncertainty and replication

95% bootstrap interval over documents; the reference value is a single text without an interval.

Evidence references

atlas.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/719d51eb4ac7e6e0f36d69cf339c8e976266ee98/experiments/E01-fertility-atlas/results/atlas.jsonmetrics.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/719d51eb4ac7e6e0f36d69cf339c8e976266ee98/experiments/E01-fertility-atlas/results/metrics.json

Limitations

The corpus is one sample of its domain; the selection rules are in results/corpora_manifest.json.