APT4's fertility tax against the Mistral-derived tokenizer is 0.5764 on Polish reviews (95% interval 0.5713 to 0.5822), above its 0.5020 on the Polish preamble.

Relation: Metric against a reference text

metric

Fertility taxQuantity measured.

subject

APT4 tokenizerTokenizer whose tax is measured.

comparator

Mistral-derived tokenizerTokenizer the tax is measured against.

evaluation text

Polish reviewsCorpus measured.

language

PolishLanguage of both texts.

value

0.5764 ratioTax on the evaluation text.

interval low

0.5713 ratioLower bound of the 95% bootstrap interval.

interval high

0.5822 ratioUpper bound of the 95% bootstrap interval.

reference text

Constitution preambleText compared with.

reference value

0.5020 ratioTax on the reference text.

above reference

trueWhether value exceeds reference_value.

Method and evaluation protocol

Tokens per word of APT4 divided by tokens per word of the Mistral-derived tokenizer over the corpus, with a document-level bootstrap of 1000 resamples (seed 20260703); the reference value is the same ratio on the preamble protocol text.

Dataset

1,512 Polish consumer reviews from Allegro Reviews, at least 15 words each.Version: unspecified · Access: restricted

Reported results

0.5764 [0.5713, 0.5822] against 0.5020.

Uncertainty and replication

95% bootstrap interval over documents; the reference value is a single text without an interval.

Evidence references

atlas.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/719d51eb4ac7e6e0f36d69cf339c8e976266ee98/experiments/E01-fertility-atlas/results/atlas.jsonmetrics.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/719d51eb4ac7e6e0f36d69cf339c8e976266ee98/experiments/E01-fertility-atlas/results/metrics.json

Limitations

The corpus is one sample of its domain; the selection rules are in results/corpora_manifest.json.