APT4's pooled English-science excess fertility tax against the Mistral-derived tokenizer is 0.5523 (95% interval 0.5377 to 0.5656) and APT3's 0.5007 (0.4827 to 0.5174).

Relation: Metric comparison

metric

Pooled English-science excess fertility taxQuantity compared.

subject

APT4 tokenizerTokenizer measured.

subject value

0.5523 ratioValue of the metric for the subject.

comparator

APT3 tokenizerTokenizer compared with.

comparator value

0.5007 ratioValue of the metric for the comparator.

evaluation text

English arXiv abstracts, LaTeX method sections, Python code and GSM8K problems, pooledCorpora measured.

language

EnglishLanguage of the corpora.

setting

excess tax against the Mistral-derived tokenizerTokenizer both values are measured against.

Method and evaluation protocol

Sum each tokenizer's tokens over the four English corpora, divide by the Mistral-derived tokenizer's sum and subtract 1.

Dataset

Four English corpora of about 120k words each: arXiv abstracts, LaTeX method sections, Python code, GSM8K problems; two are restricted.Version: unspecified · Access: restricted

Reported results

APT4 0.5523 [0.5377, 0.5656]; APT3 0.5007 [0.4827, 0.5174]; Polish-only arm 0.5277 [0.5131, 0.5410].

Uncertainty and replication

95% percentile interval over 1000 document resamples within each corpus, paired across tokenizers (seed 20260703); tokenizer training is deterministic and was run once per arm.

Evidence references

analysis.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/c5299916db7f792a209ff2944e66c673c3ebb817/experiments/E09-vocabulary-allocation/results/analysis.jsonmetrics.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/c5299916db7f792a209ff2944e66c673c3ebb817/experiments/E09-vocabulary-allocation/results/metrics.json

Limitations

Each corpus is one sample of its domain.