At a fixed 32,000-token vocabulary, the science-slice 32k tokenizer has at most half the pooled English-science excess fertility tax of the Polish-only 32k tokenizer (ratio 0.3997) at a relative Polish token cost below 0.05 pooled (0.0215) and below 0.07 on each Polish corpus (at most 0.0258).

Relation: Meets a decision rule

subject

Science-slice 32k tokenizerTokenizer the rule is applied to.

comparator

Polish-only 32k tokenizerTokenizer compared with.

first result

Exact publication c7cdbb52-799a-4c5b-a448-9597a83d0ccdResult on the English-science tax.

second result

Exact publication 07fd3382-a184-4c36-9047-709faa73f7d6Result on the Polish token cost.

rule

ratio of pooled English-science excess fertility taxes at most 0.50; pooled Polish relative token cost below 0.05; each per-corpus Polish relative token cost below 0.07Thresholds applied.

Method and evaluation protocol

Apply the locked decision rule to the pooled ratio and to the pooled and per-corpus Polish costs of the science-slice arm.

Dataset

Seven English and Polish corpora of about 120k words each; four are restricted.Version: unspecified · Access: restricted

Reported results

H1 bin confirmed (R 0.3997); H2 bin confirmed (pooled 0.0215, largest per corpus 0.0258); verdict confirmed. The whitespace arm meets the same thresholds.

Uncertainty and replication

Bins read on point estimates; every relevant 95% interval lies on the same side of its threshold.

Evidence references

analysis.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/c5299916db7f792a209ff2944e66c673c3ebb817/experiments/E09-vocabulary-allocation/results/analysis.jsonmetrics.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/c5299916db7f792a209ff2944e66c673c3ebb817/experiments/E09-vocabulary-allocation/results/metrics.json

Limitations

Fertility only; no model is trained on the tokenizers. Each corpus is one sample of its domain. The design, amendments and results were committed together. The rule was committed with the results.