Recomputing tokens per word, characters per token and fertility tax against the Mistral-derived tokenizer for nine tokenizers on seven corpora and two preamble texts reproduces all 243 values exactly.

Relation: Reproduces exactly

target

tokens per word, characters per token and fertility tax against the Mistral-derived tokenizer of nine tokenizers on seven corpora and the two preamble protocol textsValues recomputed.

protocol

whitespace-split words; characters counted with whitespace; no special tokens; the same corpus order and resamplesCounting protocol.

cells matched

243 cellsValues reproduced exactly.

cells total

243 cellsValues compared.

Method and evaluation protocol

Extend the atlas with the five arms appended after the nine reference tokenizers and compare every reference value with the earlier atlas; the run aborts on any difference above 1e-9.

Dataset

Seven corpora and two preamble protocol texts; four corpora are restricted.Version: unspecified · Access: restricted

Reported results

243 of 243 values equal; largest absolute difference 0.0.

Uncertainty and replication

Exact comparison of point values; no sampling.

Evidence references

atlas.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/c5299916db7f792a209ff2944e66c673c3ebb817/experiments/E09-vocabulary-allocation/results/atlas.jsonmetrics.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/c5299916db7f792a209ff2944e66c673c3ebb817/experiments/E09-vocabulary-allocation/results/metrics.json

Limitations

The same code path and tokenizer files as the earlier atlas; this checks the rig, not the tokenizers' provenance.