All 5 arms pass the structural, digit-policy, fidelity and separator-piece gates, with 0 piece and 0 decode mismatches between SentencePiece and the fast tokenizer on 10,000 lines per arm.

Relation: Passes gates

gates

G1 APT4 pipeline shape, id layout, piece length and whitespace-only pieces; G2 single-digit policy with no multi-digit pieces; G4 SentencePiece and fast tokenizer give the same pieces and decodes; G5 separator pieces present exactly in the arms that add themGates applied to every arm.

arms passed

5 tokenizersArms passing every gate.

arms total

5 tokenizersArms checked.

fidelity lines

10000 linesLines compared per arm by G4.

piece mismatches

0 linesLines with different pieces, all arms.

decode mismatches

0 linesLines with different decodes, all arms.

Method and evaluation protocol

Compare each arm's tokenizer.json pipeline with APT4's, classify digit segmentation on synthetic integers, encode 10,000 training lines with both the SentencePiece model and the fast tokenizer, and check the separator pieces.

Dataset

First 2,000 lines of each training text and synthetic integer sets.Version: unspecified · Access: restricted

Reported results

G1 5/5, G2 5/5, G4 5/5, G5 5/5; whitespace-only pieces in the whitespace arm 13; the digit audit reproduces the earlier audit's reference rows in 6 of 6 checks.

Uncertainty and replication

Exact checks; no sampling.

Evidence references

tokenizers_e9_manifest.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/c5299916db7f792a209ff2944e66c673c3ebb817/experiments/E09-vocabulary-allocation/results/tokenizers_e9_manifest.jsonpolicy_audit.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/c5299916db7f792a209ff2944e66c673c3ebb817/experiments/E09-vocabulary-allocation/results/policy_audit.jsonmetrics.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/c5299916db7f792a209ff2944e66c673c3ebb817/experiments/E09-vocabulary-allocation/results/metrics.json

Limitations

G4 lines come from the training text itself. The gates were recorded in the run that failed the determinism gate.