APT4 splits all 5,000 classified integers into one token per digit and has no multi-digit tokens in its vocabulary.

Relation: Digit policy measurement

subject

APT4 tokenizerTokenizer measured.

policy

Single-digit policyDigit policy class.

share single digit

1 share of integersShare of classified integers split one token per digit.

share three digit left to right

0 share of integersShare of classified integers split into three-digit chunks from the left.

integers classified

5000 integersIntegers classified.

multi digit tokens

0 tokensVocabulary tokens of two or more digits.

max digit run

1 digitsLongest digit run in one vocabulary token.

place value boundary coverage

1 share of integersShare of integers whose thousands-group boundaries are all token boundaries.

single token coverage 0 999

0.01 share of integersShare of the integers 0 to 999 encoded as one token.

Method and evaluation protocol

Encode each number after the prefix "a " without special tokens and count the tokens whose character spans overlap it; classify digit policy by comparing token boundaries inside 5,000 integers with single-digit and three-digit chunk predictions.

Dataset

Synthetic number sets S1 to S8 generated with seed 20260703, tokenized by tokenizer files at pinned revisions.Version: unspecified · Access: public

Reported results

Single-digit share 1; multi-digit vocabulary tokens 0; longest digit token 1; place-value boundary coverage 1.

Uncertainty and replication

Exact counts over deterministic sets.

Evidence references

audit.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/2fc1a1b415a06b7648cdb37319ad573ccdc96a81/experiments/E02-digit-policy/results/audit.jsonmetrics.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/2fc1a1b415a06b7648cdb37319ad573ccdc96a81/experiments/E02-digit-policy/results/metrics.json

Limitations

The classified integers are 3,000 of 2 to 4 digits and 2,000 of 5 to 10 digits; the hypothesis this supports was written after the tokenizer files had been inspected.