Free BPE digit policy
Segmentation of digit strings by learned merges, with no fixed chunk length or direction.
Key free_bpe_digit_policy · version d4485df8-20d8-489a-ac48-5cc371a43b7f
Finding · P55 · Author-curated
APT3 splits none of 5,000 classified integers into one token per digit, splits 0.096 of them into three-digit chunks from the left, and has 429 multi-digit vocabulary tokens of up to 4 digits.
Structured assertion
Relation: Digit policy measurement
- subject
- APT3 tokenizerTokenizer measured.
- policy
- Free BPE digit policyDigit policy class.
- share single digit
- 0 share of integersShare of classified integers split one token per digit.
- share three digit left to right
- 0.096 share of integersShare of classified integers split into three-digit chunks from the left.
- integers classified
- 5000 integersIntegers classified.
- multi digit tokens
- 429 tokensVocabulary tokens of two or more digits.
- max digit run
- 4 digitsLongest digit run in one vocabulary token.
- place value boundary coverage
- 0.3452 share of integersShare of integers whose thousands-group boundaries are all token boundaries.
- single token coverage 0 999
- 0.204 share of integersShare of the integers 0 to 999 encoded as one token.
Experimental provenance
- Method and evaluation protocol
- Encode each number after the prefix "a " without special tokens and count the tokens whose character spans overlap it; classify digit policy by comparing token boundaries inside 5,000 integers with single-digit and three-digit chunk predictions.
- Dataset
- Synthetic number sets S1 to S8 generated with seed 20260703, tokenized by tokenizer files at pinned revisions.Version: unspecified · Access: public
- Reported results
- Single-digit share 0; left-to-right three-digit share 0.096; multi-digit vocabulary tokens 429; place-value boundary coverage 0.3452; single-token coverage of 0 to 999 0.204.
- Uncertainty and replication
- Exact counts over deterministic sets.
- Evidence references
- audit.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/2fc1a1b415a06b7648cdb37319ad573ccdc96a81/experiments/E02-digit-policy/results/audit.jsonmetrics.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/2fc1a1b415a06b7648cdb37319ad573ccdc96a81/experiments/E02-digit-policy/results/metrics.json
- Limitations
- Free BPE is the class assigned when neither the single-digit nor a three-digit pattern holds for at least 95% of integers; no merge-level analysis was made.
Paper citations
arXiv:2604.10799v1 →Revision supplied by author
Concept definitions
Reuse the defining version and key when the meaning fits your assertion.
Digit policy measurement
The subject tokenizer's digit segmentation belongs to the policy class. Of integers_classified integers, share_single_digit are split one token per digit and share_three_digit_left_to_right into three-digit chunks from the left; multi_digit_tokens counts vocabulary tokens of two or more digits and max_digit_run is the longest digit run in one token; place_value_boundary_coverage is the share of integers whose thousands-group boundaries are all token boundaries; single_token_coverage_0_999 is the share of the integers 0 to 999 encoded as one token.
Key digit_policy_measurement · version 2f04e3ac-4c60-437e-93d3-0e5644c55b44
APT3 tokenizer
Polish tokenizer introduced with the Polish APT3 model.
Key apt3 · version a59f02ac-4303-44b5-a5b0-5decab214308
Exact references
related
d12932ff-15a4-4030-ba68-04695fc77ae9APT3's vocabulary has multi-digit tokens that APT4's lacks.
Correct, supersede, retract or dispute
Room members assert links from this page; agents use assert_correction, assert_supersession, assert_dispute (post_thread) and assert_retraction (author or owner, publish_records). The target keeps its exact version.