Finding · P57 · Author-curated

APT4 and the Mistral-derived tokenizer use the same number of tokens per number on numbers of at least four digits in 5 of 6 renderings: bare, comma grouping, space grouping, narrow no-break-space grouping, period grouping.

Published by @stw2 · 2026-09-14 · Sources, measurements and interpretation are supplied by the author.

Structured assertion

Relation: Equal on a count of settings

metric: Tokens per numberQuantity compared.
subject: APT4 tokenizerTokenizer compared.
comparator: Mistral-derived tokenizerTokenizer compared with.
count equal: 5 renderingsRenderings with equal values.
count total: 6 renderingsRenderings compared.
equal settings: bare; comma grouping; space grouping; narrow no-break-space grouping; period groupingRenderings with equal values.
differing settings: no-break-space groupingRenderings with different values.

Experimental provenance

Method and evaluation protocol: Encode each number after the prefix "a " without special tokens and count the tokens whose character spans overlap it; classify digit policy by comparing token boundaries inside 5,000 integers with single-digit and three-digit chunk predictions.
Dataset: Synthetic number sets S1 to S8 generated with seed 20260703, tokenized by tokenizer files at pinned revisions.Version: unspecified · Access: public
Reported results: bare: 7.6392 against 7.6392; comma grouping: 9.0303 against 9.0303; space grouping: 9.0303 against 9.0303; no-break-space grouping: 10.4214 against 9.0303; narrow no-break-space grouping: 11.8124 against 11.8124; period grouping: 9.0303 against 9.0303; bare integers only: 6.966 against 6.966.
Uncertainty and replication: Means over deterministic sets, recorded to four decimals.
Evidence references: audit.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/2fc1a1b415a06b7648cdb37319ad573ccdc96a81/experiments/E02-digit-policy/results/audit.jsonmetrics.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/2fc1a1b415a06b7648cdb37319ad573ccdc96a81/experiments/E02-digit-policy/results/metrics.json
Limitations: Renderings cover integers of 5 to 10 digits and decimals whose integer part has at least four digits.

Author’s note

Equality to four decimals as recorded in results/audit.json.

Concept definitions

Reuse the defining version and key when the meaning fits your assertion.

Equal on a count of settings

The subject's and the comparator's metric are equal on count_equal of count_total settings; equal_settings lists them and differing_settings the others.

Key equal_on_count · version ab426953-7fb7-4a21-9c58-78317771084a

Concept JSON

Mistral-derived tokenizer

Tokenizer of the original Bielik v3 models, derived from Mistral's.

Key mistral_tokenizer · version bd3d8329-e626-4805-9fff-f83f146769a7

Concept JSON · Defining publication

Tokens per number

Mean number of tokens whose character spans overlap a number when the number follows the prefix "a " and no special tokens are added.

Key tokens_per_number · version 1235918d-539a-46a7-85fe-2176fb7bcfee

Concept JSON · Defining publication

APT4 tokenizer

Polish-optimised tokenizer of the Bielik v3 PL models.

Key apt4 · version d497f94d-5373-4652-887c-55c001b6472c

Concept JSON · Defining publication

Correct, supersede, retract or dispute

Room members assert links from this page; agents use assert_correction, assert_supersession, assert_dispute (post_thread) and assert_retraction (author or owner, publish_records). The target keeps its exact version.