Continued pretraining of Qwen2.5-1.5B with its own tokenizer on 0.5B tokens lowers digit-probe accuracy from 0.5867 to 0.5167.

Relation: Metric comparison

metric

Digit-probe accuracyQuantity compared.

subject

Qwen2.5-1.5B continued with its own tokenizerModel whose value is subject_value.

subject value

0.5167 accuracyAccuracy of the subject.

comparator

Qwen2.5-1.5BModel whose value is comparator_value.

comparator value

0.5867 accuracyAccuracy of the comparator.

setting

500,170,752 Qwen tokens of continued pretraining without a mathematics or code slice, one runTraining of the subject.

scope

900 pairsItem-format pairs scored.

Method and evaluation protocol

300 synthetic arithmetic items from E2 in three prompt formats (bare English, bare Polish, Polish with spaced digits), 4-shot plain-text prompts, greedy decoding with batch 24, completion cut at the first newline, graded by E2's dual-locale grader after its 31-case golden suite; accuracy pooled over the 900 item-format pairs.

Dataset

300 items of E2's synthetic arithmetic probes (120 addition, 90 subtraction, 50 comparison, 40 sorting) in three prompt formats, frozen in configs/probe_digits_spec.json.Version: unspecified · Access: public

Reported results

Arm A at 500M 0.5167 (Wilson 95% [0.4841, 0.5492]); untouched 0.5867; arm A at 100M 0.6167, 300M 0.5489.

Uncertainty and replication

Wilson 95% intervals over the 900 pairs; the two models were not compared item by item.

Evidence references

probe_digits_armA.jsonl · publichttps://github.com/stw2/tokenizer-science-tax/blob/9a57aa40f4e774136f51b6730c33266d59094a45/experiments/E05-control-arm/results/probe_digits_armA.jsonlanalysis.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/9a57aa40f4e774136f51b6730c33266d59094a45/experiments/E05-control-arm/results/analysis.jsonmetrics.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/9a57aa40f4e774136f51b6730c33266d59094a45/experiments/E05-control-arm/results/metrics.json

Limitations

One run per arm; values at a constant learning rate before the decay anneal the design owed; 1.5B parameters and 0.5B tokens against the paper's 11B and 20B; Fast Vocabulary Transfer instead of FOCUS; the design was committed with the results and fixed no decision rule; the training text was not kept.