Bielik-PL-11B-v3.0-Instruct's arithmetic accuracy minus Bielik-11B-v3.0-Instruct's with space-grouped numbers, minus the same difference with no-break-space-grouped numbers, is 0.1736 (95% interval 0.1535 to 0.1956; Holm-adjusted bootstrap p-value 0 with 1,000 resamples).

Relation: Interaction contrast

metric

Arithmetic probe accuracyQuantity compared.

subject

Bielik-PL-11B-v3.0-InstructModel after the tokenizer change.

comparator

Bielik-11B-v3.0-InstructModel before the tokenizer change.

setting a

Space-grouped number formatFirst number format.

setting b

No-break-space-grouped number formatSecond number format.

value

0.1736 difference in share of itemsInteraction estimate.

interval low

0.1535 difference in share of itemsLower bound of the 95% interval.

interval high

0.1956 difference in share of itemsUpper bound of the 95% interval.

p holm

0 p-valueHolm-adjusted bootstrap p-value.

scope

Items with at least four digits in addition, subtraction, sorting and unit conversion, with English and Polish instructions; comparison excluded as uninformative.Items the contrast is computed on.

Method and evaluation protocol

Paired item-level bootstrap over the probe items within each task, 1,000 resamples, seed 20260703; Holm adjustment over C1, C4', C2 for the PL model (space to no-break space) and C5; strata where both models' pooled accuracy lies between 5% and 95%.

Dataset

2,500 synthetic arithmetic items generated with seed 20260703, each in four number formats and two instruction languages.Version: unspecified · Access: public

Reported results

0.1736 [0.1535, 0.1956], Holm-adjusted p 0; truncated answers with no-break-space-grouped numbers 15 and 1 of 5,000, ambiguous 50 and 50.

Uncertainty and replication

95% paired bootstrap interval over items, 1,000 resamples. A bootstrap p-value of 0 means no resample of 1,000 fell on the other side of zero.

Evidence references

probe_results.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/2fc1a1b415a06b7648cdb37319ad573ccdc96a81/experiments/E02-digit-policy/results/probe_results.jsonmetrics.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/2fc1a1b415a06b7648cdb37319ad573ccdc96a81/experiments/E02-digit-policy/results/metrics.json

Limitations

The contrast cancels model main effects, but its magnitude could still depend on the models' training exposure to no-break spaces. The two models differ by 20B tokens of continued pretraining and by post-training as well as by the tokenizer; answer-only probes do not measure chain-of-thought arithmetic; MLX bf16 numerics differ from the paper's evaluation stack.