Bielik-11B-v3.0-Instruct's arithmetic accuracy with space-grouped numbers minus its accuracy with no-break-space-grouped numbers is -0.0882 (95% interval -0.1035 to -0.0737; unadjusted bootstrap p-value 0 with 1,000 resamples).

Relation: Setting contrast

metric

Arithmetic probe accuracyQuantity compared.

subject

Bielik-11B-v3.0-InstructModel measured.

setting a

Space-grouped number formatFirst number format.

setting b

No-break-space-grouped number formatSecond number format.

value

-0.0882 difference in share of itemsContrast estimate.

interval low

-0.1035 difference in share of itemsLower bound of the 95% interval.

interval high

-0.0737 difference in share of itemsUpper bound of the 95% interval.

p bootstrap

0 p-valueUnadjusted bootstrap p-value.

scope

Items with at least four digits in addition, subtraction, sorting and unit conversion, with English and Polish instructions; comparison excluded as uninformative.Items the contrast is computed on.

Method and evaluation protocol

Paired item-level bootstrap over the probe items within each task, 1,000 resamples, seed 20260703; Holm adjustment over C1, C4', C2 for the PL model (space to no-break space) and C5; strata where both models' pooled accuracy lies between 5% and 95%.

Dataset

2,500 synthetic arithmetic items generated with seed 20260703, each in four number formats and two instruction languages.Version: unspecified · Access: public

Reported results

-0.0882 [-0.1035, -0.0737].

Uncertainty and replication

95% paired bootstrap interval over items, 1,000 resamples. A bootstrap p-value of 0 means no resample of 1,000 fell on the other side of zero.

Evidence references

probe_results.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/2fc1a1b415a06b7648cdb37319ad573ccdc96a81/experiments/E02-digit-policy/results/probe_results.jsonmetrics.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/2fc1a1b415a06b7648cdb37319ad573ccdc96a81/experiments/E02-digit-policy/results/metrics.json

Limitations

Outside the Holm family; the original report listed its p-value as Holm-adjusted. The two models differ by 20B tokens of continued pretraining and by post-training as well as by the tokenizer; answer-only probes do not measure chain-of-thought arithmetic; MLX bf16 numerics differ from the paper's evaluation stack.