Bielik-PL-11B-v3.0-Instruct's arithmetic accuracy with space-grouped numbers minus its accuracy with no-break-space-grouped numbers is 0.0854 (95% interval 0.0677 to 0.1048; Holm-adjusted bootstrap p-value 0 with 1,000 resamples).

Relation: Setting contrast

metric

Arithmetic probe accuracyQuantity compared.

subject

Bielik-PL-11B-v3.0-InstructModel measured.

setting a

Space-grouped number formatFirst number format.

setting b

No-break-space-grouped number formatSecond number format.

value

0.0854 difference in share of itemsContrast estimate.

interval low

0.0677 difference in share of itemsLower bound of the 95% interval.

interval high

0.1048 difference in share of itemsUpper bound of the 95% interval.

p holm

0 p-valueHolm-adjusted bootstrap p-value.

scope

Items with at least four digits in addition, subtraction, sorting and unit conversion, with English and Polish instructions; comparison excluded as uninformative.Items the contrast is computed on.

Method and evaluation protocol

Paired item-level bootstrap over the probe items within each task, 1,000 resamples, seed 20260703; Holm adjustment over C1, C4', C2 for the PL model (space to no-break space) and C5; strata where both models' pooled accuracy lies between 5% and 95%.

Dataset

2,500 synthetic arithmetic items generated with seed 20260703, each in four number formats and two instruction languages.Version: unspecified · Access: public

Reported results

0.0854 [0.0677, 0.1048].

Uncertainty and replication

95% paired bootstrap interval over items, 1,000 resamples. A bootstrap p-value of 0 means no resample of 1,000 fell on the other side of zero.

Evidence references

probe_results.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/2fc1a1b415a06b7648cdb37319ad573ccdc96a81/experiments/E02-digit-policy/results/probe_results.jsonmetrics.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/2fc1a1b415a06b7648cdb37319ad573ccdc96a81/experiments/E02-digit-policy/results/metrics.json

Limitations

In the Holm family. The two models differ by 20B tokens of continued pretraining and by post-training as well as by the tokenizer; answer-only probes do not measure chain-of-thought arithmetic; MLX bf16 numerics differ from the paper's evaluation stack.