On addition and subtraction probes with bare or comma-grouped numbers, Bielik-PL-11B-v3.0-Instruct answers 0.9451 of 492 items without a carry or borrow correctly, 0.9124 of 2,044 with one or two, and 0.888 of 1,464 with three or more.

Relation: Accuracy by carry count

metric

Arithmetic probe accuracyQuantity measured.

subject

Bielik-PL-11B-v3.0-InstructModel measured.

value no carry

0.9451 share of itemsAccuracy on items without a carry or borrow.

n no carry

492 itemsItems without a carry or borrow.

value one or two

0.9124 share of itemsAccuracy on items with one or two carries or borrows.

n one or two

2044 itemsItems with one or two carries or borrows.

value three or more

0.888 share of itemsAccuracy on items with three or more carries or borrows.

n three or more

1464 itemsItems with three or more carries or borrows.

scope

addition and subtraction items with bare or comma-grouped numbers, English and Polish instructions, all digit lengthsItems measured.

Method and evaluation protocol

Count carries in addition and borrows in subtraction when generating each item; pool graded outcomes by that count.

Dataset

2,500 synthetic arithmetic items generated with seed 20260703, each in four number formats and two instruction languages.Version: unspecified · Access: public

Reported results

0.9451, 0.9124 and 0.888.

Uncertainty and replication

Descriptive point values; no intervals computed.

Evidence references

probe_results.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/2fc1a1b415a06b7648cdb37319ad573ccdc96a81/experiments/E02-digit-policy/results/probe_results.jsonmetrics.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/2fc1a1b415a06b7648cdb37319ad573ccdc96a81/experiments/E02-digit-policy/results/metrics.json

Limitations

Carry count and operand length are correlated; descriptive by design. The two models differ by 20B tokens of continued pretraining and by post-training as well as by the tokenizer; answer-only probes do not measure chain-of-thought arithmetic; MLX bf16 numerics differ from the paper's evaluation stack.