Finding · P64 · Author-curated

Bielik-PL-11B-v3.0-Instruct's arithmetic accuracy minus Bielik-11B-v3.0-Instruct's with comma-grouped numbers, minus the same difference with space-grouped numbers, is -0.0035 (95% interval -0.023 to 0.0136; Holm-adjusted p 0.69).

Published by @stw2 · 2026-09-14 · Sources, measurements and interpretation are supplied by the author.

Structured assertion

Relation: Interaction contrast

metric: Arithmetic probe accuracyQuantity compared.
subject: Bielik-PL-11B-v3.0-InstructModel after the tokenizer change.
comparator: Bielik-11B-v3.0-InstructModel before the tokenizer change.
setting a: Comma-grouped number formatFirst number format.
setting b: Space-grouped number formatSecond number format.
value: -0.0035 difference in share of itemsInteraction estimate.
interval low: -0.023 difference in share of itemsLower bound of the 95% interval.
interval high: 0.0136 difference in share of itemsUpper bound of the 95% interval.
p holm: 0.69 p-valueHolm-adjusted bootstrap p-value.
scope: Items with at least four digits in addition, subtraction, sorting and unit conversion, with English and Polish instructions; comparison excluded as uninformative.Items the contrast is computed on.

Experimental provenance

Method and evaluation protocol: Paired item-level bootstrap over the probe items within each task, 1,000 resamples, seed 20260703; Holm adjustment over C1, C4', C2 for the PL model (space to no-break space) and C5; strata where both models' pooled accuracy lies between 5% and 95%.
Dataset: 2,500 synthetic arithmetic items generated with seed 20260703, each in four number formats and two instruction languages.Version: unspecified · Access: public
Reported results: -0.0035 [-0.023, 0.0136], Holm-adjusted p 0.69.
Uncertainty and replication: 95% paired bootstrap interval over items, 1,000 resamples.
Evidence references: probe_results.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/2fc1a1b415a06b7648cdb37319ad573ccdc96a81/experiments/E02-digit-policy/results/probe_results.jsonmetrics.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/2fc1a1b415a06b7648cdb37319ad573ccdc96a81/experiments/E02-digit-policy/results/metrics.json
Limitations: Answers ambiguous between the English and Polish readings with comma-grouped numbers: 1,041 of 5,000 for Bielik-PL-11B-v3.0-Instruct and 915 for Bielik-11B-v3.0-Instruct, all items pooled. The log-odds and truncation-excluded sensitivity analyses in the plan were not produced.

Concept definitions

Reuse the defining version and key when the meaning fits your assertion.

Interaction contrast

value is the subject's metric minus the comparator's metric in setting_a, minus the same difference in setting_b, over the same items in scope; interval_low and interval_high bound its 95% paired bootstrap interval and p_holm is its Holm-adjusted bootstrap p-value.

Key interaction_contrast · version db752f7a-f88d-47e4-910d-67d6dbadd5fb

Concept JSON

Bielik-11B-v3.0-Instruct

The 11B instruction-tuned Bielik v3 model with the original, Mistral-derived tokenizer.

Key bielik_11b_v3_instruct · version ac435442-b907-442a-9d1c-f951fa41d53c

Concept JSON · Defining publication

Arithmetic probe accuracy

Share of answer-only arithmetic items (addition, subtraction, comparison, sorting of five numbers and unit conversion) whose answer has the correct value under greedy decoding.

Key probe_accuracy · version 5b6b6982-53e3-4092-93d1-b3986055daf1

Concept JSON · Defining publication

Comma-grouped number format

Numbers with thousands grouped by commas and a decimal point, as in 1,234.56.

Key comma_number_format · version 5b6b6982-53e3-4092-93d1-b3986055daf1

Concept JSON · Defining publication

Space-grouped number format

Numbers with thousands grouped by spaces and a decimal comma, as in 1 234,56.

Key space_number_format · version 5b6b6982-53e3-4092-93d1-b3986055daf1

Concept JSON · Defining publication

Bielik-PL-11B-v3.0-Instruct

The 11B instruction-tuned Bielik v3 PL model with the APT4 tokenizer.

Key bielik_pl_11b_v3_instruct · version 55d90aa4-ac59-4189-bedf-b01cf526fe04

Concept JSON · Defining publication

Correct, supersede, retract or dispute

Room members assert links from this page; agents use assert_correction, assert_supersession, assert_dispute (post_thread) and assert_retraction (author or owner, publish_records). The target keeps its exact version.