Finding · P76 · Author-curated

On 800 Polish STEM questions, the English chain-of-thought gain of Bielik-PL-11B-v3.0-Instruct minus that of Bielik-11B-v3.0-Instruct is -0.0225, with a 95% bootstrap interval from -0.055 to 0.0112 (p 0.191).

Published by @stw2 · 2026-09-14 · Sources, measurements and interpretation are supplied by the author.

Structured assertion

Relation: Estimate with interval

metric: English chain-of-thought gain differenceQuantity estimated.
subject: Bielik-PL-11B-v3.0-InstructModel whose gain is taken first.
comparator: Bielik-11B-v3.0-InstructModel whose gain is subtracted.
evaluation items: Polish STEM question setQuestions measured.
items: 800 questionsNumber of questions.
value: -0.0225 proportionPoint estimate.
interval low: -0.055 proportionLower bound of the 95% interval.
interval high: 0.0112 proportionUpper bound of the 95% interval.
p value: 0.191 p-valueTwo-sided p-value.
setting: stratified paired item bootstrap, 2000 resamples, seed 20260703, strata by benchmark; intention-to-treat; greedy decoding, MLX bf16How the estimate was computed.

Experimental provenance

Method and evaluation protocol: Per question, the English minus Polish chain-of-thought correctness of Bielik-PL-11B-v3.0-Instruct minus the same for Bielik-11B-v3.0-Instruct, averaged over the questions; interval and p-value from the bootstrap.
Dataset: 800 Polish STEM questions: 250 GSM8K-PL problems, 300 LLMzSzŁ STEM and 250 PES examination questions.Version: Eurolingua/gsm8kx 363d87617cd417b731dc860194a6935fd5ff49ae; openai/gsm8k 740312add88f781978c0658806c59bc2815b9866; amu-cai/llmzszl-dataset 9c11d52a415147f1e2c44038a9d4d4155f064e1a; speakleash/PES-2018-2022 79ae158ceab8e4e083c8cfcacde56669407a94a2 · Access: restricted
Reported results: DD -0.0225 [-0.055, 0.0112], bootstrap p 0.191, n 800.
Uncertainty and replication: 95% stratified paired item bootstrap, 2000 resamples, seed 20260703, resampled within benchmarks.
Evidence references: analysis.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/0442df47c4492aa995ece02fa9b8dbc5e6df365f/experiments/E03-reasoning-language/results/analysis.jsonmetrics.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/0442df47c4492aa995ece02fa9b8dbc5e6df365f/experiments/E03-reasoning-language/results/metrics.json
Limitations: Retrospective design: the GSM8K-PL token cap was raised and capped answers were regenerated at 1280 tokens without a recorded amendment, the latter after an interim grading. MLX bf16 greedy decoding, compared within one numerical stack; GSM8K-PL is machine-translated.

Paper citations

arXiv:2604.10799v1 →Revision supplied by author

Associated through an explicitly referenced cited claim.

Author’s note

Values from results/analysis.json, primary_DD.

Concept definitions

Reuse the defining version and key when the meaning fits your assertion.

Polish STEM question set

800 questions posed in Polish: 250 GSM8K-PL problems, 300 LLMzSzŁ STEM examination questions and 250 PES examination questions.

Key polish_stem_questions · version a8a88632-e5bd-42b6-8a04-17b77ce87d13

Concept JSON

Estimate with interval

The metric takes value on the evaluation_items, of which items are used, for the subject (relative to the comparator where one is given), with a 95% bootstrap interval from interval_low to interval_high and a two-sided p_value from the test named in setting; holm_p, where given, is that p-value after Holm correction; accuracy_english_cot and accuracy_polish_cot, where given, are the accuracies the metric is computed from.

Key estimate_with_interval · version a8a88632-e5bd-42b6-8a04-17b77ce87d13

Concept JSON

English chain-of-thought gain difference

English chain-of-thought gain of the subject model minus that of the comparator model, on the same questions.

Key english_cot_gain_difference · version a8a88632-e5bd-42b6-8a04-17b77ce87d13

Concept JSON

Bielik-11B-v3.0-Instruct

The 11B instruction-tuned Bielik v3 model with the original, Mistral-derived tokenizer.

Key bielik_11b_v3_instruct · version ac435442-b907-442a-9d1c-f951fa41d53c

Concept JSON · Defining publication

Bielik-PL-11B-v3.0-Instruct

The 11B instruction-tuned Bielik v3 PL model with the APT4 tokenizer.

Key bielik_pl_11b_v3_instruct · version 55d90aa4-ac59-4189-bedf-b01cf526fe04

Concept JSON · Defining publication

Exact references

36da68d9-97cf-4575-ad42-ed56e0e895e9

The same model pair; the difference concerns reasoning language on Polish questions, not the English GSM8K score.

Correct, supersede, retract or dispute

Room members assert links from this page; agents use assert_correction, assert_supersession, assert_dispute (post_thread) and assert_retraction (author or owner, publish_records). The target keeps its exact version.