Finding

Sign in with GitHub
← Publications

Finding · P81 · Author-curated

On the 770 of 800 Polish STEM questions with no answer graded truncated for either model under either condition, the English chain-of-thought gain of Bielik-PL-11B-v3.0-Instruct minus that of Bielik-11B-v3.0-Instruct is -0.026, with a 95% bootstrap interval from -0.0597 to 0.0078.

Published by @stw2 · 2026-09-14 · Sources, measurements and interpretation are supplied by the author.

Structured assertion

Relation: Estimate with interval

subject
Bielik-PL-11B-v3.0-InstructModel whose gain is taken first.
comparator
Bielik-11B-v3.0-InstructModel whose gain is subtracted.
evaluation items
Polish STEM question setQuestion set the subset is drawn from.
items
770 questionsNumber of questions in the subset.
value
-0.026 proportionPoint estimate.
interval low
-0.0597 proportionLower bound of the 95% interval.
interval high
0.0078 proportionUpper bound of the 95% interval.
setting
subset: questions with no answer graded truncated in any of the four model and condition combinations; unstratified paired item bootstrap, 2000 resamples, seed 20260703Subset and computation.

Experimental provenance

Method and evaluation protocol
The pooled difference recomputed on the subset of questions.
Dataset
800 Polish STEM questions: 250 GSM8K-PL problems, 300 LLMzSzŁ STEM and 250 PES examination questions.Version: Eurolingua/gsm8kx 363d87617cd417b731dc860194a6935fd5ff49ae; openai/gsm8k 740312add88f781978c0658806c59bc2815b9866; amu-cai/llmzszl-dataset 9c11d52a415147f1e2c44038a9d4d4155f064e1a; speakleash/PES-2018-2022 79ae158ceab8e4e083c8cfcacde56669407a94a2 · Access: restricted
Reported results
DD -0.026 [-0.0597, 0.0078], n 770.
Uncertainty and replication
95% paired item bootstrap interval, not stratified by benchmark.
Limitations
Excluding truncated answers conditions on an outcome of generation. Retrospective design: the GSM8K-PL token cap was raised and capped answers were regenerated at 1280 tokens without a recorded amendment, the latter after an interim grading. MLX bf16 greedy decoding, compared within one numerical stack; GSM8K-PL is machine-translated.

Author’s note

Values from results/analysis.json, sensitivity_truncation_excluded_DD.

Concept definitions

Reuse the defining version and key when the meaning fits your assertion.

Estimate with interval

The metric takes value on the evaluation_items, of which items are used, for the subject (relative to the comparator where one is given), with a 95% bootstrap interval from interval_low to interval_high and a two-sided p_value from the test named in setting; holm_p, where given, is that p-value after Holm correction; accuracy_english_cot and accuracy_polish_cot, where given, are the accuracies the metric is computed from.

Key estimate_with_interval · version a8a88632-e5bd-42b6-8a04-17b77ce87d13

Concept JSON · Defining publication

Bielik-11B-v3.0-Instruct

The 11B instruction-tuned Bielik v3 model with the original, Mistral-derived tokenizer.

Key bielik_11b_v3_instruct · version ac435442-b907-442a-9d1c-f951fa41d53c

Concept JSON · Defining publication

Polish STEM question set

800 questions posed in Polish: 250 GSM8K-PL problems, 300 LLMzSzŁ STEM examination questions and 250 PES examination questions.

Key polish_stem_questions · version a8a88632-e5bd-42b6-8a04-17b77ce87d13

Concept JSON · Defining publication

English chain-of-thought gain difference

English chain-of-thought gain of the subject model minus that of the comparator model, on the same questions.

Key english_cot_gain_difference · version a8a88632-e5bd-42b6-8a04-17b77ce87d13

Concept JSON · Defining publication

Bielik-PL-11B-v3.0-Instruct

The 11B instruction-tuned Bielik v3 PL model with the APT4 tokenizer.

Key bielik_pl_11b_v3_instruct · version 55d90aa4-ac59-4189-bedf-b01cf526fe04

Concept JSON · Defining publication

Exact references

refines

a8a88632-e5bd-42b6-8a04-17b77ce87d13

Rerun of the primary difference on a subset of the questions.