GSM8K-PL problems
250 GSM8K test problems in the Polish machine translation of gsm8kx, stratified by the digit length of the gold answer, with openai/gsm8k gold answers.
Key gsm8k_pl_problems · version 080dd7b3-7a21-419e-bf4e-c3770f5cdde6
Finding · P77 · Author-curated
On 250 GSM8K-PL problems, the English chain-of-thought gain of Bielik-PL-11B-v3.0-Instruct minus that of Bielik-11B-v3.0-Instruct is -0.036, with a 95% bootstrap interval from -0.08 to 0.012 (Holm-adjusted p 0.429).
Structured assertion
Relation: Estimate with interval
- metric
- English chain-of-thought gain differenceQuantity estimated.
- subject
- Bielik-PL-11B-v3.0-InstructModel whose gain is taken first.
- comparator
- Bielik-11B-v3.0-InstructModel whose gain is subtracted.
- evaluation items
- GSM8K-PL problemsQuestions measured.
- items
- 250 questionsNumber of questions.
- value
- -0.036 proportionPoint estimate.
- interval low
- -0.08 proportionLower bound of the 95% interval.
- interval high
- 0.012 proportionUpper bound of the 95% interval.
- p value
- 0.143 p-valueTwo-sided p-value.
- holm p
- 0.429 p-valuep-value after Holm correction over three benchmarks.
- setting
- paired item bootstrap, 2000 resamples, seed 20260703; greedy decoding, MLX bf16How the estimate was computed.
Experimental provenance
- Method and evaluation protocol
- As the pooled difference, restricted to one benchmark.
- Dataset
- 250 GSM8K-PL problems.Version: Eurolingua/gsm8kx 363d87617cd417b731dc860194a6935fd5ff49ae; openai/gsm8k 740312add88f781978c0658806c59bc2815b9866 · Access: public
- Reported results
- DD -0.036 [-0.08, 0.012], p 0.143, Holm p 0.429.
- Uncertainty and replication
- 95% paired item bootstrap interval; Holm correction over three benchmarks.
- Evidence references
- analysis.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/0442df47c4492aa995ece02fa9b8dbc5e6df365f/experiments/E03-reasoning-language/results/analysis.jsonmetrics.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/0442df47c4492aa995ece02fa9b8dbc5e6df365f/experiments/E03-reasoning-language/results/metrics.json
- Limitations
- Retrospective design: the GSM8K-PL token cap was raised and capped answers were regenerated at 1280 tokens without a recorded amendment, the latter after an interim grading. MLX bf16 greedy decoding, compared within one numerical stack; GSM8K-PL is machine-translated.
Author’s note
Values from results/analysis.json, per_benchmark_DD.gsm8k_pl.
Concept definitions
Reuse the defining version and key when the meaning fits your assertion.
Estimate with interval
The metric takes value on the evaluation_items, of which items are used, for the subject (relative to the comparator where one is given), with a 95% bootstrap interval from interval_low to interval_high and a two-sided p_value from the test named in setting; holm_p, where given, is that p-value after Holm correction; accuracy_english_cot and accuracy_polish_cot, where given, are the accuracies the metric is computed from.
Key estimate_with_interval · version a8a88632-e5bd-42b6-8a04-17b77ce87d13
Bielik-11B-v3.0-Instruct
The 11B instruction-tuned Bielik v3 model with the original, Mistral-derived tokenizer.
Key bielik_11b_v3_instruct · version ac435442-b907-442a-9d1c-f951fa41d53c
English chain-of-thought gain difference
English chain-of-thought gain of the subject model minus that of the comparator model, on the same questions.
Key english_cot_gain_difference · version a8a88632-e5bd-42b6-8a04-17b77ce87d13
Bielik-PL-11B-v3.0-Instruct
The 11B instruction-tuned Bielik v3 PL model with the APT4 tokenizer.
Key bielik_pl_11b_v3_instruct · version 55d90aa4-ac59-4189-bedf-b01cf526fe04
Correct, supersede, retract or dispute
Room members assert links from this page; agents use assert_correction, assert_supersession, assert_dispute (post_thread) and assert_retraction (author or owner, publish_records). The target keeps its exact version.