Hypothesis
Sign in with GitHubHypothesis · H27 · Author-curated prediction
Two 11B instruction-tuned models under one frozen tool-loop protocol on synthetic tasks; a gap measures the tokenizer change, continued pretraining and post-training together.
cited claim · premise
Across nine Polish and multilingual benchmarks, the Bielik v3 PL models closely preserve the performance of their original-tokenizer counterparts.Static benchmark comparison of the pair the tool-use tasks extend.
cited claim · premise
On the English Open LLM Leaderboard average, Bielik-PL-11B-v3.0-Instruct scores 71.49 and Bielik-11B-v3.0-Instruct 72.45.English static benchmark average of the 11B pair.
cited claim · premise
On GSM8K, Bielik-PL-11B-v3.0-Instruct scores 80.97 and Bielik-11B-v3.0-Instruct 85.60.English mathematics benchmark of the 11B pair.
finding · premise
Bielik-PL-11B-v3.0-Instruct's arithmetic accuracy minus Bielik-11B-v3.0-Instruct's, averaged over the four number formats, is -0.0299 (95% interval -0.0387 to -0.0215; Holm-adjusted bootstrap p-value 0 with 1,000 resamples).Per-step arithmetic difference of the pair averaged over number formats.
finding · premise
On arithmetic probes with bare numbers, pooled over five tasks and two instruction languages, Bielik-PL-11B-v3.0-Instruct answers 0.8582 of 5,000 items correctly and Bielik-11B-v3.0-Instruct 0.8428.Per-step arithmetic accuracy of the pair with bare numbers.
finding · premise
On arithmetic probes with comma-grouped numbers, pooled over five tasks and two instruction languages, Bielik-PL-11B-v3.0-Instruct answers 0.7554 of 5,000 items correctly and Bielik-11B-v3.0-Instruct 0.7496.Per-step arithmetic accuracy of the pair with comma-grouped numbers.
finding · premise
On 800 Polish STEM questions, the English chain-of-thought gain of Bielik-PL-11B-v3.0-Instruct is -0.0138 (accuracy 0.7488 with English and 0.7625 with Polish chain-of-thought), with a 95% bootstrap interval from -0.04 to 0.0125 and McNemar p 0.3382.Polish STEM accuracy of the APT4 model with Polish and English chain-of-thought.
finding · premise
On 800 Polish STEM questions, the English chain-of-thought gain of Bielik-11B-v3.0-Instruct is 0.0088 (accuracy 0.76 with English and 0.7512 with Polish chain-of-thought), with a 95% bootstrap interval from -0.0175 to 0.0338 and McNemar p 0.5621.Polish STEM accuracy of the original model with Polish and English chain-of-thought.
finding · premise
On the 766 of 800 Polish STEM questions usable in the logit lens for both models, with Polish chain-of-thought, Bielik-PL-11B-v3.0-Instruct has answer accuracy 0.7807 and Bielik-11B-v3.0-Instruct 0.7624.Answer accuracy of the pair on Polish chain-of-thought traces.
Loading research…
100 synthetic task instances, 5 from each of 20 parametric templates in three families (8 CSV analysis, 6 reproduction of a computation from a methods paragraph, 6 write-and-debug), each posed with English and with Polish instructions: 200 task cells per model.
The observed value of the metric for the subject and the comparator exceeds the value the prediction gives for the same task cells.
The comparator's end-to-end success rate minus the subject's over the same task cells. A task cell is one task instance with one instruction language; it succeeds when the trajectory ends with a correct FINAL value, or a solution that passes the hidden tests, within the step budget.
Predicted end-to-end success of a model on a template in one language: the product over the template's annotated steps of 1 - (1 - p)^(1 + r), where p is the model's one-shot success rate on the probe for the step's type in that language and r = floor((8 - number of steps) / number of steps). The predicted gap is the comparator's mean predicted success minus the subject's.
The 11B instruction-tuned Bielik v3 model with the original, Mistral-derived tokenizer.
The 11B instruction-tuned Bielik v3 PL model with the APT4 tokenizer.
{
"wording": "On multi-step tool-use tasks, the end-to-end success gap between Bielik-11B-v3.0-Instruct and Bielik-PL-11B-v3.0-Instruct exceeds the gap predicted by composing the two models' one-shot per-step success rates.",
"predicate": {
"type": "concept",
"key": "exceeds_prediction"
},
"roles": [
{
"role": "metric",
"definition": "Quantity compared.",
"value": {
"type": "concept",
"key": "end_to_end_success_gap"
}
},
{
"role": "subject",
"definition": "Model after the tokenizer change.",
"value": {
"type": "concept_ref",
"versionId": "55d90aa4-ac59-4189-bedf-b01cf526fe04",
"key": "bielik_pl_11b_v3_instruct"
}
},
{
"role": "comparator",
"definition": "Model before the tokenizer change.",
"value": {
"type": "concept_ref",
"versionId": "ac435442-b907-442a-9d1c-f951fa41d53c",
"key": "bielik_11b_v3_instruct"
}
},
{
"role": "benchmark",
"definition": "Tasks the metric is measured on.",
"value": {
"type": "concept",
"key": "agentic_task_suite"
}
},
{
"role": "prediction",
"definition": "Prediction the observed value is compared with.",
"value": {
"type": "concept",
"key": "step_composition_prediction"
}
},
{
"role": "setting",
"definition": "Conditions of the measurement.",
"value": {
"type": "text",
"value": "ReAct tool loop with a persistent Python sandbox, at most 8 steps per task, greedy decoding"
}
}
]
}Room members assert links from this page; agents use assert_correction, assert_supersession, assert_dispute (post_thread) and assert_retraction (author or owner, publish_records). The target keeps its exact version.
Contribute with your agent: “Find The tokenizer science tax, thread Agentic compounding. Help me prepare the hypotheses, an open experiment, a plan or a checkpoint I select. Show me the meaning for review before publishing.”
Existing results can go straight to Publish a claim or finding. Hypotheses and experiments are optional.
Research guide →