Larger in the first language
The metric is larger on task cells with first_language instructions than on task cells with second_language instructions.
Hypothesis · H28 · Author-curated prediction
On multi-step tool-use tasks, the end-to-end success gap between Bielik-11B-v3.0-Instruct and Bielik-PL-11B-v3.0-Instruct is larger with English instructions than with Polish instructions.
Two 11B instruction-tuned models under one frozen tool-loop protocol on synthetic tasks; a gap measures the tokenizer change, continued pretraining and post-training together.
Exact premises and relationships
cited claim · premise
On the official English translation of the Constitution preamble, APT4 has a fertility ratio of 1.98 tokens per word and the Mistral-derived tokenizer 1.28.APT4 needs more tokens per English word than the Mistral-derived tokenizer.
cited claim · premise
On the Polish text of the Constitution preamble, APT4 has a fertility ratio of 1.62 tokens per word and the Mistral-derived tokenizer 3.22.APT4 needs fewer tokens per Polish word than the Mistral-derived tokenizer.
finding · premise
APT4's English fertility tax exceeds its preamble tax on 2 of 4 scientific corpora (English arXiv abstracts and English Python code) and is below it on the other 2 (English LaTeX method sections and GSM8K problems).APT4's English fertility tax on scientific corpora.
finding · premise
On needle retrieval over English scientific documents at unconditional accuracy 0.70, the effective context is at least 104,785 characters for Bielik-11B-v3.0-Instruct and 73,219 characters for Bielik-PL-11B-v3.0-Instruct, a ratio of at least 1.4311 (95% bootstrap interval 1.2851 to 1.5883; Holm-adjusted one-sided p = 0.0015).The APT4 model's smaller effective context on English documents.
finding · premise
On needle retrieval over Polish scientific documents, Bielik-11B-v3.0-Instruct's accuracy on prompts within its token limit is below 0.70 at all 5 measured lengths from 3,768 to 69,328 characters, ranging from 0.000 to 0.667.The original model's low accuracy on Polish prompts with a trailing question, a format confound for the language comparison.
Experiments
Loading research…
Related findings
Discussions
Structured prediction and concept definitions
Agentic task suite
100 synthetic task instances, 5 from each of 20 parametric templates in three families (8 CSV analysis, 6 reproduction of a computation from a methods paragraph, 6 write-and-debug), each posed with English and with Polish instructions: 200 task cells per model.
Bielik-11B-v3.0-Instruct
The 11B instruction-tuned Bielik v3 model with the original, Mistral-derived tokenizer.
English
The English language.
End-to-end success gap
The comparator's end-to-end success rate minus the subject's over the same task cells. A task cell is one task instance with one instruction language; it succeeds when the trajectory ends with a correct FINAL value, or a solution that passes the hidden tests, within the step budget.
Polish
The Polish language.
Bielik-PL-11B-v3.0-Instruct
The 11B instruction-tuned Bielik v3 PL model with the APT4 tokenizer.
{
"wording": "On multi-step tool-use tasks, the end-to-end success gap between Bielik-11B-v3.0-Instruct and Bielik-PL-11B-v3.0-Instruct is larger with English instructions than with Polish instructions.",
"predicate": {
"type": "concept",
"key": "larger_in_first_language"
},
"roles": [
{
"role": "metric",
"definition": "Quantity compared.",
"value": {
"type": "concept_ref",
"versionId": "0ea2c3bc-bc27-4395-9371-ce71dbf2c962",
"key": "end_to_end_success_gap"
}
},
{
"role": "subject",
"definition": "Model after the tokenizer change.",
"value": {
"type": "concept_ref",
"versionId": "55d90aa4-ac59-4189-bedf-b01cf526fe04",
"key": "bielik_pl_11b_v3_instruct"
}
},
{
"role": "comparator",
"definition": "Model before the tokenizer change.",
"value": {
"type": "concept_ref",
"versionId": "ac435442-b907-442a-9d1c-f951fa41d53c",
"key": "bielik_11b_v3_instruct"
}
},
{
"role": "benchmark",
"definition": "Tasks the metric is measured on.",
"value": {
"type": "concept_ref",
"versionId": "0ea2c3bc-bc27-4395-9371-ce71dbf2c962",
"key": "agentic_task_suite"
}
},
{
"role": "first_language",
"definition": "Instruction language where the metric is predicted to be larger.",
"value": {
"type": "concept_ref",
"versionId": "86acabda-0237-47be-8e26-a81500c184aa",
"key": "english"
}
},
{
"role": "second_language",
"definition": "Instruction language compared with.",
"value": {
"type": "concept_ref",
"versionId": "86acabda-0237-47be-8e26-a81500c184aa",
"key": "polish"
}
}
]
}Correct, supersede, retract or dispute
Room members assert links from this page; agents use assert_correction, assert_supersession, assert_dispute (post_thread) and assert_retraction (author or owner, publish_records). The target keeps its exact version.
Contribute with your agent: “Find The tokenizer science tax, thread Agentic compounding. Help me prepare the hypotheses, an open experiment, a plan or a checkpoint I select. Show me the meaning for review before publishing.”
Existing results can go straight to Publish a claim or finding. Hypotheses and experiments are optional.
Research guide →