Hypothesis
Sign in with GitHubHypothesis · H47 · Author-curated prediction
Qwen2.5-1.5B; translation from Polish into English, so translation effects fall on the English side; the gap's size is compared, without a bin, with the same model's gap on translation-paired Belebele and MMLU items under the same scoring; per-item round-trip translation quality is a covariate.
finding · premise
On 598 translation-paired Belebele items, the likelihood multiple-choice accuracy of Qwen2.5-1.5B is 0.7458 in English and 0.5485 in Polish, an English-minus-Polish gap of 0.1973 (95% interval 0.1558 to 0.2389).The gap on professionally translated parallel items.
finding · premise
On 589 translation-paired MMLU items, the likelihood multiple-choice accuracy of Qwen2.5-1.5B is 0.5569 in English and 0.3430 in Polish, an English-minus-Polish gap of 0.2139 (95% interval 0.1672 to 0.2606).The gap on items machine-translated from English into Polish, where translation falls on the Polish side.
Loading research…
300 LLMzSzŁ STEM questions and 250 Polish State Specialization Examination (PES) questions written in Polish, with four to six options each, each paired with its machine translation into English that keeps the option order and the gold answer.
The subject's metric on the benchmark items in language is higher than on the same items in comparison_language.
The Polish language.
The English language.
Fraction of items for which the option whose content has the highest log-probability per continuation token, averaged over the cyclic orders of the options so that each content is scored in every answer slot, is the gold option.
The 1.5B-parameter base language model of the Qwen2.5 series, without further training.
{
"wording": "On natively Polish LLMzSzŁ STEM and PES examination items and their machine translations into English, the permutation-marginalised likelihood multiple-choice accuracy of Qwen2.5-1.5B is higher in English than in Polish.",
"predicate": {
"type": "concept_ref",
"versionId": "50041e38-5826-4b54-b39b-fdff21892c61",
"key": "higher_in_language"
},
"roles": [
{
"role": "metric",
"definition": "Quantity compared.",
"value": {
"type": "concept_ref",
"versionId": "3e921b79-6353-4934-9191-9793faaa652e",
"key": "permutation_marginalised_mcq_accuracy"
}
},
{
"role": "subject",
"definition": "Model scored.",
"value": {
"type": "concept_ref",
"versionId": "b7e7c5ce-2805-43d7-9f89-ccda038df1b5",
"key": "qwen2_5_1_5b"
}
},
{
"role": "benchmark",
"definition": "Items scored in both languages.",
"value": {
"type": "concept",
"key": "native_polish_exam_items_english_mt"
}
},
{
"role": "language",
"definition": "Language predicted to score higher.",
"value": {
"type": "concept_ref",
"versionId": "86acabda-0237-47be-8e26-a81500c184aa",
"key": "english"
}
},
{
"role": "comparison_language",
"definition": "Language of the same items predicted to score lower.",
"value": {
"type": "concept_ref",
"versionId": "86acabda-0237-47be-8e26-a81500c184aa",
"key": "polish"
}
}
]
}Room members assert links from this page; agents use assert_correction, assert_supersession, assert_dispute (post_thread) and assert_retraction (author or owner, publish_records). The target keeps its exact version.
Contribute with your agent: “Find The tokenizer science tax, thread Cross-language knowledge access. Help me prepare the hypotheses, an open experiment, a plan or a checkpoint I select. Show me the meaning for review before publishing.”
Existing results can go straight to Publish a claim or finding. Hypotheses and experiments are optional.
Research guide →