On 589 translation-paired MMLU items, Qwen2.5-1.5B's English likelihood multiple-choice accuracy is 0.3964 on the 111 STEM pairs and 0.5941 on the 478 non-STEM pairs.

Relation: Metric comparison

metric

Likelihood multiple-choice accuracyQuantity compared.

benchmark

Translation-paired MMLU itemsItem pairs scored.

language

EnglishLanguage of the items.

setting

Qwen2.5-1.5BModel scored.

subject

MMLU STEM subjectsFirst subset.

subject value

0.3964 accuracyAccuracy on the STEM pairs.

comparator

MMLU pairs of the other 38 subjectsSecond subset.

comparator value

0.5941 accuracyAccuracy on the non-STEM pairs.

Method and evaluation protocol

Length-normalised accuracy of the base model's English rows of scores_base.jsonl, exclusions applied, split by the STEM tag.

Dataset

589 pairs of MMLU test questions in English and in the openGPT-X machine translation into Polish: a seed-42 sample of 600 with the structurally malformed pairs excluded; 111 pairs belong to STEM subjects.Version: cais/mmlu@c30699e8356da336a370243923dbaf21066bb9fe; openGPT-X/mmlux@7fb62cf05c1ce9ea550dbfd2ea6c145204344882 · Access: public

Reported results

STEM 0.3964, non-STEM 0.5941.

Uncertainty and replication

Point values without intervals.

Evidence references

metrics.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/5e8117ffb3f168331ee63f1032cd40a62c6296bd/experiments/E11-cross-language-access/results/metrics.json

Limitations

Computed in the port from the committed score file; the original report states the same values. One model family at 1.5B parameters; the checkpoints are pre-anneal, from one training run each. English MMLU items may be in Qwen2.5 pretraining data, so MMLU gaps can be inflated; Polish MMLU items are machine translations. The 13 excluded pairs were chosen after the base model's cells had been scored.