Values by model and condition
The metric takes the value given in each cell role on the evaluation items under the setting; each cell role names one model and one prompting condition.
Key values_by_cell · version af0896b4-cbbe-43b0-996f-f873cf2ebb93
Finding · P87 · Author-curated
On 800 Polish STEM questions, the share of answers whose reasoning is classified as written in the prompted language is 0.9425 for Bielik-PL-11B-v3.0-Instruct and 0.9738 for Bielik-11B-v3.0-Instruct with Polish chain-of-thought, and 0.9238 and 0.905 with English chain-of-thought.
Structured assertion
Relation: Values by model and condition
- metric
- Trace language complianceQuantity measured.
- evaluation items
- Polish STEM question setQuestions measured.
- setting
- few-shot; greedy decoding; MLX bf16Decoding.
- bielik pl polish cot
- 0.9425 proportionValue for Bielik-PL-11B-v3.0-Instruct, Polish chain-of-thought.
- bielik pl english cot
- 0.9238 proportionValue for Bielik-PL-11B-v3.0-Instruct, English chain-of-thought.
- bielik polish cot
- 0.9738 proportionValue for Bielik-11B-v3.0-Instruct, Polish chain-of-thought.
- bielik english cot
- 0.905 proportionValue for Bielik-11B-v3.0-Instruct, English chain-of-thought.
Experimental provenance
- Method and evaluation protocol
- Classify the text before the final-answer marker as Polish, English or mixed from the share of Polish diacritics and counts of Polish and English stopwords; count matches with the prompted language.
- Dataset
- 800 Polish STEM questions: 250 GSM8K-PL problems, 300 LLMzSzŁ STEM and 250 PES examination questions.Version: Eurolingua/gsm8kx 363d87617cd417b731dc860194a6935fd5ff49ae; openai/gsm8k 740312add88f781978c0658806c59bc2815b9866; amu-cai/llmzszl-dataset 9c11d52a415147f1e2c44038a9d4d4155f064e1a; speakleash/PES-2018-2022 79ae158ceab8e4e083c8cfcacde56669407a94a2 · Access: restricted
- Reported results
- Bielik-PL-11B-v3.0-Instruct, Polish chain-of-thought 0.9425; Bielik-PL-11B-v3.0-Instruct, English chain-of-thought 0.9238; Bielik-11B-v3.0-Instruct, Polish chain-of-thought 0.9738; Bielik-11B-v3.0-Instruct, English chain-of-thought 0.905.
- Uncertainty and replication
- Exact counts over 800 answers per cell.
- Evidence references
- analysis.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/0442df47c4492aa995ece02fa9b8dbc5e6df365f/experiments/E03-reasoning-language/results/analysis.jsonmetrics.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/0442df47c4492aa995ece02fa9b8dbc5e6df365f/experiments/E03-reasoning-language/results/metrics.json
- Limitations
- Heuristic classifier; reported, not graded.
Author’s note
Values from results/analysis.json, compliance.<cell>.match.
Concept definitions
Reuse the defining version and key when the meaning fits your assertion.
Trace language compliance
Fraction of answers whose text before the final-answer marker is classified as written in the prompted language.
Key trace_language_compliance · version af0896b4-cbbe-43b0-996f-f873cf2ebb93
Polish STEM question set
800 questions posed in Polish: 250 GSM8K-PL problems, 300 LLMzSzŁ STEM examination questions and 250 PES examination questions.
Key polish_stem_questions · version a8a88632-e5bd-42b6-8a04-17b77ce87d13
Correct, supersede, retract or dispute
Room members assert links from this page; agents use assert_correction, assert_supersession, assert_dispute (post_thread) and assert_retraction (author or owner, publish_records). The target keeps its exact version.