Answers at the token cap
Number of answers whose generation stops because it reaches the maximum number of generated tokens.
Key answers_at_token_cap · version 95f8ccf1-de11-482c-99fa-d5240c8f6edd
Finding · P89 · Author-curated
On 300 LLMzSzŁ STEM questions with a 768-token cap, 47 answers of Bielik-PL-11B-v3.0-Instruct and 26 of Bielik-11B-v3.0-Instruct reach the cap with Polish chain-of-thought, and 15 and 1 with English chain-of-thought.
Structured assertion
Relation: Values by model and condition
- metric
- Answers at the token capQuantity counted.
- evaluation items
- LLMzSzŁ STEM questionsQuestions measured.
- setting
- max_tokens 768; few-shot; greedy decoding; MLX bf16Decoding.
- bielik pl polish cot
- 47 answersValue for Bielik-PL-11B-v3.0-Instruct, Polish chain-of-thought.
- bielik pl english cot
- 15 answersValue for Bielik-PL-11B-v3.0-Instruct, English chain-of-thought.
- bielik polish cot
- 26 answersValue for Bielik-11B-v3.0-Instruct, Polish chain-of-thought.
- bielik english cot
- 1 answersValue for Bielik-11B-v3.0-Instruct, English chain-of-thought.
Experimental provenance
- Method and evaluation protocol
- Count the answers marked as regenerated by the extension step, which regenerated exactly the answers that had stopped at the cap.
- Dataset
- 300 LLMzSzŁ STEM examination questions.Version: amu-cai/llmzszl-dataset 9c11d52a415147f1e2c44038a9d4d4155f064e1a · Access: restricted
- Reported results
- Bielik-PL-11B-v3.0-Instruct, Polish chain-of-thought 47; Bielik-PL-11B-v3.0-Instruct, English chain-of-thought 15; Bielik-11B-v3.0-Instruct, Polish chain-of-thought 26; Bielik-11B-v3.0-Instruct, English chain-of-thought 1.
- Uncertainty and replication
- Exact counts over 300 answers per cell.
- Evidence references
- truncation_audit.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/0442df47c4492aa995ece02fa9b8dbc5e6df365f/experiments/E03-reasoning-language/results/truncation_audit.jsonmetrics.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/0442df47c4492aa995ece02fa9b8dbc5e6df365f/experiments/E03-reasoning-language/results/metrics.json
- Limitations
- Counted from marks in the complete generations; the generations at the first cap were overwritten by the regeneration.
Author’s note
Values from results/truncation_audit.json, models.<model>.cells["llmzszl_stem|<condition>"].regenerated.
Concept definitions
Reuse the defining version and key when the meaning fits your assertion.
Values by model and condition
The metric takes the value given in each cell role on the evaluation items under the setting; each cell role names one model and one prompting condition.
Key values_by_cell · version af0896b4-cbbe-43b0-996f-f873cf2ebb93
LLMzSzŁ STEM questions
300 multiple-choice questions in mathematics, physics, science and biology from the LLMzSzŁ test split, excluding vocational examinations and questions that refer to figures or tables.
Key llmzszl_stem_questions · version 4c39157e-9b47-4096-b640-a79bf90a103d
Correct, supersede, retract or dispute
Room members assert links from this page; agents use assert_correction, assert_supersession, assert_dispute (post_thread) and assert_retraction (author or owner, publish_records). The target keeps its exact version.