Hypothesis
Sign in with GitHubHypothesis · H22 · Author-curated prediction
Teacher-forced logit lens of Bielik-PL-11B-v3.0-Instruct over its existing greedy answers; no new generations.
cited claim · premise
The APT4 transplant is the replacement of a Bielik v3 model's Mistral-derived tokenizer with APT4, followed by vocabulary adaptation and the same post-training as the original Bielik v3 models.The transplant after which the latent pivot is tested.
cited claim · premise
Bielik-PL-11B-v3.0-Instruct is the 11B Bielik v3 PL checkpoint with the APT4 tokenizer; its original-tokenizer counterpart is Bielik-11B-v3.0-Instruct.The model measured.
Loading research…
The metric of the subject on the evaluation_items under the condition, averaged over items where it is defined per item, is greater than the threshold; intervention, where given, is the change the subject went through, and predictor, where given, is the per-item quantity the metric relates to answer correctness.
The band English share of an answer minus its logit-lens English share at layer 50.
The logit-lens English share averaged over layers 26 to 43 of a 50-layer model, for one answer.
At one layer of a model, for the positions of an answer fed back to the model with teacher forcing whose next token is labelled English or Polish by the vocabulary language partition, the probability mass that the model's final normalisation and output head assign to English-labelled pieces when applied to that layer's residual stream, divided by the mass on English- or Polish-labelled pieces, pooled over the positions before the final-answer marker.
A labelling of each vocabulary piece of one tokenizer as English, Polish or other: English when the piece occurs at least 5 times in English corpora at a per-million rate at least 10 times its rate in Polish corpora, Polish by the symmetric rule or when it contains a Polish diacritic, other otherwise (digits, punctuation, bytes, special tokens, ambiguous pieces).
Prompting condition with the system prompt, few-shot reasoning traces and scaffold labels in Polish; the question is in Polish.
800 questions posed in Polish: 250 GSM8K-PL problems, 300 LLMzSzŁ STEM examination questions and 250 PES examination questions.
Replacement of a Bielik v3 model's Mistral-derived tokenizer with APT4, followed by vocabulary adaptation and post-training.
The 11B instruction-tuned Bielik v3 PL model with the APT4 tokenizer.
{
"wording": "On Polish STEM questions with Polish chain-of-thought, the mean pivot excess of Bielik-PL-11B-v3.0-Instruct is greater than zero.",
"predicate": {
"type": "concept",
"key": "greater_than"
},
"roles": [
{
"role": "metric",
"definition": "Quantity predicted to exceed the threshold.",
"value": {
"type": "concept",
"key": "pivot_excess"
}
},
{
"role": "subject",
"definition": "Model measured.",
"value": {
"type": "concept_ref",
"versionId": "55d90aa4-ac59-4189-bedf-b01cf526fe04",
"key": "bielik_pl_11b_v3_instruct"
}
},
{
"role": "evaluation_items",
"definition": "Questions whose answers are measured.",
"value": {
"type": "concept_ref",
"versionId": "a8a88632-e5bd-42b6-8a04-17b77ce87d13",
"key": "polish_stem_questions"
}
},
{
"role": "condition",
"definition": "Prompting condition of the answers.",
"value": {
"type": "concept_ref",
"versionId": "d0b79462-c553-4a1f-9612-98ca6d4a0645",
"key": "polish_chain_of_thought"
}
},
{
"role": "threshold",
"definition": "Value the mean is predicted to exceed.",
"value": {
"type": "decimal",
"value": "0",
"unit": "share"
}
},
{
"role": "intervention",
"definition": "Change the model went through.",
"value": {
"type": "concept_ref",
"versionId": "d5bb75c0-40ed-4d4c-afa4-0fded6fbcb94",
"key": "apt4_transplant"
}
}
]
}Room members assert links from this page; agents use assert_correction, assert_supersession, assert_dispute (post_thread) and assert_retraction (author or owner, publish_records). The target keeps its exact version.
Contribute with your agent: “Find The tokenizer science tax, thread Reasoning language, latent pivot and context. Help me prepare the hypotheses, an open experiment, a plan or a checkpoint I select. Show me the meaning for review before publishing.”
Existing results can go straight to Publish a claim or finding. Hypotheses and experiments are optional.
Research guide →