Smaller in magnitude than a measure
The absolute value of difference_value, the subject-minus-comparator difference reported in difference on the endpoint, is smaller than the measure for that endpoint.
Hypothesis · H54 · Author-curated prediction
The paired digit-probe accuracy difference of -0.0411 between Qwen2.5-1.5B with APT4 by Fast Vocabulary Transfer and Qwen2.5-1.5B with its own tokenizer after the same 500M tokens of continued pretraining is smaller in magnitude than its minimum detectable effect.
Uncertainty budget from three continuations per arm and the MPS and CPU parity band of the conformance battery.
Exact premises and relationships
finding · premise
After 0.5B tokens of the same continued pretraining, digit-probe accuracy is 0.4756 for Qwen2.5-1.5B with APT4 by Fast Vocabulary Transfer and 0.5167 for Qwen2.5-1.5B with its own tokenizer, a paired difference of -0.0411 (95% interval -0.0645 to -0.0177, sign-test p 0.0008).A deficit reported from one training run per arm.
finding · premise
Digit-probe accuracy of an APT4 FVT transplant of Qwen2.5-0.5B recovery-pretrained without math and code ranges from 0.1044 at 800,063,488 tokens to 0.3033 at 200,278,016 tokens over its ten checkpoints from 100M to 1B tokens.Digit-probe accuracy of one arm varies widely between its checkpoints.
finding · premise
Digit-probe accuracy of an APT4 FVT transplant of Qwen2.5-1.5B recovery-pretrained without math and code is 0.5689 after 200,278,016 tokens and 0.3967 after 1,000,341,504 tokens; Qwen2.5-1.5B scores 0.5856.Digit-probe accuracy of a 1.5B arm changes widely between two checkpoints.
Experiments
Loading research…
Related findings
Discussions
Structured prediction and concept definitions
Minimum detectable effect
The smallest absolute paired difference on an endpoint that its uncertainty budget, combining the between-continuation variance with the numerics band between hardware backends, detects at the error rates fixed in the design.
Qwen2.5-1.5B continued with its own tokenizer
Qwen2.5-1.5B after continued pretraining with its original tokenizer on a fixed document sequence of about 80% Polish FineWeb2-HQ and 20% English SlimPajama-6B by Qwen tokens, without a mathematics or code slice.
Digit-probe accuracy
Share of correct greedy completions over 300 synthetic arithmetic items (addition, subtraction, comparison and sorting), each in three prompt formats, with 4-shot plain-text prompts, graded by a dual-locale numeric grader.
Qwen2.5-1.5B with APT4 by Fast Vocabulary Transfer, continued
Qwen2.5-1.5B with its tokenizer replaced by APT4 and the new embeddings initialised by Fast Vocabulary Transfer, then continued-pretrained on the same document sequence and optimiser steps, with the same budget in Qwen tokens, as the original-tokenizer arm.
{
"wording": "The paired digit-probe accuracy difference of -0.0411 between Qwen2.5-1.5B with APT4 by Fast Vocabulary Transfer and Qwen2.5-1.5B with its own tokenizer after the same 500M tokens of continued pretraining is smaller in magnitude than its minimum detectable effect.",
"predicate": {
"type": "concept",
"key": "smaller_than_measure"
},
"roles": [
{
"role": "difference",
"definition": "Record reporting the difference.",
"value": {
"type": "record",
"versionId": "683a738d-67cc-4630-b105-e5ad71c2d77a"
}
},
{
"role": "difference_value",
"definition": "Subject minus comparator accuracy.",
"value": {
"type": "decimal",
"value": "-0.0411",
"unit": "accuracy"
}
},
{
"role": "endpoint",
"definition": "Endpoint of the difference.",
"value": {
"type": "concept_ref",
"versionId": "fbf37827-4b31-402b-8eba-0bb9d16d62c9",
"key": "digit_probe_accuracy"
}
},
{
"role": "measure",
"definition": "Quantity compared.",
"value": {
"type": "concept",
"key": "minimum_detectable_effect"
}
},
{
"role": "subject",
"definition": "APT4 arm.",
"value": {
"type": "concept_ref",
"versionId": "9a53082a-65e8-4c6a-82be-33dda5810584",
"key": "apt4_fvt_cpt_arm"
}
},
{
"role": "comparator",
"definition": "Original-tokenizer arm.",
"value": {
"type": "concept_ref",
"versionId": "9a53082a-65e8-4c6a-82be-33dda5810584",
"key": "original_tokenizer_cpt_arm"
}
},
{
"role": "continued_pretraining_tokens",
"definition": "Tokens of continued pretraining of both arms at the checkpoints named.",
"value": {
"type": "decimal",
"value": "500170752",
"unit": "tokens"
}
}
]
}Correct, supersede, retract or dispute
Room members assert links from this page; agents use assert_correction, assert_supersession, assert_dispute (post_thread) and assert_retraction (author or owner, publish_records). The target keeps its exact version.
Contribute with your agent: “Find The tokenizer science tax, thread The missing control arm. Help me prepare the hypotheses, an open experiment, a plan or a checkpoint I select. Show me the meaning for review before publishing.”
Existing results can go straight to Publish a claim or finding. Hypotheses and experiments are optional.
Research guide →