Attributed research checkpoint
E2: APT4 splits digits like the Mistral-derived tokenizer but splits the no-break space into bytes, which lowers the PL model's arithmetic accuracy on no-break-space-grouped numbers; the digit mechanism for the GSM8K regression is partial.
Assignment, status and accepted plans remain authoritative on each experiment.
Reported state
- Reported progress
- E2 completed with a retrospective plan: digit and separator audit of nine tokenizers, 40,000 answer-only arithmetic generations on the 11B pair, and a 250-problem GSM8K subsample.
- Open issues
- The plan was never committed, and H1 and H2 were written after the tokenizer files had been inspected. The log-odds and truncation-excluded sensitivity analyses, the error-bucketing statement and per-rendering verdicts for H2 and H5 were not produced. Answers ambiguous between English and Polish readings differ between the models with comma-grouped numbers (1,041 against 915 of 5,000). The GSM8K subsample is sign-only, with an interval covering zero, under a protocol different from the paper's. The models differ by continued pretraining and post-training as well as by the tokenizer, so the model main effect is not attributed to the tokenizer. What causes the GSM8K difference remains open.
- Suggested next action
- Test whether mathematics and code in the recovery pretraining mix remove the GSM8K regression (E6).
- Access needs
- Both 11B model repositories are gated on Hugging Face; the downloaded snapshots and MLX conversions are restricted materials: ask the Room owner, or rebuild them at the pinned revisions.
Exact references
Exact record · P57
ab426953-7fb7-4a21-9c58-78317771084aH1 kill criterion not met: no segmentation difference on bare or comma-grouped numbers. H2 kill criterion met at the five renderings other than no-break-space grouping.
Exact record · P58
3d25bdbe-b09d-4c1a-9e65-5fa802f51d9dH2 at the no-break-space rendering: the ratio of tokens per number differs by more than 5%.
Exact record · P59
bd8acd13-639f-4b0d-bb2b-7a180e1f837cH5: its kill criterion (identical segmentation everywhere) is not met, but a degree sign after a space also differs beyond the two characters named.
Exact record · P64
db752f7a-f88d-47e4-910d-67d6dbadd5fbH3 killed: the interval of the pooled interaction covers 0.
Exact record · P65
73e6d8ab-194e-4a4a-ae68-c1cb3586ca61H4 confirmed: the interval of the no-break-space differential excludes 0.
Exact record · P69
d7c031db-103f-410d-b439-1f3ba6cae904Model main effect, attributed to the transplant pipeline by the attribution rule.
Exact record · P72
59f84e17-2dbc-4e7a-b481-9fe8fc19f29bSign-only check of the GSM8K difference: same sign as Table 6, interval covers 0.
Exact record · P73
2dbc96aa-9e0d-4ec0-97da-5cbab3338222GSM8K's exposure to the separator channel, as the decision rule requires.
Exact record · P74
b0f7daf6-d1ff-4a89-a255-3c00464a3b96Mechanism hypothesis under the plan's decision rule: partial.
Experiment · E2
Open experiment →Accepted plan
Exact plan →