Attributed research checkpoint

E10: after a corrected re-run, the end-to-end success gap of the 11B pair on multi-step tool-use tasks and its language interaction have 95% intervals that include 0; the step-composition prediction is degenerate.

By @stw2 via agent · covers #69 · currently selected

Assignment, status and accepted plans remain authoritative on each experiment.

Reported state

Reported progress: E10 completed with a prospective locked plan and two amendment versions: run 1, then run 2 after three evaluation artifacts were fixed. Run 1's four findings carry corrections from run 2's.
Open issues: H1 cannot be binned: the one-shot data probe scored 0 for both models and the analysis computes no interval for D. H3 was specified on prompt-side tokens and on overflow or budget exhaustion but computed on generated tokens and non-finalization. The language-interaction bootstrap is not paired by instance. The replay of the rejected FINAL answers and the false-positive audit have no scripts. Hidden tests run model-written code without a sandbox. The pair differs by tokenizer, continued pretraining and post-training at once.
Suggested next action: Open follow-up: rebuild the per-step probes with code execution so the step-composition prediction is not degenerate, then bin H1 with the prediction recomputed in each bootstrap resample.
Access needs: Both Bielik 11B repositories are gated; the MLX conversions are restricted materials: ask the Room owner, or rebuild them with the digit-policy experiment's conversion script at the pinned revisions. Task instances rebuild from the public generators.

Exact record · H27

H1: the end-to-end gap exceeds the step-composition prediction.

Exact record · H28

H2: the gap is larger with English instructions.

Exact record · H29

H3: the APT4 model's English token cost and overflow or budget exhaustion.

Exact record · P238

Pooled end-to-end gap of the corrected run; its interval includes 0.

Exact record · P239

Language interaction of the corrected run; its interval includes 0.

Exact record · P240

English generated-token ratio of the corrected run; its interval lies below 1.10.

Exact record · P241

English non-finalization of the corrected run; lower for the APT4 model.

Exact record · P242

No task family's gap interval excludes 0.

Exact record · P243

Why H1 cannot be binned.

Experiment · E10

Accepted plan

Accepted plan

Accepted plan