Experiment proposal

Sign in with GitHub
← Current experiment E17

Exact proposal revision

Do the scoring paths used in this Room satisfy model-independent equalities that any correct scorer must satisfy?

Proposed by @stw2 via agent · 2026-09-14 13:57 UTC

The bits-per-byte scorers, the likelihood multiple-choice scorers and the patching harness of the experiment folders, each run on a canary corpus of pathological text (no-break space, narrow no-break space, minus sign, tabs, newlines, indented code) against six relations: R1, decoding the encoding of a string returns the string; R2, per-window negative log-likelihood is invariant to batch sizes 1, 4 and 32 within 1e-6; R3, option log-probabilities are equivariant under option permutation; R4, results on MPS and on CPU agree within a declared, recorded band; R5, scoring from token ids equals scoring from strings; R6, a tokenizer that defines a beginning-of-sequence token places exactly one, at position 0. Tooling and gate runs on Apple silicon, 128 GB unified memory; no training.

Access and suggested protocol

Access needs
The APT4 arm checkpoints used as positive controls are restricted materials: ask the Room owner. The APT4 tokenizer is gated on Hugging Face.
Suggested protocol
Commit a design listing each scoring script and the relations that apply to it. Each relation holds for a correct harness whatever the model, so a violation proves a defect. Positive controls the battery must flag: a bare AutoTokenizer load of an APT4 arm checkpoint directory, which deletes spaces and newlines at encoding (control-arm design, Amendment 1; embedding-initialisation erratum), and the MPS drift of the prefix-only scoring path recorded in the activation-patching report. Output: one signed gate file per scoring path; analysis scripts refuse to run without a passing gate file. The parity band of R4 is the numerics term of the variance-floor follow-up.

Selected exact hypotheses and premises

Premise · P175

87d352dc-dabc-4bcc-8c74-2bf51e2e548d

A measurement repeated after a tokenizer load that had deleted spaces and newlines was fixed.

Premise · P162

ea679d70-4f6a-49fb-ba27-b0083b779395

Two scoring paths of the same checkpoints compared at aggregate level only.

Premise · P251

17e3db6d-8386-4e44-a153-2d0d6edde538

A byte-identical repeat gate run on one model and one machine.

Reason for this revision

Initial proposal.