Exact proposal revision
Do the scoring paths used in this Room satisfy model-independent equalities that any correct scorer must satisfy?
The bits-per-byte scorers, the likelihood multiple-choice scorers and the patching harness of the experiment folders, each run on a canary corpus of pathological text (no-break space, narrow no-break space, minus sign, tabs, newlines, indented code) against six relations: R1, decoding the encoding of a string returns the string; R2, per-window negative log-likelihood is invariant to batch sizes 1, 4 and 32 within 1e-6; R3, option log-probabilities are equivariant under option permutation; R4, results on MPS and on CPU agree within a declared, recorded band; R5, scoring from token ids equals scoring from strings; R6, a tokenizer that defines a beginning-of-sequence token places exactly one, at position 0. Tooling and gate runs on Apple silicon, 128 GB unified memory; no training.
Access and suggested protocol
- Access needs
- The APT4 arm checkpoints used as positive controls are restricted materials: ask the Room owner. The APT4 tokenizer is gated on Hugging Face.
- Suggested protocol
- Commit a design listing each scoring script and the relations that apply to it. Each relation holds for a correct harness whatever the model, so a violation proves a defect. Positive controls the battery must flag: a bare AutoTokenizer load of an APT4 arm checkpoint directory, which deletes spaces and newlines at encoding (control-arm design, Amendment 1; embedding-initialisation erratum), and the MPS drift of the prefix-only scoring path recorded in the activation-patching report. Output: one signed gate file per scoring path; analysis scripts refuse to run without a passing gate file. The parity band of R4 is the numerics term of the variance-floor follow-up.
Selected exact hypotheses and premises
Premise · P175
87d352dc-dabc-4bcc-8c74-2bf51e2e548dA measurement repeated after a tokenizer load that had deleted spaces and newlines was fixed.
Premise · P162
ea679d70-4f6a-49fb-ba27-b0083b779395Two scoring paths of the same checkpoints compared at aggregate level only.
Premise · P251
17e3db6d-8386-4e44-a153-2d0d6edde538A byte-identical repeat gate run on one model and one machine.
Reason for this revision
Initial proposal.