Experiment

Sign in with GitHub
← Experiments

Experiment · E12

When Qwen2.5-1.5B answers a translation-paired multiple-choice item correctly in English and incorrectly in Polish, is the knowledge present in its forward pass but not reached from the Polish prompt, or absent from what the Polish prompt can reach?

Completed · Proposed by @stw2 · Assigned to @stw2

Qwen2.5-1.5B and two continued-pretraining arms at 500M tokens on 600 Belebele and 600 MMLU translation pairs: behavioural controls (scaffold language swap, translation assist), then residual-stream patching from English into Polish twins on the forward-discordant pairs with irrelevant-source, breakage and anchor controls, the reverse direction, the arms and a STEM slice. Inference only.

Prerequisites and protocol

Access needs
The two continued-pretraining checkpoints are restricted materials: ask the Room owner. The APT4 tokenizer reference used for the APT4 arm is gated on Hugging Face.
Suggested protocol
Scripts 01 to 07 in experiments/E12-activation-patching, in the order of its README.

Accepted plan

Plan accepted by @stw2

Cross-lingual residual-stream patching from English into Polish twins of translation-paired multiple-choice items on Qwen2.5-1.5B, with behavioural controls and two continued-pretraining arms

prospective

Responsibility and reported status

Taking starts no computation. Progress and completion are author reports; completion does not mean scientific success. Release does not prove a process stopped.

Room members can take or report on this experiment. Visit the Room to request membership.

Related findings

Execution attempts · 20

20 succeeded

Each attempt pins one public commit and configuration and delivers its own reported start and outcome. Reported times come from the registrant’s tool; received times are the server’s. An attempt without a delivered outcome stays unknown. Attempts verify no computation or result, and findings never require them.

  1. A118 Derives the forward and reverse discordant translation pairs from the committed base-model scores, gated to the expected counts, and freezes the scan subsample, derangement, breakage guard and entity spans.

    Succeeded

    planned attempt · stw2/tokenizer-science-tax @ 7253735a8d11 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  2. A119 Re-scores the English and Polish baseline cells of Qwen2.5-1.5B and checks them against the committed per-item scores.

    Succeeded

    planned attempt · stw2/tokenizer-science-tax @ 7253735a8d11 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  3. A120 Re-scores the English and Polish baseline cells of Qwen2.5-1.5B after 500M tokens of continued pretraining with its own tokenizer and checks them against the committed per-item scores.

    Succeeded

    planned attempt · stw2/tokenizer-science-tax @ 7253735a8d11 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  4. A121 Re-scores the English and Polish baseline cells of Qwen2.5-1.5B with APT4 by Fast Vocabulary Transfer after 500M tokens of continued pretraining and checks them against the committed per-item scores.

    Succeeded

    planned attempt · stw2/tokenizer-science-tax @ 7253735a8d11 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  5. A122 Runs the patch harness smoke gates on Qwen2.5-1.5B: self-identity, byte-identical repeat, and calibration of every benchmark and anchor at source layer 23 into target layer 13.

    Succeeded

    planned attempt · stw2/tokenizer-science-tax @ 7253735a8d11 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  6. A123 Scores Qwen2.5-1.5B on both translation-pair sets with the answer scaffold in the other language.

    Succeeded

    planned attempt · stw2/tokenizer-science-tax @ 7253735a8d11 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  7. A124 Scores Qwen2.5-1.5B after 500M tokens of continued pretraining with its own tokenizer on both translation-pair sets with the answer scaffold in the other language.

    Succeeded

    planned attempt · stw2/tokenizer-science-tax @ 7253735a8d11 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  8. A125 Scores Qwen2.5-1.5B with APT4 by Fast Vocabulary Transfer after 500M tokens of continued pretraining on both translation-pair sets with the answer scaffold in the other language.

    Succeeded

    planned attempt · stw2/tokenizer-science-tax @ 7253735a8d11 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  9. A126 Scores Qwen2.5-1.5B on both translation-pair sets with the English twin's prompt prepended to each Polish prompt.

    Succeeded

    planned attempt · stw2/tokenizer-science-tax @ 7253735a8d11 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  10. A127 Scores Qwen2.5-1.5B after 500M tokens of continued pretraining with its own tokenizer on both translation-pair sets with the English twin's prompt prepended to each Polish prompt.

    Succeeded

    planned attempt · stw2/tokenizer-science-tax @ 7253735a8d11 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  11. A128 Scores Qwen2.5-1.5B with APT4 by Fast Vocabulary Transfer after 500M tokens of continued pretraining on both translation-pair sets with the English twin's prompt prepended to each Polish prompt.

    Succeeded

    planned attempt · stw2/tokenizer-science-tax @ 7253735a8d11 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  12. A129 Scans the scaffold-final residual patch over 196 odd-layer source and target pairs on the 100-item subsample of Qwen2.5-1.5B, with specific and irrelevant sources.

    Succeeded

    planned attempt · stw2/tokenizer-science-tax @ 633b3ba1ce91 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  13. A130 Selects confirmation layer pairs from the scan and patches all forward-discordant pairs of Qwen2.5-1.5B at them, with the irrelevant-source control, the anchor ladder at two pairs and the breakage guard.

    Succeeded

    planned attempt · stw2/tokenizer-science-tax @ 633b3ba1ce91 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  14. A131 Patches the reverse direction, Polish twin into the English pass, on the 82 reverse-discordant pairs of Qwen2.5-1.5B at the confirmation layer pairs.

    Succeeded

    planned attempt · stw2/tokenizer-science-tax @ 633b3ba1ce91 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  15. A132 Patches Qwen2.5-1.5B after 500M tokens of continued pretraining with its own tokenizer at the confirmation layer pairs on the base model's forward-discordant pairs, with the irrelevant-source control.

    Succeeded

    planned attempt · stw2/tokenizer-science-tax @ 633b3ba1ce91 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  16. A133 Patches Qwen2.5-1.5B with APT4 by Fast Vocabulary Transfer after 500M tokens of continued pretraining at the confirmation layer pairs on the base model's forward-discordant pairs, with the irrelevant-source control.

    Succeeded

    planned attempt · stw2/tokenizer-science-tax @ 633b3ba1ce91 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  17. A134 Computes every pre-registered endpoint and the transport rates, with item bootstrap intervals and exact McNemar tests.

    Succeeded

    planned attempt · stw2/tokenizer-science-tax @ 633b3ba1ce91 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  18. A135 Reruns the set derivation on the committed inputs; the output is byte-identical to the committed sets.json.

    Succeeded

    rerun attempt · stw2/tokenizer-science-tax @ 7253735a8d11 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  19. A136 Reruns the analysis on the committed rows; the output is byte-identical to the committed analysis.json.

    Succeeded

    rerun attempt · stw2/tokenizer-science-tax @ 633b3ba1ce91 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  20. A137 Writes flat headline values from the analysis and the patch sets.

    Succeeded

    exploratory attempt · stw2/tokenizer-science-tax @ 633b3ba1ce91 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

Attempts JSON →

Responsibility and plan history

    Contribute with your agent: “Find The tokenizer science tax, thread Cross-language knowledge access. Help me prepare the hypotheses, an open experiment, a plan or a checkpoint I select. Show me the meaning for review before publishing.”

    Existing results can go straight to Publish a claim or finding. Hypotheses and experiments are optional.

    Research guide →