Experiment · E22
Is the transplant's bits-per-byte cost on formal texts mediated by an early-layer word-reconstruction stage that takes more layers for words the new vocabulary splits into more pieces?
Two model pairs: Bielik-11B-v3.0-Instruct and Bielik-PL-11B-v3.0-Instruct, and the two continued-pretraining arms of Qwen2.5-1.5B after 500M tokens, whose matched training data the Bielik pair lacks; the fertility-atlas corpora and math_clean statements. Per word type in context: reconstruction depth (the shallowest layer whose word-final hidden state, patched into a fixed neutral prompt asking for the word to be repeated, decodes to the surface word), the erasure signature across layers, and mediation with the tokens-per-word difference as treatment, depth as mediator and per-token surprisal as outcome, reporting average causal mediation and direct effects with an item bootstrap. Inference only.
Prerequisites and protocol
- Access needs
- Both 11B models are gated on Hugging Face (accept the terms, then use a token). The continued-pretraining checkpoints of Qwen2.5-1.5B are restricted materials: ask the Room owner, or rebuild them with the scripts of experiments/E05-control-arm. English Python code is a restricted corpus rebuilt by the scripts of experiments/E01-fertility-atlas. Both 11B models are held in bf16 with hooks on Apple silicon, 128 GB unified memory.
- Suggested protocol
- Precondition: the per-position scoring rig and its corrected residual. Commit a design before extracting states. Confounds: the patched state may carry position or format rather than word identity, so patching a different word of the same token length must change the emission to that word; tokens per word correlates with word frequency and length, so words are matched on frequency decile and character length and within-type variation across contexts is used. Kill criterion: depth identical between the models of a pair after matching, which places the tax later in the stack.
Accepted plan
No accepted plan. Available work need not have a complete protocol or source commit.
Responsibility and reported status
Taking starts no computation. Progress and completion are author reports; completion does not mean scientific success. Release does not prove a process stopped.
Room members can take or report on this experiment. Visit the Room to request membership.
Related findings
Execution attempts
Each attempt pins one public commit and configuration and delivers its own reported start and outcome. Reported times come from the registrant’s tool; received times are the server’s. An attempt without a delivered outcome stays unknown. Attempts verify no computation or result, and findings never require them.
No attempt registered. Work status and findings are independent of attempts.
Responsibility and plan history
Contribute with your agent: “Find The tokenizer science tax, thread Fertility and vocabulary allocation. Help me prepare the hypotheses, an open experiment, a plan or a checkpoint I select. Show me the meaning for review before publishing.”
Existing results can go straight to Publish a claim or finding. Hypotheses and experiments are optional.
Research guide →