Experiment

Sign in with GitHub
← Experiments

Experiment · E9

At a fixed 32k vocabulary, does reserving a small science slice of the tokenizer's training text remove most of the English-science fertility tax at under 5% cost to Polish fertility?

Completed · Proposed by @stw2 · Assigned to @stw2

Five 32k SentencePiece BPE tokenizers in APT4's artifact shape that differ only in training-data allocation, separator pieces and whitespace pieces, measured on seven English and Polish corpora and two preamble texts against the Mistral-derived tokenizer and APT4.

Prerequisites and protocol

Access needs
Training text rebuilds from pinned dataset revisions; the atlas needs the gated reference tokenizers and four restricted corpora of the fertility atlas.
Suggested protocol
Scripts 01 to 07 in experiments/E09-vocabulary-allocation, in order.

Accepted plan

Plan accepted by @stw2

Five 32k SentencePiece BPE tokenizers differing in training-data allocation, audited for digit and separator handling and measured on a fertility atlas of seven corpora

retrospective

Responsibility and reported status

Taking starts no computation. Progress and completion are author reports; completion does not mean scientific success. Release does not prove a process stopped.

Room members can take or report on this experiment. Visit the Room to request membership.

Related findings

Execution attempts · 9

8 succeeded · 1 failed

Each attempt pins one public commit and configuration and delivers its own reported start and outcome. Reported times come from the registrant’s tool; received times are the server’s. An attempt without a delivered outcome stays unknown. Attempts verify no computation or result, and findings never require them.

  1. A83 Builds the LaTeX training text from the first proof-pile-2 arXiv train shard. Unlogged; the source flag is inferred from the manifest order and file times.

    Succeeded

    planned attempt · stw2/tokenizer-science-tax @ a77445db2440 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  2. A84 Builds the Polish, Python, mathematical web and general English training texts at pinned revisions, skipping the existing LaTeX text. The first 2,000 documents of the unshuffled Polish and English streams are skipped; this does not exclude E5's shuffled evaluation holdout, so one Polish holdout document is fully and 43 partly contained in pl.txt.

    Succeeded

    planned attempt · stw2/tokenizer-science-tax @ a77445db2440 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  3. A85 Trains the five 32k arms and converts each to a fast tokenizer; gates G1, G4 and G5 pass for every arm. Then retrains the Polish-only arm for the determinism gate G3, which fails: the retrained model file is not byte-identical, tokenizer.json is. The arm inputs of the two Polish-only arms are byte-identical to pl.txt and are not listed. The G3 comparison code of this invocation predates Amendment 2 and was not kept; the pinned commit holds the amended code.

    Failed

    planned attempt · stw2/tokenizer-science-tax @ f52d5afd8e13 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  4. A86 Re-runs step 02 with --g3 after Amendment 2: the five built arms are skipped and the existing retrain is reused; model content with the output path cleared and tokenizer.json are identical, raw model bytes are not. The run was not logged; its result is recorded only in the tokenizers manifest, whose original lists the unscrubbed model digests.

    Succeeded

    rerun attempt · stw2/tokenizer-science-tax @ 7a2cffbe78cf · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  5. A87 Audits digit and separator tokenization of the five arms, APT4 and the Mistral-derived tokenizer on the synthetic audit sets; gate G2 passes for every arm and the two reference rows reproduce the digit-policy audit. The script was last written nine seconds before this output; earlier invocations are unrecorded.

    Succeeded

    planned attempt · stw2/tokenizer-science-tax @ 7a2cffbe78cf · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  6. A88 Measures fertility of fourteen tokenizers (nine references and the five arms) on seven corpora and two preamble texts with a paired document bootstrap, and checks that every reference-tokenizer value equals the earlier atlas.

    Succeeded

    planned attempt · stw2/tokenizer-science-tax @ 7a2cffbe78cf · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  7. A89 Computes pooled excess taxes, the H1 ratios, H2 Polish costs, H3 criteria, the exploratory comparison, arm taxes relative to APT4 and science context windows, with bins and the verdict.

    Succeeded

    planned attempt · stw2/tokenizer-science-tax @ 7a2cffbe78cf · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  8. A90 Clears trainer_spec.input and trainer_spec.model_prefix, which held absolute paths, in the five arm copies of spm.model and updates their digests in the tokenizers manifest. Encoding is unchanged: the scrubbed models give the same pieces as the originals and as tokenizer.json on the gate's 10,000 lines.

    Succeeded

    planned attempt · stw2/tokenizer-science-tax @ 7a2cffbe78cf · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  9. A91 Writes flat headline values from the analysis, atlas, audit and tokenizer manifest, and re-checks the reference-tokenizer cells against the earlier atlas.

    Succeeded

    planned attempt · stw2/tokenizer-science-tax @ 7a2cffbe78cf · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

Attempts JSON →

Responsibility and plan history

    Contribute with your agent: “Find The tokenizer science tax, thread Fertility and vocabulary allocation. Help me prepare the hypotheses, an open experiment, a plan or a checkpoint I select. Show me the meaning for review before publishing.”

    Existing results can go straight to Publish a claim or finding. Hypotheses and experiments are optional.

    Research guide →