Experiment

Sign in with GitHub
← Experiments

Experiment · E1

Is APT4's fertility tax against the Mistral-derived tokenizer higher on English scientific text than on the English preamble?

Completed · Proposed by @stw2 · Assigned to @stw2

The nine tokenizers of Table 1 on the Polish and English preamble and on seven corpora of about 120k words each: English arXiv abstracts, LaTeX method sections, Python code and GSM8K problems; Polish PES examination questions, Polish Wikipedia science articles and Polish reviews.

Prerequisites and protocol

Access needs
Gated tokenizer repositories need accepted terms and a token; four corpora rebuild from pinned dataset revisions.
Suggested protocol
Scripts 01 to 07 in experiments/E01-fertility-atlas, in order.

Accepted plan

Plan accepted by @stw2

Table 1 recomputation and a fertility atlas of nine tokenizers on seven corpora

retrospective

Responsibility and reported status

Taking starts no computation. Progress and completion are author reports; completion does not mean scientific success. Release does not prove a process stopped.

Room members can take or report on this experiment. Visit the Room to request membership.

Related findings

Execution attempts · 6

6 succeeded

Each attempt pins one public commit and configuration and delivers its own reported start and outcome. Reported times come from the registrant’s tool; received times are the server’s. An attempt without a delivered outcome stays unknown. Attempts verify no computation or result, and findings never require them.

  1. A1 Extracts the Polish and English preamble texts and their counting variants from the seed paper PDF.

    Succeeded

    planned attempt · stw2/tokenizer-science-tax @ 1855bbfabc83 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  2. A2 Fetches the nine Table 1 tokenizers and records their revisions and vocabulary sizes.

    Succeeded

    planned attempt · stw2/tokenizer-science-tax @ 1855bbfabc83 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  3. A3 Builds the seven corpora and freezes the arXiv API responses.

    Succeeded

    planned attempt · stw2/tokenizer-science-tax @ 1855bbfabc83 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  4. A4 Recomputes Table 1 under candidate counting protocols and selects the best-matching protocol.

    Succeeded

    planned attempt · stw2/tokenizer-science-tax @ 1855bbfabc83 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  5. A5 Measures fertility of the nine tokenizers on the seven corpora and the preamble with bootstrap intervals.

    Succeeded

    planned attempt · stw2/tokenizer-science-tax @ 1855bbfabc83 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  6. A6 Writes flat headline values from the Table 1 and atlas results.

    Succeeded

    planned attempt · stw2/tokenizer-science-tax @ 1855bbfabc83 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

Attempts JSON →

Responsibility and plan history

    Contribute with your agent: “Find The tokenizer science tax, thread Fertility and vocabulary allocation. Help me prepare the hypotheses, an open experiment, a plan or a checkpoint I select. Show me the meaning for review before publishing.”

    Existing results can go straight to Publish a claim or finding. Hypotheses and experiments are optional.

    Research guide →