Experiment

Sign in with GitHub
← Experiments

Experiment · E6

Does a math and code share in recovery pretraining of an APT4 transplant remove its regression on arithmetic text and digit arithmetic at a fixed token budget, and what does it cost on Polish text?

Completed · Proposed by @stw2 · Assigned to @stw2

APT4 FVT transplants of Qwen2.5-0.5B recovery-pretrained on 1B APT4 tokens with 0%, 10% or 30% math and code, the rest Polish and English web text at 4 to 1; evaluation by bits per byte on ten texts and by a synthetic digit probe.

Prerequisites and protocol

Access needs
One 4090-class GPU, 24 GB, per arm; the APT4 tokenizer repository is gated; one evaluation holdout, four evaluation corpora and the packed training streams are restricted materials.
Suggested protocol
In experiments/E06-math-code-rescue: scripts/00_bootstrap_cuda.sh once per arm on its own GPU, then scripts/06_analyze.py.

Accepted plan

Plan accepted by @stw2

Recovery pretraining of APT4 FVT transplants with math and code at 0.5B and a 1.5B confirmation pair, with the confirmation analysis as run

retrospective

Responsibility and reported status

Taking starts no computation. Progress and completion are author reports; completion does not mean scientific success. Release does not prove a process stopped.

Room members can take or report on this experiment. Visit the Room to request membership.

Related findings

Execution attempts · 11

11 succeeded

Each attempt pins one public commit and configuration and delivers its own reported start and outcome. Reported times come from the registrant’s tool; received times are the server’s. An attempt without a delivered outcome stays unknown. Attempts verify no computation or result, and findings never require them.

  1. A53 mix10: builds and packs the training stream, builds the FVT transplant of Qwen2.5-0.5B, runs the gates, trains 1B tokens with checkpoint scoring, and scores the final checkpoint per document.

    Succeeded

    planned attempt · stw2/tokenizer-science-tax @ 1e3dd3f73685 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  2. A54 mix30: builds and packs the training stream, builds the FVT transplant of Qwen2.5-0.5B, runs the gates, trains 1B tokens with checkpoint scoring, and scores the final checkpoint per document.

    Succeeded

    planned attempt · stw2/tokenizer-science-tax @ 1e3dd3f73685 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  3. A55 Scores Qwen2.5-0.5B per document on the ten evaluation texts.

    Succeeded

    planned attempt · stw2/tokenizer-science-tax @ 1e3dd3f73685 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  4. A56 Scores Qwen2.5-0.5B on the ten evaluation texts with PyTorch on Apple silicon, for comparison with the CUDA rows (gate G1).

    Succeeded

    planned attempt · stw2/tokenizer-science-tax @ 1e3dd3f73685 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  5. A57 Computes H1 rescue fractions with paired bootstraps, the H2 pooled Polish cost, the H3 recovery tokens and the consistency check at 0.5B.

    Succeeded

    planned attempt · stw2/tokenizer-science-tax @ 1e3dd3f73685 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  6. A58 conf00: builds and packs the training stream, builds the FVT transplant of Qwen2.5-1.5B, runs the gates, trains 1B tokens with checkpoint scoring, and scores the final checkpoint per document.

    Succeeded

    planned attempt · stw2/tokenizer-science-tax @ 16a182f7c359 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  7. A59 conf30: builds and packs the training stream, builds the FVT transplant of Qwen2.5-1.5B, runs the gates, trains 1B tokens with checkpoint scoring, and scores the final checkpoint per document.

    Succeeded

    planned attempt · stw2/tokenizer-science-tax @ 16a182f7c359 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  8. A60 Scores Qwen2.5-1.5B per document on the ten evaluation texts.

    Succeeded

    planned attempt · stw2/tokenizer-science-tax @ 16a182f7c359 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  9. A61 Computes the 1.5B rescue fractions with paired bootstraps, the pooled Polish cost with an unpaired bootstrap, and the scale contrast with the 0.5B analysis.

    Succeeded

    planned attempt · stw2/tokenizer-science-tax @ 15faa7a8d8b3 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  10. A62 Writes flat headline values from the analyses, the score rows and the data manifests.

    Succeeded

    planned attempt · stw2/tokenizer-science-tax @ 2457520fc798 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

Attempts JSON →

Responsibility and plan history

    Contribute with your agent: “Find The tokenizer science tax, thread Digit handling and the GSM8K regression. Help me prepare the hypotheses, an open experiment, a plan or a checkpoint I select. Show me the meaning for review before publishing.”

    Existing results can go straight to Publish a claim or finding. Hypotheses and experiments are optional.

    Research guide →