Recovery pretraining of APT4 FVT transplants with 0%, 10% and 30% math and code for 1B tokens at 0.5B, and a 0% against 30% confirmation pair at 1.5B

https://github.com/stw2/tokenizer-science-tax @ 16a182f7c3592e9e2e84c5a9bc824d2e9b263a06

Reference checked 2026-09-14 13:12 UTC. No code was executed or scientific result verified.

• experiments/E06-math-code-rescue/DESIGN.md
• experiments/E06-math-code-rescue/configs
• experiments/E06-math-code-rescue/scripts

Prediction

At 0.5B as in the locked plan; at 1.5B the same bins apply to conf30 against conf00 with Qwen2.5-1.5B as the base model.

Protocol

Per arm on its own GPU, scripts/00_bootstrap_cuda.sh: fetch and check inputs; build the arm's pools (01, gate G6) and pack them by APT4-token shares (02, gate G2); build the FVT transplant twice and compare (E05's 03, gate G3); probe smoke (08, gate G4); resume smoke (E05's 04, gate G5); train 1B tokens (E05's 04) while scoring the base model, the transplant and every 100M-token checkpoint for bits per byte on ten texts (05) and the digit probe (08); per-document scores of the final checkpoint (05b). Per-document scores of the base model (05b); base-model bits per byte on a second backend (G1, 05); analysis (06). Confirmation: the same per-arm pipeline with configs/e6_conf15.json for arms conf00 and conf30 on Qwen2.5-1.5B, per-document scores of Qwen2.5-1.5B, and the same estimators and bins.

Dataset

Training: packed streams built from FineWeb2-HQ pol_Latn (revision c0c06e94, inferred), SlimPajama-6B (b5f90f41, inferred), OpenWebMath (open-web-math/open-web-math, no revision recorded or inferable) and codeparrot-clean-train (3e6ab65f, inferred), shuffled with seed 42; the first 2,000 filtered Polish and English documents are skipped as evaluation holdouts; the packed streams, identified by digest, are the training inputs. Evaluation: the first 200 documents of ten texts (Polish and English web holdouts, English arXiv abstracts, LaTeX method sections, Python code and GSM8K problems, Polish Wikipedia science articles, PES questions and reviews, math_clean) and a 900-completion synthetic digit probe.

Split

No train and test split of one dataset: evaluation texts are separate files, and the Polish and English holdouts are skipped in the training streams (gate G6).

Access needs

The APT4 tokenizer repository is gated (accept the terms, then use a token); the English evaluation holdout and four evaluation corpora are restricted materials: ask the Room owner, or rebuild them.

Configurations

mix00: Polish 80%, English 20%, OpenWebMath 0%, Python code 0%; mix10: Polish 72%, English 18%, OpenWebMath 5%, Python code 5%; mix30: Polish 56%, English 14%, OpenWebMath 15%, Python code 15%; conf00: Polish 80%, English 20%, OpenWebMath 0%, Python code 0%; conf30: Polish 56%, English 14%, OpenWebMath 15%, Python code 15% (shares of APT4 tokens). 0.5B recipe: micro-batch 32 x gradient accumulation 8; 1.5B recipe: micro-batch 8 x gradient accumulation 32; both sequence 2048, learning rate 1e-4, 50 warmup steps, no decay, checkpoints every 100M tokens.

Metric

Bits per byte on each text; digit-probe accuracy; rescue fraction R = 1 - (30% arm's gap) / (0% arm's gap) to the base model; pooled Polish bits per byte difference.

Seeds

Stream shuffle seed 42; bootstrap seed 20260707; digit-probe item selection seed 20260706 (frozen specification); one training run per arm.

Interpretation rule

Per endpoint of H1 (math_clean bits per byte with a paired document bootstrap; digit-probe accuracy with a paired item bootstrap; 2,000 resamples): CONFIRMED if the rescue fraction R is at least 0.8 and its 95% interval's lower bound exceeds 0.5; REFUTED if R is at most 0.5 and the interval's upper bound is below 0.8; VACUOUS if the interval of the 0% arm's gap contains 0; PARTIAL otherwise. H1 is CONFIRMED or REFUTED only when both endpoints agree, else PARTIAL; the design also names REFUTED if the 30% arm is no better than the 0% arm, and a Holm adjustment of H1 and H2, neither computed. H2: CONFIRMED if the 95% interval of the pooled Polish difference (30% minus 0%) is at most 0.05 bits per byte, REFUTED if its lower bound exceeds 0.05, PARTIAL otherwise. H3 is reported without bins. Gates G1 to G6 must pass before results count. Amendment 1: the same rules apply at 1.5B with Qwen2.5-1.5B as the base model.

Resources

One 4090-class GPU, 24 GB, per arm; about 8.5 hours per 0.5B arm and an estimated 31 to 46 hours per 1.5B arm; analysis on a CPU.

Prior work

arXiv:2604.10799v1 reports GSM8K 85.60 for Bielik-11B-v3.0-Instruct and 80.97 for its APT4 transplant, without the math and code share of its 20B-token adaptation data.