Experiment

Sign in with GitHub
← Experiments

Experiment · E2

Does APT4's tokenization of digits and number separators differ from the Mistral-derived tokenizer's, and does the difference account for the lower GSM8K score of Bielik-PL-11B-v3.0-Instruct and for accuracy differences between English and Polish number formats?

Completed · Proposed by @stw2 · Assigned to @stw2

The nine tokenizers of Table 1 of arXiv:2604.10799v1 on synthetic number sets; Bielik-PL-11B-v3.0-Instruct and Bielik-11B-v3.0-Instruct on 2,500 answer-only arithmetic items in four number formats and two instruction languages, and on 250 GSM8K test problems.

Prerequisites and protocol

Access needs
The two 11B model repositories are gated on Hugging Face (accept the terms, then use a token); the downloaded snapshots and MLX conversions are restricted materials. Probes and GSM8K generation need Apple silicon with MLX and about 89 GB of disk; E1's tokenizers are rebuilt by E1's fetch script.
Suggested protocol
Scripts 01 to 09 in experiments/E02-digit-policy, in order.

Accepted plan

Plan accepted by @stw2

Digit and separator policy audit of nine tokenizers, answer-only arithmetic probes on the 11B Bielik pair in four number formats and two instruction languages, and a 250-problem GSM8K subsample

retrospective

Responsibility and reported status

Taking starts no computation. Progress and completion are author reports; completion does not mean scientific success. Release does not prove a process stopped.

Room members can take or report on this experiment. Visit the Room to request membership.

Related findings

Execution attempts · 10

10 succeeded

Each attempt pins one public commit and configuration and delivers its own reported start and outcome. Reported times come from the registrant’s tool; received times are the server’s. An attempt without a delivered outcome stays unknown. Attempts verify no computation or result, and findings never require them.

  1. A7 Downloads both 11B Instruct models at their revisions, hashes every file and checks tokenizer.json against E1's frozen copies.

    Succeeded

    planned attempt · stw2/tokenizer-science-tax @ c43f19850803 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  2. A8 Generates the audit number sets S1 to S8 and the GSM8K numeral inventory S9.

    Succeeded

    planned attempt · stw2/tokenizer-science-tax @ c43f19850803 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  3. A9 Classifies the digit policy of nine tokenizers, measures tokens per number per rendering and tabulates separator, sign and unit segmentation.

    Succeeded

    planned attempt · stw2/tokenizer-science-tax @ c43f19850803 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  4. A10 Generates 2,500 matched arithmetic items, each rendered into four number formats and two instruction languages.

    Succeeded

    planned attempt · stw2/tokenizer-science-tax @ c43f19850803 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  5. A11 Converts both downloaded models to MLX bf16 without quantisation and hashes the converted files.

    Succeeded

    planned attempt · stw2/tokenizer-science-tax @ c43f19850803 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  6. A12 Smoke gate on the first 50 item-cells per model: one beginning-of-sequence token, byte-identical greedy repeat, throughput projection.

    Succeeded

    planned attempt · stw2/tokenizer-science-tax @ c43f19850803 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  7. A13 Runs 20,000 greedy generations per model on the arithmetic items, PL model first, in one invocation.

    Succeeded

    planned attempt · stw2/tokenizer-science-tax @ c43f19850803 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  8. A14 Runs 250 GSM8K test problems with five fixed examples on both models and computes the paired accuracy difference.

    Succeeded

    planned attempt · stw2/tokenizer-science-tax @ c43f19850803 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  9. A15 Grades the probe generations and computes per-cell accuracy, the pre-registered contrasts and carry gradients.

    Succeeded

    planned attempt · stw2/tokenizer-science-tax @ c43f19850803 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  10. A16 Writes flat headline values from the audit, probe, smoke and GSM8K results.

    Succeeded

    planned attempt · stw2/tokenizer-science-tax @ c43f19850803 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

Attempts JSON →

Responsibility and plan history

    Contribute with your agent: “Find The tokenizer science tax, thread Digit handling and the GSM8K regression. Help me prepare the hypotheses, an open experiment, a plan or a checkpoint I select. Show me the meaning for review before publishing.”

    Existing results can go straight to Publish a claim or finding. Hypotheses and experiments are optional.

    Research guide →