Experiment

Sign in with GitHub
← Experiments

Experiment · E10

Does the end-to-end success gap between Bielik-11B-v3.0-Instruct and Bielik-PL-11B-v3.0-Instruct on multi-step tool-use tasks exceed the gap predicted from their one-shot per-step success rates?

Completed · Proposed by @stw2 · Assigned to @stw2

Both 11B models under one frozen ReAct tool-loop protocol on 100 synthetic task instances (CSV analysis, reproduction of a computation from a methods paragraph, write-and-debug) with English and Polish instructions, plus one-shot per-step probes; MLX bf16, greedy decoding.

Prerequisites and protocol

Access needs
Both model repositories are gated; the MLX conversions are restricted materials rebuilt with experiments/E02-digit-policy/scripts/02_convert_mlx.py at the pinned revisions.
Suggested protocol
Scripts 01 to 07 in experiments/E10-agentic-compounding: 01 builds the instances, scripts/run_all.sh runs the rest.

Accepted plan

Plan accepted by @stw2

Paired tool-use trajectories and one-shot per-step probes of Bielik-PL-11B-v3.0-Instruct and Bielik-11B-v3.0-Instruct, compared with a step-composition prediction

prospective

Responsibility and reported status

Taking starts no computation. Progress and completion are author reports; completion does not mean scientific success. Release does not prove a process stopped.

Room members can take or report on this experiment. Visit the Room to request membership.

Related findings

Execution attempts · 17

17 succeeded

Each attempt pins one public commit and configuration and delivers its own reported start and outcome. Reported times come from the registrant’s tool; received times are the server’s. An attempt without a delivered outcome stays unknown. Attempts verify no computation or result, and findings never require them.

  1. A92 Generates the 100 task instances from the 20 templates and freezes the manifest.

    Succeeded

    planned attempt · stw2/tokenizer-science-tax @ 03e30b6cecb9 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  2. A93 Runs all 100 instances with English and Polish instructions through the tool loop with Bielik-PL-11B-v3.0-Instruct.

    Succeeded

    planned attempt · stw2/tokenizer-science-tax @ 03e30b6cecb9 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  3. A94 Runs all 100 instances with English and Polish instructions through the tool loop with Bielik-11B-v3.0-Instruct.

    Succeeded

    planned attempt · stw2/tokenizer-science-tax @ 03e30b6cecb9 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  4. A95 Poses each instance once, without the tool loop, to Bielik-PL-11B-v3.0-Instruct as a per-step probe in both languages.

    Succeeded

    planned attempt · stw2/tokenizer-science-tax @ 03e30b6cecb9 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  5. A96 Poses each instance once, without the tool loop, to Bielik-11B-v3.0-Instruct as a per-step probe in both languages, and aggregates both models' probe success rates.

    Succeeded

    planned attempt · stw2/tokenizer-science-tax @ 03e30b6cecb9 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  6. A97 Grades run 1's trajectories with the locked grader, which reads only a written solution.py for write-and-debug tasks.

    Succeeded

    planned attempt · stw2/tokenizer-science-tax @ 03e30b6cecb9 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  7. A98 Computes run 1's endpoints from the first grading.

    Succeeded

    planned attempt · stw2/tokenizer-science-tax @ 03e30b6cecb9 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  8. A99 Grades every trajectory (numeric tolerance or hidden tests) and summarises accuracy, steps, tokens and non-finalization by model, language and family.

    Succeeded

    rerun attempt · stw2/tokenizer-science-tax @ df809d2de201 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  9. A100 Computes the end-to-end gaps with paired bootstrap intervals, the language interaction, the English token ratio and non-finalization rates, the step-composition prediction and Holm-adjusted p.

    Succeeded

    rerun attempt · stw2/tokenizer-science-tax @ df809d2de201 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  10. A101 Writes flat headline values from the analysis, the graded rows and the public trajectories.

    Succeeded

    planned attempt · stw2/tokenizer-science-tax @ df809d2de201 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  11. A102 Runs all 100 instances with English and Polish instructions through the tool loop with Bielik-PL-11B-v3.0-Instruct.

    Succeeded

    rerun attempt · stw2/tokenizer-science-tax @ d35f2683bfcb · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  12. A103 Runs all 100 instances with English and Polish instructions through the tool loop with Bielik-11B-v3.0-Instruct.

    Succeeded

    rerun attempt · stw2/tokenizer-science-tax @ d35f2683bfcb · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  13. A104 Poses each instance once, without the tool loop, to Bielik-PL-11B-v3.0-Instruct as a per-step probe in both languages.

    Succeeded

    rerun attempt · stw2/tokenizer-science-tax @ d35f2683bfcb · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  14. A105 Poses each instance once, without the tool loop, to Bielik-11B-v3.0-Instruct as a per-step probe in both languages, and aggregates both models' probe success rates.

    Succeeded

    rerun attempt · stw2/tokenizer-science-tax @ d35f2683bfcb · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  15. A106 Grades every trajectory (numeric tolerance or hidden tests) and summarises accuracy, steps, tokens and non-finalization by model, language and family.

    Succeeded

    rerun attempt · stw2/tokenizer-science-tax @ d35f2683bfcb · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  16. A107 Computes the end-to-end gaps with paired bootstrap intervals, the language interaction, the English token ratio and non-finalization rates, the step-composition prediction and Holm-adjusted p.

    Succeeded

    rerun attempt · stw2/tokenizer-science-tax @ d35f2683bfcb · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  17. A108 Writes flat headline values from the analysis, the graded rows and the public trajectories.

    Succeeded

    rerun attempt · stw2/tokenizer-science-tax @ d35f2683bfcb · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

Attempts JSON →

Responsibility and plan history

    Contribute with your agent: “Find The tokenizer science tax, thread Agentic compounding. Help me prepare the hypotheses, an open experiment, a plan or a checkpoint I select. Show me the meaning for review before publishing.”

    Existing results can go straight to Publish a claim or finding. Hypotheses and experiments are optional.

    Research guide →