Experiment

Sign in with GitHub
← Experiments

Experiment · E3

Does forcing English chain-of-thought on Polish STEM questions raise the accuracy of Bielik-PL-11B-v3.0-Instruct more than that of Bielik-11B-v3.0-Instruct?

Completed · Proposed by @stw2 · Assigned to @stw2

800 questions posed in Polish (250 GSM8K-PL problems, 300 LLMzSzŁ STEM examination questions, 250 PES examination questions), each answered by both models under Polish and English chain-of-thought with greedy decoding.

Prerequisites and protocol

Access needs
Both models are gated on Hugging Face; the LLMzSzŁ and PES item sets, the exemplars and the complete generations are restricted materials, rebuilt from pinned dataset revisions; generation needs Apple silicon as written.
Suggested protocol
Scripts 01 to 08 in experiments/E03-reasoning-language, in order.

Accepted plan

Plan accepted by @stw2

Reasoning-language forcing: two Bielik 11B models under Polish and English chain-of-thought on 800 Polish STEM questions

retrospective

Responsibility and reported status

Taking starts no computation. Progress and completion are author reports; completion does not mean scientific success. Release does not prove a process stopped.

Room members can take or report on this experiment. Visit the Room to request membership.

Related findings

Execution attempts · 8

8 succeeded

Each attempt pins one public commit and configuration and delivers its own reported start and outcome. Reported times come from the registrant’s tool; received times are the server’s. An attempt without a delivered outcome stays unknown. Attempts verify no computation or result, and findings never require them.

  1. A17 Builds the GSM8K-PL, LLMzSzŁ STEM and PES item sets and the few-shot exemplars at pinned dataset revisions, after the gold-answer parity check.

    Succeeded

    planned attempt · stw2/tokenizer-science-tax @ c10ec1755e97 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  2. A18 Generates answers to all 800 items under Polish and English chain-of-thought with both models, Bielik-PL-11B-v3.0-Instruct first.

    Succeeded

    planned attempt · stw2/tokenizer-science-tax @ 02380f96e12e · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  3. A19 Regenerates, for both models, every answer that stopped at its token cap with a 1280-token cap and rewrites the generation files.

    Succeeded

    planned attempt · stw2/tokenizer-science-tax @ d3c436d89b85 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  4. A20 Grades every answer after a self-test gate and classifies the language of its reasoning.

    Succeeded

    planned attempt · stw2/tokenizer-science-tax @ d3c436d89b85 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  5. A21 Computes accuracies, English chain-of-thought gains, the difference in gains with bootstrap intervals, sensitivity analyses and generated-token counts.

    Succeeded

    planned attempt · stw2/tokenizer-science-tax @ d3c436d89b85 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  6. A22 Counts, per model, benchmark and condition, the answers regenerated at the higher cap, those that reached it again and their graded outcomes.

    Succeeded

    planned attempt · stw2/tokenizer-science-tax @ d3c436d89b85 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  7. A23 Copies the header and GSM8K-PL records of the complete generations, byte for byte, to the redistributable files.

    Succeeded

    planned attempt · stw2/tokenizer-science-tax @ d3c436d89b85 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  8. A24 Writes flat headline values from the analysis, the token-cap audit and the item manifest.

    Succeeded

    planned attempt · stw2/tokenizer-science-tax @ d3c436d89b85 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

Attempts JSON →

Responsibility and plan history

    Contribute with your agent: “Find The tokenizer science tax, thread Reasoning language, latent pivot and context. Help me prepare the hypotheses, an open experiment, a plan or a checkpoint I select. Show me the meaning for review before publishing.”

    Existing results can go straight to Publish a claim or finding. Hypotheses and experiments are optional.

    Research guide →