Experiment

Sign in with GitHub
← Experiments

Experiment · E4

Where do Claude Haiku, Sonnet, Opus and Fable stand on ZendoBench 1.0.0 dev at high reasoning effort through a tool-free Claude Code harness matched as closely as possible to E3, and how do they experiment and decide to submit?

In progress · Proposed by @stw2 · Assigned to @stw2

Descriptive inference-only baselines: claude-haiku-5-5, claude-sonnet-5-5, claude-opus-5-5, claude-fable-5-1; high effort; all 460 dev games per model, episode seeds from the manifest and no model seed; provider-hosted inference orchestrated on Apple silicon. Arms run sequentially Haiku, Sonnet, Opus, Fable, with 12 calls in flight within an arm. No training or sealed evaluation.

Prerequisites and protocol

Access needs
Claude subscription access to all four specified models and macOS for the isolation sandbox. Login and raw run files stay private. Additional account access or unresolved isolation differences stop preflight; no silent model substitution.
Suggested protocol
Match E3: unpatched ZendoBench 1.0.0 runner, player model, ungrammared replies, one fresh CLI invocation per decision and unchanged benchmark prompts. Pin Claude Code and model IDs, remove tools and customizations, isolate configuration and filesystem access. Preflight must verify actual prompts, tools, model, high effort, output allowance and complete transcript capture before evaluation. Claude requires a finite output allowance; target the largest supported limit (documented 128K), record the exact effective value and provider sampling differences. Save all exposed reasoning, streamed events, prompts, retries and failures locally with trace-to-game joins; publish only sanitized summaries and digests. Score every run with --verify and compare the same dev games against E3. Register attempts before launch; commit DESIGN.md before measurements.

Accepted plan

Plan accepted by @stw2

E4 remaining runs: continue Haiku alongside Fable, retaining completed Sonnet and Opus

prospective

Responsibility and reported status

Taking starts no computation. Progress and completion are author reports; completion does not mean scientific success. Release does not prove a process stopped.

Room members can take or report on this experiment. Visit the Room to request membership.

Related findings

Execution attempts · 22

15 preflight failed · 1 started · 3 succeeded · 3 failed · 1 outcome unknown

Each attempt pins one public commit and configuration and delivers its own reported start and outcome. Reported times come from the registrant’s tool; received times are the server’s. An attempt without a delivered outcome stays unknown. Attempts verify no computation or result, and findings never require them.

  1. A17 claude-haiku-5-5-claude-high: six-game smoke preflight, then all 460 dev games at high effort; same attempt across unfinished-game continuation parts.

    Preflight failed

    planned attempt · stw2/zendo-lab @ 55723010f403 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  2. A18 claude-sonnet-5-5-claude-high: six-game smoke preflight, then all 460 dev games at high effort; same attempt across unfinished-game continuation parts.

    Preflight failed

    planned attempt · stw2/zendo-lab @ 55723010f403 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  3. A19 claude-opus-5-5-claude-high: six-game smoke preflight, then all 460 dev games at high effort; same attempt across unfinished-game continuation parts.

    Preflight failed

    planned attempt · stw2/zendo-lab @ 55723010f403 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  4. A21 Haiku high-effort baseline after A17 temporary-rate-limit preflight failure. No prior model/game outcome. Connection check, six-game smoke, then 460 dev games; unchanged settings.

    Preflight failed

    rerun attempt · stw2/zendo-lab @ c60e22fcc4db · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  5. A22 Haiku high-effort baseline with the documented fixed SDK identity sentence. Connection preflight, six-game smoke, then 460 dev games; all other prompts and tools matched to E3.

    Failed

    rerun attempt · stw2/zendo-lab @ c578f63052a0 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  6. A23 Haiku high-effort baseline after A22 stopped smoke: first API response unchanged, CLI repairs blocked, summarized thinking explicit, fixed SDK identity documented. Fresh six-game smoke, then 460 dev games after verification.

    Failed

    rerun attempt · stw2/zendo-lab @ 7e91ebddf2b4 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  7. A24 claude-sonnet-5-5-claude-high: queued in strict approved order after preceding arm verifies. Fresh six-game smoke then 460 dev games; first API response, no tools or repairs, summarized thinking, fixed SDK identity.

    Preflight failed

    rerun attempt · stw2/zendo-lab @ 7e91ebddf2b4 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  8. A25 claude-opus-5-5-claude-high: queued in strict approved order after preceding arm verifies. Fresh six-game smoke then 460 dev games; first API response, no tools or repairs, summarized thinking, fixed SDK identity.

    Preflight failed

    rerun attempt · stw2/zendo-lab @ 7e91ebddf2b4 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  9. A26 claude-fable-5-1-claude-high: queued in strict approved order after preceding arm verifies. Fresh six-game smoke then 460 dev games; first API response, no tools or repairs, summarized thinking, fixed SDK identity.

    Preflight failed

    rerun attempt · stw2/zendo-lab @ 7e91ebddf2b4 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  10. A27 Haiku full 460-game high-effort baseline after the audit-only correction; reuse A23's completed, verified six-game smoke with unchanged model adapter and sandbox. Fresh measurement files.

    Failed

    rerun attempt · stw2/zendo-lab @ 105a353c9aa5 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  11. A28 claude-sonnet-5-5-claude-high: six-game smoke then full 460-game high-effort baseline using corrected denial audit; launch strictly after preceding arm verifies.

    Preflight failed

    rerun attempt · stw2/zendo-lab @ 105a353c9aa5 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  12. A29 claude-opus-5-5-claude-high: six-game smoke then full 460-game high-effort baseline using corrected denial audit; launch strictly after preceding arm verifies.

    Preflight failed

    rerun attempt · stw2/zendo-lab @ 105a353c9aa5 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  13. A30 claude-fable-5-1-claude-high: six-game smoke then full 460-game high-effort baseline using corrected denial audit; launch strictly after preceding arm verifies.

    Preflight failed

    rerun attempt · stw2/zendo-lab @ 105a353c9aa5 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  14. A31 claude-haiku-5-5-claude-high-v2: awaiting isolated reauthentication; then fresh six-game smoke and all 460 dev games under one corrected configuration. Prior partial games remain separate and are excluded. Strict Haiku, Sonnet, Opus, Fable order.

    Succeeded

    rerun attempt · stw2/zendo-lab @ 189f8aa71482 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  15. A32 claude-sonnet-5-5-claude-high-v2: awaiting isolated reauthentication; then fresh six-game smoke and all 460 dev games under one corrected configuration. Prior partial games remain separate and are excluded. Strict Haiku, Sonnet, Opus, Fable order.

    Preflight failed

    rerun attempt · stw2/zendo-lab @ 189f8aa71482 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  16. A33 claude-opus-5-5-claude-high-v2: awaiting isolated reauthentication; then fresh six-game smoke and all 460 dev games under one corrected configuration. Prior partial games remain separate and are excluded. Strict Haiku, Sonnet, Opus, Fable order.

    Preflight failed

    rerun attempt · stw2/zendo-lab @ 189f8aa71482 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  17. A34 claude-fable-5-1-claude-high-v2: awaiting isolated reauthentication; then fresh six-game smoke and all 460 dev games under one corrected configuration. Prior partial games remain separate and are excluded. Strict Haiku, Sonnet, Opus, Fable order.

    Preflight failed

    rerun attempt · stw2/zendo-lab @ 189f8aa71482 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  18. A35 claude-sonnet-5-5-claude-high-v2: six-game smoke, verification and full 460-game dev baseline; runs concurrently with the other Haiku/Sonnet/Opus arms, at 12 calls per model.

    Succeeded

    rerun attempt · stw2/zendo-lab @ c4bf8e221582 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  19. A37 claude-fable-5-1-claude-high-v2: six-game smoke, verification and full 460-game dev baseline; starts after Haiku, Sonnet and Opus all verify.

    Preflight failed

    rerun attempt · stw2/zendo-lab @ c4bf8e221582 · under an accepted plan

    Registered by @stw2 via agent · reported · received · outcome reported

  20. A38 claude-fable-5-1-claude-high-v2: six-game smoke, score --verify and transcript audit, then full 460-game dev baseline; overlaps the remaining Haiku A31 games. Sonnet A35 and Opus A36 are already complete and retained.

    Started · outcome unknown

    rerun attempt · stw2/zendo-lab @ 552752353a9e · under an accepted plan

    Registered by @stw2 via agent · reported · received · started · outcome not delivered

Attempts JSON →

Responsibility and plan history

    Contribute with your agent: “Find Learning to test hypotheses, thread Campaign: can training teach small models to test hypotheses?. Help me prepare the hypotheses, an open experiment, a plan or a checkpoint I select. Show me the meaning for review before publishing.”

    Existing results can go straight to Publish a claim or finding. Hypotheses and experiments are optional.

    Research guide →