Execution attempt

Sign in with GitHub
← Experiment E4 · Where do Claude Haiku, Sonnet, Opus and Fable stand on ZendoBench 1.0.0 dev at high reasoning effort through a tool-free Claude Code harness matched as closely as possible to E3, and how do they experiment and decide to submit?

Execution attempt · A35 · rerun

claude-sonnet-5-5-claude-high-v2: six-game smoke, verification and full 460-game dev baseline; runs concurrently with the other Haiku/Sonnet/Opus arms, at 12 calls per model.

Succeeded · Registered by @stw2 via agent. Reported and received times are kept apart; no computation or result is verified.

Pinned source and configuration

https://github.com/stw2/zendo-lab @ c4bf8e221582d3fbfa576ec88253ede4c9ccc03e

Reference checked 2026-10-08 20:21 UTC. Later commits, branches or plan changes do not retarget this attempt.

Command
bash experiments/E04-claude-baselines/scripts/01_run.sh claude-sonnet-5-5-claude-high-v2
Working directory
.
Configuration paths
experiments/E04-claude-baselines/DESIGN.md
Parameters
{"reasoning_effort": "high", "thinking": "adaptive", "max_tokens": 128000, "sampling": "provider defaults", "manifest": "dev", "games": 460, "batch": 12, "player": "model", "backend": "function", "seed": "manifest episode seeds; model unseeded", "model": "claude-sonnet-5-5", "sdk_identity": "You are a Claude agent, built on Anthropic's Claude Agent SDK.", "response_policy": "first complete API response; no CLI repairs", "thinking_display": "summarized"}
Environment
Claude Code 2.1.293; macOS; Apple M4 Max 128 GB local orchestrator; hosted Anthropic inference, provider hardware unknown; isolated Claude subscription login. Owner-requested parallel scheduling: Haiku/Sonnet/Opus overlap; Fable follows after verification.
Output directory
Not recorded

Inputs

Materials the registrant named when registering this attempt, by content identity; obtainability is derived from their location reports. Nothing is fetched or verified.

Latest reported metrics

Values the registrant’s tool attached to its latest progress or outcome event. Substrate stores them; it verifies no measurement.

claude-sonnet-5-5-claude-high-v2.eig
0.4021656997676348
claude-sonnet-5-5-claude-high-v2.zero
0.5191784211828755
claude-sonnet-5-5-claude-high-v2.calls
9931
claude-sonnet-5-5-claude-high-v2.T1.wins
30
claude-sonnet-5-5-claude-high-v2.T2.wins
58
claude-sonnet-5-5-claude-high-v2.T3.wins
141
claude-sonnet-5-5-claude-high-v2.T4.wins
20
claude-sonnet-5-5-claude-high-v2.T5.wins
77
claude-sonnet-5-5-claude-high-v2.T6.wins
36
claude-sonnet-5-5-claude-high-v2.p_first
0.8194216400476783
claude-sonnet-5-5-claude-high-v2.repeats
8
claude-sonnet-5-5-claude-high-v2.headline
0.722
claude-sonnet-5-5-claude-high-v2.T1.scored
30
claude-sonnet-5-5-claude-high-v2.T2.scored
60
claude-sonnet-5-5-claude-high-v2.T3.scored
150
claude-sonnet-5-5-claude-high-v2.T4.scored
60
claude-sonnet-5-5-claude-high-v2.T5.scored
100
claude-sonnet-5-5-claude-high-v2.T6.scored
60
claude-sonnet-5-5-claude-high-v2.alive_first
1
claude-sonnet-5-5-claude-high-v2.nodes_ratio
1.224689730478321
claude-sonnet-5-5-claude-high-v2.bits_per_exp
0.4073023614358135
claude-sonnet-5-5-claude-high-v2.exp_per_game
17.569565217391304
claude-sonnet-5-5-claude-high-v2.certain_wrong
110
claude-sonnet-5-5-claude-high-v2.subs_per_game
1.2956521739130435

and 10 more in the attempt JSON.

Delivered events

  1. Registered

    #1

    claude-sonnet-5-5-claude-high-v2: six-game smoke, verification and full 460-game dev baseline; runs concurrently with the other Haiku/Sonnet/Opus arms, at 12 calls per model.

    reported · received · @stw2 via agent · posted to the Thread

  2. Started

    #2

    Starting separate six-game smoke; full measurement follows only after score --verify and transcript audit pass. Arm claude-sonnet-5-5-claude-high-v2, model claude-sonnet-5-5, high effort, adaptive summarized thinking, provider-default sampling, max_tokens 128000, Claude Code 2.1.293 function backend; dev manifest, 460 full-run games, 12 concurrent calls per arm; manifest episode seeds, model unseeded; Apple M4 Max 128 GB local orchestration, hosted inference hardware unknown. Owner-authorized Haiku/Sonnet/Opus overlap, followed by Fable; fixed SDK identity, disabled tools and full private transcript capture unchanged.

    reported · received · @stw2 via agent · attempt only

  3. Progress report

    #3

    This arm stopped and requires diagnosis. Other already-running arms may finish; no new arm or part starts until reviewed. No model loss inferred. Arm claude-sonnet-5-5-claude-high-v2, model claude-sonnet-5-5, high effort, adaptive summarized thinking, provider-default sampling, max_tokens 128000, Claude Code 2.1.293 function backend; dev manifest, 460 full-run games, 12 concurrent calls per arm; manifest episode seeds, model unseeded; Apple M4 Max 128 GB local orchestration, hosted inference hardware unknown. Owner-authorized Haiku/Sonnet/Opus overlap, followed by Fable; fixed SDK identity, disabled tools and full private transcript capture unchanged.

    Exit code 5.

    reported · received · @stw2 via agent · attempt only

  4. Progress report

    #4

    Reviewed incomplete provider streams followed by guard-blocked native CLI retries: exactly one original request was forwarded per rejected call, no completion marker was received, and no second inference request reached the provider. Sandbox denial audits flag no unexpected access. Additional Sonnet shutdown-finalization flags concern aborted in-flight streams; all retained call traces have end records, completed-game transcript joins pass, and unfinished games are excluded from scoring. No guard, retry classifier, prompt, model or measurement source is changed for continuation. Continuing only unfinished games in a fresh immutable part under the same attempt and byte-identical measurement configuration. Completed games and verified smoke evidence are retained, not resampled. Arm claude-sonnet-5-5-claude-high-v2, model claude-sonnet-5-5, high effort, adaptive summarized thinking, provider-default sampling, max_tokens 128000, Claude Code 2.1.293 function backend; dev manifest, 460 full-run games, 12 concurrent calls per arm; manifest episode seeds, model unseeded; Apple M4 Max 128 GB local orchestration, hosted inference hardware unknown. Owner-authorized Haiku/Sonnet/Opus overlap, followed by Fable; fixed SDK identity, disabled tools and full private transcript capture unchanged.

    reported · received · @stw2 via agent · attempt only

  5. Succeeded

    #5

    Full baseline completed, score --verify and transcript audit passed; paired E3 comparisons completed. Arm claude-sonnet-5-5-claude-high-v2, model claude-sonnet-5-5, high effort, adaptive summarized thinking, provider-default sampling, max_tokens 128000, Claude Code 2.1.293 function backend; dev manifest, 460 full-run games, 12 concurrent calls per arm; manifest episode seeds, model unseeded; Apple M4 Max 128 GB local orchestration, hosted inference hardware unknown. Owner-authorized Haiku/Sonnet/Opus overlap, followed by Fable; fixed SDK identity, disabled tools and full private transcript capture unchanged.

    Exit code 0.

    claude-sonnet-5-5-claude-high-v2.eig=0.4021656997676348 · claude-sonnet-5-5-claude-high-v2.zero=0.5191784211828755 · claude-sonnet-5-5-claude-high-v2.calls=9931 · claude-sonnet-5-5-claude-high-v2.T1.wins=30 · claude-sonnet-5-5-claude-high-v2.T2.wins=58 · claude-sonnet-5-5-claude-high-v2.T3.wins=141 · claude-sonnet-5-5-claude-high-v2.T4.wins=20 · claude-sonnet-5-5-claude-high-v2.T5.wins=77 · …

    • comparisons/claude-sonnet-5-5-claude-high-v2__vs__gpt-5.6-terra-codex-high.json · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/comparisons/claude-sonnet-5-5-claude-high-v2__vs__gpt-5.6-terra-codex-high.json; not public; ask reporter. · restricted · sha256 f2df11492c63… · 31779 bytes · M146
    • comparisons/claude-sonnet-5-5-claude-high-v2__vs__gpt-5.6-terra-codex-high.md · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/comparisons/claude-sonnet-5-5-claude-high-v2__vs__gpt-5.6-terra-codex-high.md; not public; ask reporter. · restricted · sha256 ff5bef4e77a4… · 5720 bytes · M147
    • comparisons/claude-sonnet-5-5-claude-high-v2__vs__gpt-6-astra-codex-high.json · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/comparisons/claude-sonnet-5-5-claude-high-v2__vs__gpt-6-astra-codex-high.json; not public; ask reporter. · restricted · sha256 3c6b6757665d… · 31543 bytes · M148
    • comparisons/claude-sonnet-5-5-claude-high-v2__vs__gpt-6-astra-codex-high.md · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/comparisons/claude-sonnet-5-5-claude-high-v2__vs__gpt-6-astra-codex-high.md; not public; ask reporter. · restricted · sha256 f6bb9cd39e6d… · 5801 bytes · M149
    • comparisons/claude-sonnet-5-5-claude-high-v2__vs__gpt-6-luna-codex-high.json · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/comparisons/claude-sonnet-5-5-claude-high-v2__vs__gpt-6-luna-codex-high.json; not public; ask reporter. · restricted · sha256 446c72b9cc10… · 35712 bytes · M150
    • comparisons/claude-sonnet-5-5-claude-high-v2__vs__gpt-6-luna-codex-high.md · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/comparisons/claude-sonnet-5-5-claude-high-v2__vs__gpt-6-luna-codex-high.md; not public; ask reporter. · restricted · sha256 e6124d69442d… · 5706 bytes · M151
    • comparisons/claude-sonnet-5-5-claude-high-v2__vs__gpt-6-sol-codex-high.json · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/comparisons/claude-sonnet-5-5-claude-high-v2__vs__gpt-6-sol-codex-high.json; not public; ask reporter. · restricted · sha256 30c6de4d5eb5… · 31499 bytes · M152
    • comparisons/claude-sonnet-5-5-claude-high-v2__vs__gpt-6-sol-codex-high.md · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/comparisons/claude-sonnet-5-5-claude-high-v2__vs__gpt-6-sol-codex-high.md; not public; ask reporter. · restricted · sha256 dfcefbacc427… · 5770 bytes · M153
    • comparisons/claude-sonnet-5-5-claude-high-v2__vs__gpt-6.1-sol-codex-high.json · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/comparisons/claude-sonnet-5-5-claude-high-v2__vs__gpt-6.1-sol-codex-high.json; not public; ask reporter. · restricted · sha256 a7502b227fa5… · 31569 bytes · M154
    • comparisons/claude-sonnet-5-5-claude-high-v2__vs__gpt-6.1-sol-codex-high.md · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/comparisons/claude-sonnet-5-5-claude-high-v2__vs__gpt-6.1-sol-codex-high.md; not public; ask reporter. · restricted · sha256 cf17caf3a53c… · 5801 bytes · M155
    • runs/claude-sonnet-5-5-claude-high-v2.cmd · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/runs/claude-sonnet-5-5-claude-high-v2.cmd; not public; ask reporter. · restricted · sha256 bf9754133e2f… · 266 bytes · M156
    • runs/claude-sonnet-5-5-claude-high-v2.denials.json · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/runs/claude-sonnet-5-5-claude-high-v2.denials.json; not public; ask reporter. · restricted · sha256 a7719b30ac7c… · 721 bytes · M157
    • runs/claude-sonnet-5-5-claude-high-v2.diagnose.json · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/runs/claude-sonnet-5-5-claude-high-v2.diagnose.json; not public; ask reporter. · restricted · sha256 5226dfd1ae01… · 1523531 bytes · M158
    • runs/claude-sonnet-5-5-claude-high-v2.exit · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/runs/claude-sonnet-5-5-claude-high-v2.exit; not public; ask reporter. · restricted · sha256 f0b5c2c2211c… · 2 bytes · M88
    • runs/claude-sonnet-5-5-claude-high-v2.jsonl · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/runs/claude-sonnet-5-5-claude-high-v2.jsonl; not public; ask reporter. · restricted · sha256 c3cbbeb59302… · 2010539 bytes · M159
    • runs/claude-sonnet-5-5-claude-high-v2.log · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/runs/claude-sonnet-5-5-claude-high-v2.log; not public; ask reporter. · restricted · sha256 6db0205a7bb6… · 54 bytes · M160
    • runs/claude-sonnet-5-5-claude-high-v2.part2.cmd · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/runs/claude-sonnet-5-5-claude-high-v2.part2.cmd; not public; ask reporter. · restricted · sha256 b2511b972502… · 272 bytes · M161
    • runs/claude-sonnet-5-5-claude-high-v2.part2.denials.json · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/runs/claude-sonnet-5-5-claude-high-v2.part2.denials.json; not public; ask reporter. · restricted · sha256 0be2ac3ca0d2… · 1209 bytes · M162
    • runs/claude-sonnet-5-5-claude-high-v2.part2.exit · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/runs/claude-sonnet-5-5-claude-high-v2.part2.exit; not public; ask reporter. · restricted · sha256 9a271f2a916b… · 2 bytes · M130
    • runs/claude-sonnet-5-5-claude-high-v2.part2.jsonl · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/runs/claude-sonnet-5-5-claude-high-v2.part2.jsonl; not public; ask reporter. · restricted · sha256 d61ff500faa9… · 51306261 bytes · M163
    • runs/claude-sonnet-5-5-claude-high-v2.part2.log · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/runs/claude-sonnet-5-5-claude-high-v2.part2.log; not public; ask reporter. · restricted · sha256 82dfbb1cf8f8… · 336 bytes · M164
    • runs/smoke/claude-sonnet-5-5-claude-high-v2.audit.json · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/runs/smoke/claude-sonnet-5-5-claude-high-v2.audit.json; not public; ask reporter. · restricted · sha256 9e7af9ea1e6e… · 346 bytes · M165
    • runs/smoke/claude-sonnet-5-5-claude-high-v2.cmd · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/runs/smoke/claude-sonnet-5-5-claude-high-v2.cmd; not public; ask reporter. · restricted · sha256 09b3109d2345… · 272 bytes · M166
    • runs/smoke/claude-sonnet-5-5-claude-high-v2.denials.json · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/runs/smoke/claude-sonnet-5-5-claude-high-v2.denials.json; not public; ask reporter. · restricted · sha256 320db9401a63… · 896 bytes · M167
    • runs/smoke/claude-sonnet-5-5-claude-high-v2.exit · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/runs/smoke/claude-sonnet-5-5-claude-high-v2.exit; not public; ask reporter. · restricted · sha256 9a271f2a916b… · 2 bytes · M130
    • runs/smoke/claude-sonnet-5-5-claude-high-v2.jsonl · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/runs/smoke/claude-sonnet-5-5-claude-high-v2.jsonl; not public; ask reporter. · restricted · sha256 b3895029cc70… · 621891 bytes · M168
    • runs/smoke/claude-sonnet-5-5-claude-high-v2.log · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/runs/smoke/claude-sonnet-5-5-claude-high-v2.log; not public; ask reporter. · restricted · sha256 71898fc62641… · 333 bytes · M169
    • runs/smoke/claude-sonnet-5-5-claude-high-v2.verify.log · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/runs/smoke/claude-sonnet-5-5-claude-high-v2.verify.log; not public; ask reporter. · restricted · sha256 dc50bd8ae3c2… · 3591 bytes · M170
    • runs/smoke/traces/claude-sonnet-5-5-claude-high-v2.traces.jsonl.gz · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/runs/smoke/traces/claude-sonnet-5-5-claude-high-v2.traces.jsonl.gz; not public; ask reporter. · restricted · sha256 cbb08d64bc1c… · 4827403 bytes · M171
    • runs/traces/claude-sonnet-5-5-claude-high-v2.part2.traces.jsonl.gz · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/runs/traces/claude-sonnet-5-5-claude-high-v2.part2.traces.jsonl.gz; not public; ask reporter. · restricted · sha256 98b5d26fff2a… · 368138807 bytes · M172
    • runs/traces/claude-sonnet-5-5-claude-high-v2.traces.jsonl.gz · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/runs/traces/claude-sonnet-5-5-claude-high-v2.traces.jsonl.gz; not public; ask reporter. · restricted · sha256 70ee52754304… · 12334354 bytes · M173
    • scores/claude-sonnet-5-5-claude-high-v2.diagnose.md · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/scores/claude-sonnet-5-5-claude-high-v2.diagnose.md; not public; ask reporter. · restricted · sha256 34e775a2a3d8… · 1623 bytes · M174
    • scores/claude-sonnet-5-5-claude-high-v2.files.json · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/scores/claude-sonnet-5-5-claude-high-v2.files.json; not public; ask reporter. · restricted · sha256 b7199f8724f5… · 326 bytes · M175
    • scores/claude-sonnet-5-5-claude-high-v2.score.json · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/scores/claude-sonnet-5-5-claude-high-v2.score.json; not public; ask reporter. · restricted · sha256 6bfb5afadf0c… · 29790 bytes · M176
    • scores/claude-sonnet-5-5-claude-high-v2.score.md · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/scores/claude-sonnet-5-5-claude-high-v2.score.md; not public; ask reporter. · restricted · sha256 bbd52b93b0c1… · 3726 bytes · M177
    • scores/claude-sonnet-5-5-claude-high-v2.traces.json · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/scores/claude-sonnet-5-5-claude-high-v2.traces.json; not public; ask reporter. · restricted · sha256 64e3b5b257b6… · 547 bytes · M178

    reported · received · @stw2 via agent · posted to the Thread

Report an event

The registrant’s capture tool normally delivers start and outcome events. Reporting here is the same author report with the browser as the reported time; it does not observe the process.

This attempt has a delivered outcome. A new execution is a new attempt.