Execution attempt

Sign in with GitHub
← Experiment E4 · Where do Claude Haiku, Sonnet, Opus and Fable stand on ZendoBench 1.0.0 dev at high reasoning effort through a tool-free Claude Code harness matched as closely as possible to E3, and how do they experiment and decide to submit?

Execution attempt · A27 · rerun

Haiku full 460-game high-effort baseline after the audit-only correction; reuse A23's completed, verified six-game smoke with unchanged model adapter and sandbox. Fresh measurement files.

Failed · Registered by @stw2 via agent. Reported and received times are kept apart; no computation or result is verified.

Pinned source and configuration

https://github.com/stw2/zendo-lab @ 105a353c9aa58ea4dae8f368cbca97e9072acd7d

Reference checked 2026-10-08 13:38 UTC. Later commits, branches or plan changes do not retarget this attempt.

Command
bash experiments/E04-claude-baselines/scripts/01_run.sh claude-haiku-5-5-claude-high
Working directory
.
Configuration paths
experiments/E04-claude-baselines/DESIGN.md
Parameters
{"reasoning_effort": "high", "thinking": "adaptive", "max_tokens": 128000, "sampling": "provider defaults", "manifest": "dev", "games": 460, "batch": 12, "player": "model", "backend": "function", "seed": "manifest episode seeds; model unseeded", "model": "claude-haiku-5-5", "sdk_identity": "You are a Claude agent, built on Anthropic's Claude Agent SDK.", "response_policy": "first complete API response; no CLI repairs", "thinking_display": "summarized"}
Environment
Claude Code 2.1.293; macOS; Apple M4 Max 128 GB local orchestrator; hosted Anthropic inference, provider hardware unknown; isolated Claude subscription login.
Output directory
Not recorded

Inputs

Materials the registrant named when registering this attempt, by content identity; obtainability is derived from their location reports. Nothing is fetched or verified.

Latest reported metrics

Values the registrant’s tool attached to its latest progress or outcome event. Substrate stores them; it verifies no measurement.

completed_games
83
unfinalized_traces
0
remaining_manifest_games
377
transcript_missing_joins
0

Delivered events

  1. Registered

    #1

    Haiku full 460-game high-effort baseline after the audit-only correction; reuse A23's completed, verified six-game smoke with unchanged model adapter and sandbox. Fresh measurement files.

    reported · received · @stw2 via agent · posted to the Thread

  2. Started

    #2

    Starting full measurement using A23's reviewed six-game smoke. Arm claude-haiku-5-5-claude-high, model claude-haiku-5-5, high effort, adaptive summarized thinking, provider-default sampling, max_tokens 128000, Claude Code 2.1.293 function backend; dev manifest, 460 full-run games, 12 concurrent calls; manifest episode seeds, model unseeded; Apple M4 Max 128 GB local orchestration, hosted inference hardware unknown. Fixed SDK identity exception documented; tools disabled; full private transcripts retained.

    reported · received · @stw2 via agent · attempt only

  3. Progress report

    #3

    Sequence stopped and requires diagnosis; no later arm launched. Private controller logs retain the failure. No success or model-loss inference is made from this stop.

    Exit code 1.

    reported · received · @stw2 via agent · attempt only

  4. Progress report

    #4

    A27 full Haiku baseline launched at 13:44:05 UTC with successful live provider responses; A23 reviewed six-game smoke supplies preflight evidence. The coordinator had briefly stopped before inference because it misread a successful event response with no timeline receipt; corrected without changing measurement source or model settings. Arm claude-haiku-5-5-claude-high, model claude-haiku-5-5, high effort, provider defaults, max_tokens 128000, function backend Claude Code 2.1.293, dev manifest, 460 planned games, 12 concurrent calls; manifest episode seeds, model unseeded; Apple M4 Max 128 GB local orchestration, hosted inference hardware unknown. No full result claimed. A28-A30 remain registered and unlaunched.

    reported · received · @stw2 via agent · attempt only

  5. Progress report

    #5

    Sequence stopped and requires diagnosis; no later arm launched. Private controller logs retain the failure. No success or model-loss inference is made from this stop.

    Exit code 5.

    reported · received · @stw2 via agent · attempt only

  6. Progress report

    #6

    Sequence stopped and requires diagnosis; no later arm launched. Private controller logs retain the failure. No success or model-loss inference is made from this stop.

    Exit code 5.

    reported · received · @stw2 via agent · attempt only

  7. Progress report

    #7

    A27 part 1 stopped after a provider Internal server error mid-response. Native CLI recovery was blocked before any second provider request; no tool ran. Diagnosis, score --verify, sandbox audit and transcript audit passed. 83 completed games retained; 12 interrupted games remain unfinished. Part 2 resumed with --exclude over the original file, under identical pinned source/settings; completed games are never replayed. Exact recurrence can be reviewed before bounded continuation; other flags still stop. Arm claude-haiku-5-5-claude-high, model claude-haiku-5-5, high effort, provider defaults, max_tokens 128000, Claude Code 2.1.293 function backend, dev manifest 460 planned games, manifest episode seeds, model unseeded; Apple M4 Max 128 GB local orchestration, hosted inference hardware unknown. No full-run result claimed; A28-A30 remain unlaunched.

    Exit code 5. Error: reviewed_provider_stream_error_and_blocked_native_recovery

    completed_games=83 · unfinalized_traces=0 · interrupted_game_records=12 · remaining_manifest_games=377 · transcript_missing_joins=0

    • runs/claude-haiku-5-5-claude-high.jsonl · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/runs/claude-haiku-5-5-claude-high.jsonl; not public; ask reporter. · restricted · sha256 bb4360a289e4… · 52038854 bytes · M103
    • runs/traces/claude-haiku-5-5-claude-high.traces.jsonl.gz · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/runs/traces/claude-haiku-5-5-claude-high.traces.jsonl.gz; not public; ask reporter. · restricted · sha256 263b5fb0958f… · 321527069 bytes · M104
    • runs/claude-haiku-5-5-claude-high.cmd · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/runs/claude-haiku-5-5-claude-high.cmd; not public; ask reporter. · restricted · sha256 bbac0faddec5… · 258 bytes · M105
    • runs/claude-haiku-5-5-claude-high.exit · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/runs/claude-haiku-5-5-claude-high.exit; not public; ask reporter. · restricted · sha256 f0b5c2c2211c… · 2 bytes · M88
    • runs/claude-haiku-5-5-claude-high.denials.json · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/runs/claude-haiku-5-5-claude-high.denials.json; not public; ask reporter. · restricted · sha256 99ee7fe28619… · 1200 bytes · M106
    • runs/haiku-part1.verified.score.json · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/runs/haiku-part1.verified.score.json; not public; ask reporter. · restricted · sha256 ef6c909337af… · 23481 bytes · M107
    • runs/haiku-part1.transcript-audit.json · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/runs/haiku-part1.transcript-audit.json; not public; ask reporter. · restricted · sha256 482386a7fb62… · 348 bytes · M108
    • runs/claude-haiku-5-5-claude-high.stream-error-review.json · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/runs/claude-haiku-5-5-claude-high.stream-error-review.json; not public; ask reporter. · restricted · sha256 f567118a856a… · 837 bytes · M109
    • runs/claude-haiku-5-5-claude-high.stream-error-review.flag.json · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/runs/claude-haiku-5-5-claude-high.stream-error-review.flag.json; not public; ask reporter. · restricted · sha256 bbf46a2a3ef8… · 501 bytes · M110

    reported · received · @stw2 via agent · attempt only

  8. Failed

    #8

    A27 ends with 83 verified completed dev games retained as fixed inputs for a new source-pinned continuation. Part 1 provider stream failure triggered blocked CLI recovery; part 2 completed zero games after isolated OAuth expiry/revocation and denied config-lock write. Reauthentication required. Original artifacts and flags retained; no completed game will be resampled. Arm claude-haiku-5-5-claude-high, model claude-haiku-5-5, high effort, provider defaults, max_tokens 128000, Claude Code 2.1.293 function backend, dev 460 planned games, manifest episode seeds, model unseeded; Apple M4 Max 128 GB orchestration, hosted inference hardware unknown. No full-run result claimed.

    Exit code 5. Error: isolated_auth_expired_and_harness_correction

    completed_games=83 · unfinalized_traces=0 · remaining_manifest_games=377 · transcript_missing_joins=0

    • runs/claude-haiku-5-5-claude-high.jsonl · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/runs/claude-haiku-5-5-claude-high.jsonl; not public; ask reporter. · restricted · sha256 bb4360a289e4… · 52038854 bytes · M103
    • runs/claude-haiku-5-5-claude-high.part2.jsonl · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/runs/claude-haiku-5-5-claude-high.part2.jsonl; not public; ask reporter. · restricted · sha256 130356e5b779… · 68401 bytes · M111
    • runs/traces/claude-haiku-5-5-claude-high.traces.jsonl.gz · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/runs/traces/claude-haiku-5-5-claude-high.traces.jsonl.gz; not public; ask reporter. · restricted · sha256 263b5fb0958f… · 321527069 bytes · M104
    • runs/traces/claude-haiku-5-5-claude-high.part2.traces.jsonl.gz · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/runs/traces/claude-haiku-5-5-claude-high.part2.traces.jsonl.gz; not public; ask reporter. · restricted · sha256 27a05127fbe2… · 15763 bytes · M112
    • runs/haiku-part1.verified.score.json · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/runs/haiku-part1.verified.score.json; not public; ask reporter. · restricted · sha256 ef6c909337af… · 23481 bytes · M107
    • runs/haiku-part2.verified.score.json · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/runs/haiku-part2.verified.score.json; not public; ask reporter. · restricted · sha256 d2fc1c9d4fec… · 21226 bytes · M113
    • runs/haiku-part1.transcript-audit.json · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/runs/haiku-part1.transcript-audit.json; not public; ask reporter. · restricted · sha256 482386a7fb62… · 348 bytes · M108
    • runs/haiku-part2.transcript-audit.json · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/runs/haiku-part2.transcript-audit.json; not public; ask reporter. · restricted · sha256 320cdb1dd1cf… · 342 bytes · M114
    • runs/preflight/a27-auth-stop/10b60199108843bb89f27821253750b0.json · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/runs/preflight/a27-auth-stop/10b60199108843bb89f27821253750b0.json; not public; ask reporter. · restricted · sha256 0644015db604… · 1005 bytes · M115
    • runs/preflight/a27-auth-stop/aaf8b788b86c4a37b14537c63805be5e.json · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/runs/preflight/a27-auth-stop/aaf8b788b86c4a37b14537c63805be5e.json; not public; ask reporter. · restricted · sha256 8ddcd5696ee1… · 500 bytes · M116

    reported · received · @stw2 via agent · posted to the Thread

Report an event

The registrant’s capture tool normally delivers start and outcome events. Reporting here is the same author report with the browser as the reported time; it does not observe the process.

This attempt has a delivered outcome. A new execution is a new attempt.