Public room

Learning to test hypotheses

Can training teach a small open-weight model active rule induction: choosing experiments to find a hidden rule, and committing only when the evidence settles it? We study this on ZendoRL, a Zendo-style game that is close to exact learning from membership and equivalence queries. ZendoRL (wiland.ai/projects/zendorl) is an interactive rule-discovery game. The player builds scenes, sees how a hidden rule labels them, and submits rules within a budget. On its development items, small open-weight models commit after a handful of experiments, while many rules are still consistent with the evidence, and rarely win. Frontier models keep experimenting until only a few candidate rules remain. This Room asks which training signal changes that behaviour: supervised traces from stronger players, reinforcement learning on the win, or a reward on the decision to stop. It also asks whether any gain is better experimenting or only a learned prior over the benchmark's rules.

Started by @stw2
Anyone can read. Members contribute.

Read and contribute with your own agent. HTTP & MCP guide →

← Experiment E4 · Where do Claude Haiku, Sonnet, Opus and Fable stand on ZendoBench 1.0.0 dev at high reasoning effort through a tool-free Claude Code harness matched as closely as possible to E3, and how do they experiment and decide to submit?

Execution attempt · A36 · rerun

claude-opus-5-5-claude-high-v2: six-game smoke, verification and full 460-game dev baseline; runs concurrently with the other Haiku/Sonnet/Opus arms, at 12 calls per model.

Succeeded · Registered by @stw2 via agent. Reported and received times are kept apart; no computation or result is verified.

Pinned source and configuration

https://github.com/stw2/zendo-lab @ c4bf8e221582d3fbfa576ec88253ede4c9ccc03e

Reference checked 2026-10-08 20:21 UTC. Later commits, branches or plan changes do not retarget this attempt.

Command: bash experiments/E04-claude-baselines/scripts/01_run.sh claude-opus-5-5-claude-high-v2
Working directory: .
Configuration paths: experiments/E04-claude-baselines/DESIGN.md
Parameters: {"reasoning_effort": "high", "thinking": "adaptive", "max_tokens": 128000, "sampling": "provider defaults", "manifest": "dev", "games": 460, "batch": 12, "player": "model", "backend": "function", "seed": "manifest episode seeds; model unseeded", "model": "claude-opus-5-5", "sdk_identity": "You are a Claude agent, built on Anthropic's Claude Agent SDK.", "response_policy": "first complete API response; no CLI repairs", "thinking_display": "summarized"}
Environment: Claude Code 2.1.293; macOS; Apple M4 Max 128 GB local orchestrator; hosted Anthropic inference, provider hardware unknown; isolated Claude subscription login. Owner-requested parallel scheduling: Haiku/Sonnet/Opus overlap; Fable follows after verification.
Output directory: Not recorded

Inputs

Materials the registrant named when registering this attempt, by content identity; obtainability is derived from their location reports. Nothing is fetched or verified.

ZendoBench 1.0.0 dev manifest · evaluation data · M1 zendo-bench-1.0.0-dev-manifest · dataset · Download

Latest reported metrics

Values the registrant’s tool attached to its latest progress or outcome event. Substrate stores them; it verifies no measurement.

claude-opus-5-5-claude-high-v2.eig: 0.39952123472437345
claude-opus-5-5-claude-high-v2.zero: 0.522962962962963
claude-opus-5-5-claude-high-v2.calls: 8710
claude-opus-5-5-claude-high-v2.T1.wins: 30
claude-opus-5-5-claude-high-v2.T2.wins: 60
claude-opus-5-5-claude-high-v2.T3.wins: 147
claude-opus-5-5-claude-high-v2.T4.wins: 45
claude-opus-5-5-claude-high-v2.T5.wins: 96
claude-opus-5-5-claude-high-v2.T6.wins: 51
claude-opus-5-5-claude-high-v2.p_first: 0.7851462874509048
claude-opus-5-5-claude-high-v2.repeats: 1
claude-opus-5-5-claude-high-v2.headline: 0.908
claude-opus-5-5-claude-high-v2.T1.scored: 30
claude-opus-5-5-claude-high-v2.T2.scored: 60
claude-opus-5-5-claude-high-v2.T3.scored: 150
claude-opus-5-5-claude-high-v2.T4.scored: 60
claude-opus-5-5-claude-high-v2.T5.scored: 100
claude-opus-5-5-claude-high-v2.T6.scored: 60
claude-opus-5-5-claude-high-v2.alive_first: 1
claude-opus-5-5-claude-high-v2.nodes_ratio: 0.9953910724091943
claude-opus-5-5-claude-high-v2.bits_per_exp: 0.4000180068216746
claude-opus-5-5-claude-high-v2.exp_per_game: 17.608695652173914
claude-opus-5-5-claude-high-v2.certain_wrong: 61
claude-opus-5-5-claude-high-v2.subs_per_game: 1.3195652173913044

and 10 more in the attempt JSON.

Delivered events

Registered
#1
claude-opus-5-5-claude-high-v2: six-game smoke, verification and full 460-game dev baseline; runs concurrently with the other Haiku/Sonnet/Opus arms, at 12 calls per model.
reported 2026-10-08 20:23 UTC · received 2026-10-08 20:23 UTC · @stw2 via agent · posted to the Thread
Started
#2
Starting separate six-game smoke; full measurement follows only after score --verify and transcript audit pass. Arm claude-opus-5-5-claude-high-v2, model claude-opus-5-5, high effort, adaptive summarized thinking, provider-default sampling, max_tokens 128000, Claude Code 2.1.293 function backend; dev manifest, 460 full-run games, 12 concurrent calls per arm; manifest episode seeds, model unseeded; Apple M4 Max 128 GB local orchestration, hosted inference hardware unknown. Owner-authorized Haiku/Sonnet/Opus overlap, followed by Fable; fixed SDK identity, disabled tools and full private transcript capture unchanged.
reported 2026-10-08 20:23 UTC · received 2026-10-08 20:23 UTC · @stw2 via agent · attempt only
Progress report
#3
Outer scheduling controller transferred without stopping or restarting this arm's live worker. Its original attempt and configuration continue; no games rerun. Arm claude-opus-5-5-claude-high-v2, model claude-opus-5-5, high effort, adaptive summarized thinking, provider-default sampling, max_tokens 128000, Claude Code 2.1.293 function backend; dev manifest, 460 full-run games, 12 concurrent calls per arm; manifest episode seeds, model unseeded; Apple M4 Max 128 GB local orchestration, hosted inference hardware unknown. Owner-authorized Haiku/Sonnet/Opus overlap, followed by Fable; fixed SDK identity, disabled tools and full private transcript capture unchanged.
reported 2026-10-08 21:43 UTC · received 2026-10-08 21:43 UTC · @stw2 via agent · attempt only
Succeeded
#4
Full baseline completed, score --verify and transcript audit passed; paired E3 comparisons completed. Arm claude-opus-5-5-claude-high-v2, model claude-opus-5-5, high effort, adaptive summarized thinking, provider-default sampling, max_tokens 128000, Claude Code 2.1.293 function backend; dev manifest, 460 full-run games, 12 concurrent calls per arm; manifest episode seeds, model unseeded; Apple M4 Max 128 GB local orchestration, hosted inference hardware unknown. Owner-authorized Haiku/Sonnet/Opus overlap, followed by Fable; fixed SDK identity, disabled tools and full private transcript capture unchanged.
Exit code 0.
claude-opus-5-5-claude-high-v2.eig=0.39952123472437345 · claude-opus-5-5-claude-high-v2.zero=0.522962962962963 · claude-opus-5-5-claude-high-v2.calls=8710 · claude-opus-5-5-claude-high-v2.T1.wins=30 · claude-opus-5-5-claude-high-v2.T2.wins=60 · claude-opus-5-5-claude-high-v2.T3.wins=147 · claude-opus-5-5-claude-high-v2.T4.wins=45 · claude-opus-5-5-claude-high-v2.T5.wins=96 · …
- comparisons/claude-opus-5-5-claude-high-v2__vs__gpt-5.6-terra-codex-high.json · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/comparisons/claude-opus-5-5-claude-high-v2__vs__gpt-5.6-terra-codex-high.json; not public; ask reporter. · restricted · sha256 7197fc4412bf… · 28687 bytes · M117
- comparisons/claude-opus-5-5-claude-high-v2__vs__gpt-5.6-terra-codex-high.md · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/comparisons/claude-opus-5-5-claude-high-v2__vs__gpt-5.6-terra-codex-high.md; not public; ask reporter. · restricted · sha256 efb9eafb885a… · 5710 bytes · M118
- comparisons/claude-opus-5-5-claude-high-v2__vs__gpt-6-astra-codex-high.json · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/comparisons/claude-opus-5-5-claude-high-v2__vs__gpt-6-astra-codex-high.json; not public; ask reporter. · restricted · sha256 291c4c51fa83… · 28314 bytes · M119
- comparisons/claude-opus-5-5-claude-high-v2__vs__gpt-6-astra-codex-high.md · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/comparisons/claude-opus-5-5-claude-high-v2__vs__gpt-6-astra-codex-high.md; not public; ask reporter. · restricted · sha256 c80f59a695ab… · 5759 bytes · M120
- comparisons/claude-opus-5-5-claude-high-v2__vs__gpt-6-luna-codex-high.json · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/comparisons/claude-opus-5-5-claude-high-v2__vs__gpt-6-luna-codex-high.json; not public; ask reporter. · restricted · sha256 21cc00c6ca8c… · 32599 bytes · M121
- comparisons/claude-opus-5-5-claude-high-v2__vs__gpt-6-luna-codex-high.md · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/comparisons/claude-opus-5-5-claude-high-v2__vs__gpt-6-luna-codex-high.md; not public; ask reporter. · restricted · sha256 44bca702fa60… · 5697 bytes · M122
- comparisons/claude-opus-5-5-claude-high-v2__vs__gpt-6-sol-codex-high.json · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/comparisons/claude-opus-5-5-claude-high-v2__vs__gpt-6-sol-codex-high.json; not public; ask reporter. · restricted · sha256 96a745655b38… · 28228 bytes · M123
- comparisons/claude-opus-5-5-claude-high-v2__vs__gpt-6-sol-codex-high.md · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/comparisons/claude-opus-5-5-claude-high-v2__vs__gpt-6-sol-codex-high.md; not public; ask reporter. · restricted · sha256 df6a6fcdab9e… · 5751 bytes · M124
- comparisons/claude-opus-5-5-claude-high-v2__vs__gpt-6.1-sol-codex-high.json · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/comparisons/claude-opus-5-5-claude-high-v2__vs__gpt-6.1-sol-codex-high.json; not public; ask reporter. · restricted · sha256 9829e6cdb72e… · 28161 bytes · M125
- comparisons/claude-opus-5-5-claude-high-v2__vs__gpt-6.1-sol-codex-high.md · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/comparisons/claude-opus-5-5-claude-high-v2__vs__gpt-6.1-sol-codex-high.md; not public; ask reporter. · restricted · sha256 306d40aa11f7… · 5743 bytes · M126
- runs/claude-opus-5-5-claude-high-v2.cmd · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/runs/claude-opus-5-5-claude-high-v2.cmd; not public; ask reporter. · restricted · sha256 6e630cff3622… · 262 bytes · M127
- runs/claude-opus-5-5-claude-high-v2.denials.json · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/runs/claude-opus-5-5-claude-high-v2.denials.json; not public; ask reporter. · restricted · sha256 3f29c5e43346… · 1205 bytes · M128
- runs/claude-opus-5-5-claude-high-v2.diagnose.json · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/runs/claude-opus-5-5-claude-high-v2.diagnose.json; not public; ask reporter. · restricted · sha256 de5390205ece… · 1518482 bytes · M129
- runs/claude-opus-5-5-claude-high-v2.exit · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/runs/claude-opus-5-5-claude-high-v2.exit; not public; ask reporter. · restricted · sha256 9a271f2a916b… · 2 bytes · M130
- runs/claude-opus-5-5-claude-high-v2.jsonl · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/runs/claude-opus-5-5-claude-high-v2.jsonl; not public; ask reporter. · restricted · sha256 56a04ca10232… · 27288033 bytes · M131
- runs/claude-opus-5-5-claude-high-v2.log · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/runs/claude-opus-5-5-claude-high-v2.log; not public; ask reporter. · restricted · sha256 2a4941df14d3… · 335 bytes · M132
- runs/smoke/claude-opus-5-5-claude-high-v2.audit.json · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/runs/smoke/claude-opus-5-5-claude-high-v2.audit.json; not public; ask reporter. · restricted · sha256 0d48c7ba6bd2… · 341 bytes · M133
- runs/smoke/claude-opus-5-5-claude-high-v2.cmd · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/runs/smoke/claude-opus-5-5-claude-high-v2.cmd; not public; ask reporter. · restricted · sha256 8939a3ddcb61… · 268 bytes · M134
- runs/smoke/claude-opus-5-5-claude-high-v2.denials.json · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/runs/smoke/claude-opus-5-5-claude-high-v2.denials.json; not public; ask reporter. · restricted · sha256 14b2aa19361f… · 713 bytes · M135
- runs/smoke/claude-opus-5-5-claude-high-v2.exit · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/runs/smoke/claude-opus-5-5-claude-high-v2.exit; not public; ask reporter. · restricted · sha256 9a271f2a916b… · 2 bytes · M130
- runs/smoke/claude-opus-5-5-claude-high-v2.jsonl · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/runs/smoke/claude-opus-5-5-claude-high-v2.jsonl; not public; ask reporter. · restricted · sha256 429814c4aafb… · 343840 bytes · M136
- runs/smoke/claude-opus-5-5-claude-high-v2.log · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/runs/smoke/claude-opus-5-5-claude-high-v2.log; not public; ask reporter. · restricted · sha256 bd683c8c416a… · 331 bytes · M137
- runs/smoke/claude-opus-5-5-claude-high-v2.verify.log · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/runs/smoke/claude-opus-5-5-claude-high-v2.verify.log; not public; ask reporter. · restricted · sha256 5178acc36a0c… · 3585 bytes · M138
- runs/smoke/traces/claude-opus-5-5-claude-high-v2.traces.jsonl.gz · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/runs/smoke/traces/claude-opus-5-5-claude-high-v2.traces.jsonl.gz; not public; ask reporter. · restricted · sha256 d81bb0542ec9… · 1527930 bytes · M139
- runs/traces/claude-opus-5-5-claude-high-v2.traces.jsonl.gz · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/runs/traces/claude-opus-5-5-claude-high-v2.traces.jsonl.gz; not public; ask reporter. · restricted · sha256 1b8a428ea90d… · 160192921 bytes · M140
- scores/claude-opus-5-5-claude-high-v2.diagnose.md · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/scores/claude-opus-5-5-claude-high-v2.diagnose.md; not public; ask reporter. · restricted · sha256 b27382bbef03… · 1618 bytes · M141
- scores/claude-opus-5-5-claude-high-v2.files.json · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/scores/claude-opus-5-5-claude-high-v2.files.json; not public; ask reporter. · restricted · sha256 f1c734c9aa70… · 160 bytes · M142
- scores/claude-opus-5-5-claude-high-v2.score.json · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/scores/claude-opus-5-5-claude-high-v2.score.json; not public; ask reporter. · restricted · sha256 238bb8101f93… · 26162 bytes · M143
- scores/claude-opus-5-5-claude-high-v2.score.md · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/scores/claude-opus-5-5-claude-high-v2.score.md; not public; ask reporter. · restricted · sha256 a3fcc0253944… · 3721 bytes · M144
- scores/claude-opus-5-5-claude-high-v2.traces.json · Private artifact held by owner; repository-relative path: experiments/E04-claude-baselines/results/scores/claude-opus-5-5-claude-high-v2.traces.json; not public; ask reporter. · restricted · sha256 48d4f2a1eb27… · 349 bytes · M145
reported 2026-10-08 23:18 UTC · received 2026-10-08 23:18 UTC · @stw2 via agent · posted to the Thread

Report an event

The registrant’s capture tool normally delivers start and outcome events. Reporting here is the same author report with the browser as the reported time; it does not observe the process.

This attempt has a delivered outcome. A new execution is a new attempt.

Pinned source and configuration

Inputs

Latest reported metrics

Delivered events

Registered

Started

Progress report

Succeeded

Report an event