Execution attempt

Sign in with GitHub
← Experiment E3 · Where do OpenAI's current frontier models stand on ZendoBench 1.0.0 dev when they play through the Codex CLI with every tool off, and how do they play: how much information do their experiments gain, and how many rule classes are still alive when they submit?

Execution attempt · A14 · planned

E3 arm gpt-6.1-sol-codex-high: all 460 dev games through the Codex CLI with no tools, run per the accepted plan; launched after the arm before it by scripts/01_all.sh

Succeeded · Registered by @stw2 via agent. Reported and received times are kept apart; no computation or result is verified.

Pinned source and configuration

https://github.com/stw2/zendo-lab @ 9939c141fc598a90e9d05b4735ce584dc8277bdd

Reference checked 2026-10-06 21:45 UTC. Later commits, branches or plan changes do not retarget this attempt.

Command
bash experiments/E03-codex-frontier/scripts/01_run.sh gpt-6.1-sol-codex-high
Working directory
.
Configuration paths
experiments/E03-codex-frontier/scripts/sandbox.sb experiments/E03-codex-frontier/scripts/codex-cli/package-lock.json
Parameters
model=gpt-6.1-sol; backend=Codex CLI 0.160.0 (codex exec, no tools), 12 calls in flight; hardware=provider; sampling=reasoning effort high, summary detailed (Codex defaults: temperature 1.0, top_p 0.98, verbosity low); output cap=none
Environment
macOS 26.5 on Apple M4 Max 128 GB (harness only); Python 3.13 via uv; ZendoBench 1.0.0; Codex CLI 0.160.0 (codex exec, no tools) under sandbox-exec, a new empty folder per call; ChatGPT sign-in in a Codex home of its own; the model served by OpenAI
Output directory
Not recorded

Inputs

Materials the registrant named when registering this attempt, by content identity; obtainability is derived from their location reports. Nothing is fetched or verified.

Latest reported metrics

Values the registrant’s tool attached to its latest progress or outcome event. Substrate stores them; it verifies no measurement.

T1.wins
30
T2.wins
60
T3.wins
150
T4.wins
43
T5.wins
90
T6.wins
52
T1.scored
30
T2.scored
60
T3.scored
150
T4.scored
60
T5.scored
100
T6.scored
60
T1.win_rate
1
T2.win_rate
1
T3.win_rate
1
T4.win_rate
0.7166666666666667
T5.win_rate
0.9
T6.win_rate
0.8666666666666666
diagnose.eig
0.3682627416392839
diagnose.zero
0.5698498498498499
headline_items
430
headline_t2_t6
0.8966666666666667
headline_ci_low
0.8473396743935219
diagnose.p_first
0.7318050347645095

and 11 more in the attempt JSON.

Delivered events

  1. Registered

    #1

    E3 arm gpt-6.1-sol-codex-high: all 460 dev games through the Codex CLI with no tools, run per the accepted plan; launched after the arm before it by scripts/01_all.sh

    reported · received · @stw2 via agent · posted to the Thread

  2. Started

    #2

    Launched 2026-10-07T01:58:16Z at d351665 by scripts/01_all.sh, after the arm before it and this arm's connection check (exit 0): bash experiments/E03-codex-frontier/scripts/01_run.sh gpt-6.1-sol-codex-high.

    reported · received · @stw2 via agent · attempt only

  3. Succeeded

    #3

    Exit 0; 430/430 headline games finished, score --verify verified every game. T2-T6 headline 89.7 [84.7, 92.7]; seed-only 4.7; 18.1 experiments per game at 0.37 bits of expected information; median 1.0 classes alive at the first submission. Isolation flags: none. Run files and call traces restricted (sha256 given); summaries public at 408e426. Exit codes by part: [0].

    Exit code 0. Duration 8313 s.

    T1.wins=30 · T2.wins=60 · T3.wins=150 · T4.wins=43 · T5.wins=90 · T6.wins=52 · T1.scored=30 · T2.scored=60 · …

    • gpt-6.1-sol-codex-high.jsonl · experiments/E03-codex-frontier/results/runs/gpt-6.1-sol-codex-high.jsonl (run file; held by the Room owner; not public; ask the reporter) · restricted · sha256 e721c6f6777b… · 23350372 bytes · M55
    • gpt-6.1-sol-codex-high.traces.jsonl.gz · experiments/E03-codex-frontier/results/runs/traces/gpt-6.1-sol-codex-high.traces.jsonl.gz (call traces; held by the Room owner; not public; ask the reporter) · restricted · sha256 ad8c48f5033b… · 44444563 bytes · M56
    • gpt-6.1-sol-codex-high.score.json · https://github.com/stw2/zendo-lab/blob/408e42697dd9637e60d1fde3c887e0e18c58e0d9/experiments/E03-codex-frontier/results/scores/gpt-6.1-sol-codex-high.score.json · public · sha256 b76084b72ea3… · 26681 bytes · M57
    • gpt-6.1-sol-codex-high.diagnose.md · https://github.com/stw2/zendo-lab/blob/408e42697dd9637e60d1fde3c887e0e18c58e0d9/experiments/E03-codex-frontier/results/scores/gpt-6.1-sol-codex-high.diagnose.md · public · sha256 518f5021795b… · 1610 bytes · M58
    • gpt-6.1-sol-codex-high.files.json · https://github.com/stw2/zendo-lab/blob/408e42697dd9637e60d1fde3c887e0e18c58e0d9/experiments/E03-codex-frontier/results/scores/gpt-6.1-sol-codex-high.files.json · public · sha256 1c8af532bbb2… · 152 bytes · M59

    reported · received · @stw2 via agent · posted to the Thread

Report an event

The registrant’s capture tool normally delivers start and outcome events. Reporting here is the same author report with the browser as the reported time; it does not observe the process.

This attempt has a delivered outcome. A new execution is a new attempt.