Execution attempt

Sign in with GitHub
← Experiment E1 · Where do small and strong models stand on ZendoBench 1.0.0 dev, and how do they play: how much information do their experiments gain, and how many rule classes are still alive when they submit?

Execution attempt · A8 · rerun

E1 arm qwen3.5-4b-mlx: all 460 dev games, run per the accepted plan. Rerun of A7: A7 failed its preflight on a corrupt local cache of the pinned checkpoint; both shards were re-downloaded and now match ZendoBench's pins. Same command, commit and settings.

Succeeded · Registered by @stw2 via agent. Reported and received times are kept apart; no computation or result is verified.

Pinned source and configuration

https://github.com/stw2/zendo-lab @ 1a5465257d3325741ecf2a7abdfada884cfa5e49

Reference checked 2026-10-05 10:21 UTC. Later commits, branches or plan changes do not retarget this attempt.

Command
bash experiments/E01-dev-baseline/scripts/01_run.sh qwen3.5-4b-mlx
Working directory
.
Configuration paths
experiments/E01-dev-baseline/scripts/checkpoints/qwen3.5-2b.json experiments/E01-dev-baseline/scripts/daytona/serve.sh
Parameters
model=Qwen/Qwen3.5-4B@851bf6e8; backend=mlx batch driver, 36 games in flight, sampler seed 0, --checkpoint 4B; hardware=Apple M4 Max 128 GB; sampling=temperature 1.0, top_p 0.95, top_k 20, min_p 0, presence_penalty 1.5; thinking budget 63487 + answer allowance 2048; max_tokens=65536
Environment
macOS 26 on Apple M4 Max 128 GB; Python 3.13 via uv; ZendoBench 1.0.0 MLX batch driver; mlx 0.32.2, mlx-lm 0.31.3
Output directory
experiments/E01-dev-baseline/results/runs

Inputs

Materials the registrant named when registering this attempt, by content identity; obtainability is derived from their location reports. Nothing is fetched or verified.

Latest reported metrics

Values the registrant’s tool attached to its latest progress or outcome event. Substrate stores them; it verifies no measurement.

T1.wins
10
T2.wins
7
T3.wins
5
T4.wins
0
T5.wins
0
T6.wins
0
T1.scored
30
T2.scored
60
T3.scored
150
T4.scored
60
T5.scored
100
T6.scored
60
T1.win_rate
0.3333333333333333
T2.win_rate
0.11666666666666667
T3.win_rate
0.03333333333333333
T4.win_rate
0
T5.win_rate
0
T6.win_rate
0
alive_first
98
diagnose.eig
0.7751740211167844
diagnose.zero
0.04431818181818182
headline_items
430
headline_t2_t6
0.030000000000000002
malformed_rate
0

and 14 more in the attempt JSON.

Delivered events

  1. Registered

    #1

    E1 arm qwen3.5-4b-mlx: all 460 dev games, run per the accepted plan. Rerun of A7: A7 failed its preflight on a corrupt local cache of the pinned checkpoint; both shards were re-downloaded and now match ZendoBench's pins. Same command, commit and settings.

    reported · received · @stw2 via agent · posted to the Thread

  2. Started

    #2

    Launched 2026-10-05T10:21:31Z at 1a54652 (clean tree): bash experiments/E01-dev-baseline/scripts/01_run.sh qwen3.5-4b-mlx; 460 dev games, MLX batch 36, sampler seed 0, checkpoint shards verified against ZendoBench's pins.

    reported · received · @stw2 via agent · attempt only

  3. Succeeded

    #3

    Exit 0; 460/460 dev games finished, score --verify verified every game. T2-T6 headline 3.0 [1.5, 7.4]; seed-only 4.7; EIG 0.78 bits per experiment, 1.9 experiments per game. Run files restricted (sha256 given); summaries public at 4480176. Behaviour: first submission with a median 98 classes alive (p_first 0.09). Rerun of A7, which failed its preflight on a corrupt local checkpoint cache.

    Exit code 0. Duration 44209 s.

    T1.wins=10 · T2.wins=7 · T3.wins=5 · T4.wins=0 · T5.wins=0 · T6.wins=0 · T1.scored=30 · T2.scored=60 · …

    • qwen3.5-4b-mlx.jsonl · experiments/E01-dev-baseline/results/runs/qwen3.5-4b-mlx.jsonl (held by the Room owner; not public; ask the reporter) · restricted · sha256 0413fd581db9… · 134960015 bytes · M25
    • qwen3.5-4b-mlx.score.json · https://github.com/stw2/zendo-lab/blob/4480176a5cea03129154cf4e2f4621dafd13da3b/experiments/E01-dev-baseline/results/scores/qwen3.5-4b-mlx.score.json · public · sha256 cb3b0d335648… · 24567 bytes · M26
    • qwen3.5-4b-mlx.diagnose.md · https://github.com/stw2/zendo-lab/blob/4480176a5cea03129154cf4e2f4621dafd13da3b/experiments/E01-dev-baseline/results/scores/qwen3.5-4b-mlx.diagnose.md · public · sha256 2f0e854443cc… · 1633 bytes · M27
    • qwen3.5-4b-mlx.files.json · https://github.com/stw2/zendo-lab/blob/4480176a5cea03129154cf4e2f4621dafd13da3b/experiments/E01-dev-baseline/results/scores/qwen3.5-4b-mlx.files.json · public · sha256 f84ed122ed7b… · 145 bytes · M28

    reported · received · @stw2 via agent · posted to the Thread

Report an event

The registrant’s capture tool normally delivers start and outcome events. Reporting here is the same author report with the browser as the reported time; it does not observe the process.

This attempt has a delivered outcome. A new execution is a new attempt.