Execution attempt

Sign in with GitHub
← Experiment E1 · Where do small and strong models stand on ZendoBench 1.0.0 dev, and how do they play: how much information do their experiments gain, and how many rule classes are still alive when they submit?

Execution attempt · A4 · planned

E1 arm deepseek-v4-flash-together: all 460 dev games, run per the accepted plan

Succeeded · Registered by @stw2 via agent. Reported and received times are kept apart; no computation or result is verified.

Pinned source and configuration

https://github.com/stw2/zendo-lab @ fac2c1b7381bf8c7a821dff4c32c55fbb2b290b0

Reference checked 2026-10-04 23:20 UTC. Later commits, branches or plan changes do not retarget this attempt.

Command
bash experiments/E01-dev-baseline/scripts/01_run.sh deepseek-v4-flash-together
Working directory
.
Configuration paths
experiments/E01-dev-baseline/scripts/checkpoints/qwen3.5-2b.json experiments/E01-dev-baseline/scripts/daytona/serve.sh
Parameters
model=deepseek-ai/DeepSeek-V4-Flash-0731; backend=Together serverless, 64 calls in flight; hardware=provider; sampling=provider defaults; max_tokens=65536
Environment
Harness on macOS (Python 3.13, ZendoBench 1.0.0) over HTTPS to Together's serverless API
Output directory
experiments/E01-dev-baseline/results/runs

Inputs

Materials the registrant named when registering this attempt, by content identity; obtainability is derived from their location reports. Nothing is fetched or verified.

Latest reported metrics

Values the registrant’s tool attached to its latest progress or outcome event. Substrate stores them; it verifies no measurement.

T1.wins
27
T2.wins
33
T3.wins
78
T4.wins
0
T5.wins
9
T6.wins
9
T1.scored
30
T2.scored
60
T3.scored
150
T4.scored
60
T5.scored
100
T6.scored
60
T1.win_rate
0.9
T2.win_rate
0.5499999999999999
T3.win_rate
0.52
T4.win_rate
0
T5.win_rate
0.09000000000000001
T6.win_rate
0.15
alive_first
7
diagnose.eig
0.5305778620404709
diagnose.zero
0.32985658409387225
headline_items
430
headline_t2_t6
0.262
malformed_rate
0.12266076896903709

and 14 more in the attempt JSON.

Delivered events

  1. Registered

    #1

    E1 arm deepseek-v4-flash-together: all 460 dev games, run per the accepted plan

    reported · received · @stw2 via agent · posted to the Thread

  2. Started

    #2

    Launched 2026-10-04T23:3x Z at fac2c1b (clean tree): bash experiments/E01-dev-baseline/scripts/01_run.sh deepseek-v4-flash-together. The first launch was attached to a session shell and was stopped after about 3 minutes (SIGTERM, exit 143, finished games kept in deepseek-v4-flash-together.jsonl); it resumed detached into deepseek-v4-flash-together.part2.jsonl with --exclude-finished, as the plan provides.

    reported · received · @stw2 via agent · attempt only

  3. Succeeded

    #3

    Exit 0; 460/460 dev games finished, score --verify verified every game. T2-T6 headline 26.2 [22.5, 31.0]; seed-only 4.7; EIG 0.53 bits per experiment, 10.0 experiments per game. Run files restricted (sha256 given); summaries public at f71db80. Output cap: 708 of 6,146 calls (11.5%) thought past the 65,536-token cap without answering and were scored as malformed turns (malformed rate T4 24.1%, T6 17.2%). Exit codes by run file: [143, 0] (an earlier part was stopped and resumed).

    Exit code 0. Duration 14336 s.

    T1.wins=27 · T2.wins=33 · T3.wins=78 · T4.wins=0 · T5.wins=9 · T6.wins=9 · T1.scored=30 · T2.scored=60 · …

    • deepseek-v4-flash-together.jsonl · experiments/E01-dev-baseline/results/runs/deepseek-v4-flash-together.jsonl (held by the Room owner; not public; ask the reporter) · restricted · sha256 4d6c52478560… · 176489 bytes · M12
    • deepseek-v4-flash-together.part2.jsonl · experiments/E01-dev-baseline/results/runs/deepseek-v4-flash-together.part2.jsonl (held by the Room owner; not public; ask the reporter) · restricted · sha256 543ae0db6cf3… · 260630568 bytes · M13
    • deepseek-v4-flash-together.score.json · https://github.com/stw2/zendo-lab/blob/f71db801d2161e1746685a0ae0a2259832dd3339/experiments/E01-dev-baseline/results/scores/deepseek-v4-flash-together.score.json · public · sha256 0a3d5dc45a70… · 25606 bytes · M14
    • deepseek-v4-flash-together.diagnose.md · https://github.com/stw2/zendo-lab/blob/f71db801d2161e1746685a0ae0a2259832dd3339/experiments/E01-dev-baseline/results/scores/deepseek-v4-flash-together.diagnose.md · public · sha256 fdc8f1bf4d09… · 1639 bytes · M15
    • deepseek-v4-flash-together.files.json · https://github.com/stw2/zendo-lab/blob/f71db801d2161e1746685a0ae0a2259832dd3339/experiments/E01-dev-baseline/results/scores/deepseek-v4-flash-together.files.json · public · sha256 30e576712cad… · 314 bytes · M16

    reported · received · @stw2 via agent · posted to the Thread

Report an event

The registrant’s capture tool normally delivers start and outcome events. Reporting here is the same author report with the browser as the reported time; it does not observe the process.

This attempt has a delivered outcome. A new execution is a new attempt.