Issue #794: Long jobs: CI is the verifier, measuring only where the point is a number

Open and answer this issue on GitHub.

decision · opened 2026-10-24 · updated 2026-10-24 · 0 comments

How it came up

The owner proposed (2026-10-24) a dedicated agent or service that runs the long jobs (callgrind, lower-std, cargo builds, the full suite, the oracle) once for all builders, batched, and hands back counts. Trigger: the #667 builder spent 25+ minutes in one callgrind run of the whole lower-std. The architect checked the jobs; basis in docs/long-jobs.md.

What was checked

• CI on develop already runs the full suite: 8 min tests + 1.5 min sweep in bootstrap_compiler, about 17 min the whole run, arm64 in parallel, off this machine. Locally the same suite is about 45 min on the 4 shared cores.
• This VM has no hardware counters (/sys/bus/event_source/devices has no cpu, no perf): callgrind is the only instruction counter, 50-100x slower than the program. One small file (lower overlaps.nika, 42.6 M Ir) takes seconds; the whole lower-std (~40 s natively) over 25 min.
• In stage 1 one builder hands in one change at a time: there is nothing to batch, and a local verifier would compete for the same cores, target/ lock and disk.
• Nobody reads CI: develop has been red since 9f090392 (U-d2): self-hosted.svg out of date (scripts/badges.py --check) and an arm64 Test failure (log not readable from here).

Recommendation (for the owner)

1. No measuring agent and no queue now. CI is the verifier; the main session watches each push with one background gh run watch <id> --exit-status, and a red run becomes the next order. (Written into docs/architect.md.)
2. Builders never run the full suite, only the touched tests. (Work order Done when.)
3. Measuring only in a task whose point is a number, named in a new Measure line of the work order (default none); a work count the tool prints first (files re-lowered of 213, codegen units re-optimised), callgrind only on one file, never on a whole lower-std or a test run.
4. Shorten the jobs: #667 per-file keys, #667 step 4, #793 step 3 (one test binary), later nextest in CI.
5. A shared measuring tool in Nikaia (tools/*.nika behind nikaia) only if stage 2/3 shows two builders queueing on the machine.

To build

1. CLAUDE.md and .claude/agents/builder.md (not edited by the architect, rule files): never run the full suite locally; CI runs it; measure only when the order’s Measure line says so; callgrind only on one file or small program. Replace “A full test run takes about 45 minutes” with that rule.
2. The main session: repair red develop (run scripts/badges.py; read the arm64 log) before the next step.
3. #667: its measure is the reused/re-lowered file count lower-std prints, not a callgrind of the whole run.
4. After #667 and #793 step 3: re-observe the table in docs/long-jobs.md and replace it.

Back to Open Issues