Issue #821: All tests in Nikaia: build ADR-269 D22 (tests/NAME.nika + .stdout/.stderr), then move the Rust suite leaves first

Open and answer this issue on GitHub.

ready · opened 2026-10-24 · updated 2026-11-03 · 0 comments

See the To build list at the end. (Full text follows.)

Where this comes from

Owner, 2026-10-24: all tests are to be written in Nikaia, as soon as possible. On that day 74 new #[test] were added, 58 in crates/nikaia/tests/it/selfhosting_defects.rs. Architect session; decision basis plus build order.

From zero

The compiler’s tests are Rust: crates/nikaia/tests/it/*.rs, 269 files. Each embeds a Nikaia program as a string, writes it to a scratch directory, runs nikaia lower, compiles with rustc, runs the binary and compares output (e.g. printed("view-of-a-call", r#"..."#) then assert_eq!).

Nikaia already has its own test form (Part III 14.1): test "name" { assert(..) } blocks; output tests tests/NAME.stdout / .out/ / .in/ / .args / .stdin with --bless and --both-settings; and test programs tests/NAME.nika with NAME.stdout, and NAME.stderr holding the compiler’s whole refusal where it must be refused (ADR-269 D22, decided 2026-10-09, #535). The form is specified; the question is what is built.

Facts, checked (develop, target/fast/nikaia)

Probe package: a test block, tests/main.stdout, tests/ok.nika + ok.stdout, tests/bad.nika (type error) + empty bad.stderr.

• test blocks work (assert 1 + 1 == 2 is NK1195; assert(..) is the form).
• Output tests of the package program work.
• D22 is not built. ok.stdout was compared against the package’s main (printed hi); bad.nika/bad.stderr is not discovered (3 tests, not 4); --bless writes nothing. Cause: project::output_tests (crates/nikaia/src/project.rs) collects .stdout/.out stems only.
• Rust tests (file counts, overlapping): 220 inspect text with .contains, 157 name an NK code, 145 use a scratch dir, 218 call the compiler’s Rust API in-process; 45 only drive the binary.

How others do it

Rust: #[test] plus tests/ crates; rustc’s UI suite is .rs + .stderr golden files + --bless (the model D22 copies). Go: _test.go, testdata/ golden files, testscript txtar for driving the tool. Zig: test "x" {} blocks; compiler cases are test/cases/*.zig with the expected error/output in a trailing comment. Elixir: ExUnit test "..." do, doctests, assert_raise. Common: an expected refusal is a golden file or annotation beside the source; in-process unit tests stay beside their code.

Where each kind goes

1. Program -> output (selfhosting_defects, run/lower families): tests/NAME.nika + NAME.stdout. Needs D22.
2. Program -> refusal with an NK code: tests/NAME.nika + NAME.stderr. Stricter than .contains("NK1234"); matches Part III C (every NK code has a reproduction test).
3. In-process API tests: test blocks inside tools/*.nika, moving with the code (#125). Rust-only internals stay Rust until they move.
4. Inspecting lowered Rust text (printed_and_lowered): not about meaning. Not decided here; proposed: these few stay Rust until a case needs a Nikaia form.

Gaps (findings, fixed in the compiler)

• G1: D22 discovery/running/bless, programs compiled together (ADR-286 D38).
• G2: a Nikaia test package for the compiler, crates/nikaia/tests/nika/, run by one it:: shim test, so cargo test and CI see it with no workflow change.
• G3: test blocks in tools/*.nika are run by nothing yet.
• G4: cost: each Rust test spawns its own rustc; D22’s batch compile is the gain. Measured as rustc invocations, not time.
• Not needed for kinds 1-2: process spawning or scratch dirs inside Nikaia (the runner does it).

Recommendation

Build D22 now; from then every new program-level test is a tests/NAME.nika. Move selfhosting_defects.rs first (newest, pure program->stdout), then NK families leaves first, by a converter written in Nikaia. Risks: golden stderr churns on message changes (--bless, reviewed as a diff); a batch compile can hide which program broke the backend (D22: rebuilt alone).

To build

1. D22 in nikaia test: project::output_tests discovers tests/NAME.nika; a refused program is only checked and its message (path tests/NAME.nika) compared with NAME.stderr; one that builds is compiled as its own package of the test build, its stdout/stderr compared; --bless writes both. Test: an it:: test over a fixture package with one ok, one refused, one wrongly refused, one wrongly built program.
2. Compiler’s Nikaia test package crates/nikaia/tests/nika/ plus it::nika_suite running nikaia test there; seed with 5 cases moved from selfhosting_defects.rs (Rust twins deleted). Rule into CLAUDE.md: a new program-level test is a tests/NAME.nika.
3. Converter (Nikaia, nikaia run): from an it/*.rs, write NAME.nika + empty expectation per printed(..)/refusal-shaped test, then --bless; report what it could not convert. Move the rest of selfhosting_defects.rs.
4. NK refusal families, leaves first, one file per commit, deleting each Rust test as its twin passes; report rustc calls before/after.
5. G3: run test blocks of tools/*.nika in CI; in-process Rust tests then move with their code under #125.

---

Generated by Claude Code

Back to Open Issues