Issue #535: A refusal is tested as tests/NAME.nika with the compiler’s message in NAME.stderr, blessed like output tests (ADR-269 D22)

Open and answer this issue on GitHub.

tools, ready · opened 2026-10-07 · updated 2026-10-10 · 1 comment

Where this comes from

0.0.650 and 0.0.651 moved 375 tests of what a program computes from Rust into test blocks (tests/language, tests/build-time). The owner’s aim is to have as few Rust tests as possible. What stayed in Rust is mostly refusals: a program the compiler must reject, with a given code (NK1132, …) and a given help. In jumps.rs alone that is 11 of the 19 tests that remain. A test block cannot hold one: a refused program does not build, and in a package it would take every other test down with it.

What exists

• tests/errors/ holds one broken .nika per case and EXPECTED.txt with what each produces (errors.rs compares). That is a corpus of messages, not of codes per rule.
• ADR-269 D13 gives nikaia test output tests (tests/NAME.stdout) but nothing for a program that must be refused.
• Other languages: rustc’s UI tests (.rs + .stderr, //~ ERROR annotations), trybuild’s compile-fail tests, Go’s // ERROR "regexp" comments in test files.

The question

Whether nikaia test gets a refusal test, and in which form, for example:

• A: a package’s tests/NAME.nika with // refused: NK1132 lines (the code, optionally a fragment of the help) on the line that must carry the finding; nikaia test checks it and compares the findings. Each file is checked alone, nothing is compiled, so these are cheap.
• B: the same files, checked by a Rust runner in this repository only (no language feature).
• C: stay as Rust tests.

To be prepared before it is asked: how many of the remaining Rust tests are refusals of a single program, and how many also check the help’s wording (which an annotation would have to carry).

Comments

keywan-ghadami, 2026-10-09

Decided (owner, 2026-10-09): A, as a feature of nikaia test like trybuild, in the form of the output tests

Counted first (heuristic over crates/nikaia/tests):

• 1 998 Rust test functions in 202 files;
• 654 of them name an NK code, in 129 files, with 152 distinct codes;
• 373 of those 654 also check text (a help or message fragment).

So an expectation has to carry wording, not just a code.

The owner’s correction: the expectation is not an annotation in the source. It is a file, as NAME.stdout already is, like rustc’s UI tests (.rs + .stderr). The owner’s condition was that tests still run together. They do: a refusal never reaches rustc, so it costs a parse and a check, and the programs that build are compiled together as packages of one crate (#538, #549).

Where it is written: ADR-269 D22 and Part III 14.1 (0.0.713).

• tests/NAME.nika is a test program that may use the package.
• NAME.stdout and NAME.stderr say what it must produce. A refusal is the compiler’s whole message in NAME.stderr, with the path written tests/NAME.nika.
• What happened is compared with what is written. A refusal with no .stderr holding it fails.
• --bless writes both files.
• The package’s own output tests take a .stderr the same way.

To build

1. Discovery and run.
   • nikaia test finds tests/*.nika beside a package and checks each as a program that depends on the package.
   • A refused one is compared against NAME.stderr, with paths normalised.
   • The ones that build are compiled together (ADR-286 D38) and run as output tests (D13), with NAME.stderr compared where it exists.
   • A test program that the backend rejects is rebuilt alone, to name it.

   Tests:

   • a refused program with a matching .stderr passes;
   • the same program with one word changed in .stderr fails, with a line diff;
   • a refused program with no .stderr fails;
   • a program that builds while .stderr holds a refusal fails;
   • --bless writes the .stderr.
2. .stderr for the package’s own output tests. Test: a program that writes to standard error, with a matching .stderr.
3. tests/errors/ becomes a package of test programs, one .stderr each, replacing EXPECTED.txt and errors.rs.
4. Migration, in steps: the Rust tests that refuse one program (jumps.rs’s 11 first) move into tests/*.nika + .stderr. Each step removes the Rust test it replaces.

---

Generated by Claude Code

Back to Open Issues