Status: Accepted
Date: September 28, 2026
Extends: ADR-269 D13 (an output test is three files and no code)
Answers: the owner’s question whether a program whose result is a file it
writes needs a command of its own, and how other languages and test tools do
it
Related: ADR-108 (a path goes under a Root), Part III 14.1
ADR-269 D13 makes an output test three files: NAME.stdout, and beside it
NAME.stdin and NAME.args. That compares standard output only, and some
programs - the examples tally, report and inventory/ - produce a file.
The owner asked, before anything was built, whether this belongs in the language at all, and what other languages and test tools do. The comparison:
| layer | what it holds | in other languages |
|---|---|---|
| the language | test blocks, assert |
Zig test "…" {}, D unittest, Go func TestX, Rust #[test] |
| the standard library | a fresh directory per test, cleaned up | Go t.TempDir(), Zig std.testing.tmpDir(), pytest tmp_path, ExUnit @tag :tmp_dir |
| the test tool | run a program, compare its output and the files it wrote with expectations | Go’s testscript (how the go command tests itself), Rust’s trycmd, rustc’s UI tests, LLVM’s lit |
No language has syntax for it. The tools that do it agree on four things:
trycmd
copies a test’s .in/ directory in as the working directory and compares
its .out/ directory afterwards; testscript holds both in one archive.rustc --bless,
cargo insta review, jest -u and Go’s -update convention write what the
program did as the new expectation, and the person reviews the change in
version control. An expectation written by hand is a chore for every
intended change, and the chore is where mistakes come from.And they share one piece of advice about programs: compute the content in a function and write it in one call, so that most tests check the function and a few check the file.
What this language already has is the other half of point 1. A path goes
under an fs::Root (ADR-108), so a function that writes takes
the directory it writes under. A test can hand it a directory of the test’s
own - if std gives it one.
Against the four, an output test that runs in the package’s directory
writes into the source tree, keeps its input files loose in tests/, has its
expectations written by hand, and prints both texts in full on a failure.
nikaia test creates a fresh, empty directory for each output test at each
setting, runs the program with it as the working directory, and removes it
afterwards. Nothing the program writes reaches the package. The same holds
for a test block: it runs in the package’s directory, as Go’s tests do, so
it can read what the package holds, and D5 is how it writes.
NAME.in/, and what must come out is NAME.out/.| file | is |
|---|---|
NAME.stdout |
what the program must print, byte for byte |
NAME.out/ |
files the program must have written, byte for byte, each under the name it has in the working directory |
NAME.in/ |
optional: copied into the working directory before the run |
NAME.stdin |
optional: standard input |
NAME.args |
optional: the arguments, one per line |
A test is named by a NAME.stdout or a NAME.out/, and has at least one of
the two. Standard output is compared where NAME.stdout exists. A file under
NAME.out/ must exist with the same bytes; a file the program writes that
NAME.out/ does not name is not compared - a test says what it is about. The
program must end successfully, as ADR-269 D13 says.
An input file is NAME.in/FILE, and the arguments name it as FILE, never as
tests/FILE.
nikaia test --bless writes the expectations from what the program did.For each output test, --bless writes what the program printed into
NAME.stdout (where the test has one) and each file it wrote into the
NAME.out/ entry of the same name. It blesses only a run that ended
successfully, and at --both-settings only one whose two runs agree - an
outcome that differs between the settings is a fault to report, never an
expectation. A test block has no expectation to bless: its claims are
written in its body. The review is the difference in version control.
A new output test is started by creating the empty NAME.stdout, or the
NAME.out/ entries it should compare, and blessing.
Where standard output or a written file differs from its expectation, the
report is a line difference - - the expected line, + the one the program
produced, and two lines of context - rather than both texts. A file the
program did not write is reported as not written.
fs::scratch() is a fresh directory as a Root.test "the page is written where it is asked for" {
let root = fs::scratch()
publish(entries, "page.html", root)
let page = fs::read_to_string("page.html", root)
assert(page.contains("<table>"))
}fs::scratch() -> fs::Root throws creates an empty directory and hands it
back as fs::Root::Dir, so what a test writes is checked to stay inside it
(ADR-108 D3). Under nikaia test it is created inside the
test’s own directory, which is removed when the test’s process ends. Outside a
test it is a new directory under the system’s temporary directory, and removing
it is the operating system’s business, as it is for any temporary file.
Placeholders for what differs between runs - a timestamp, a path, a
process id. trycmd writes [..], insta redacts; a program that prints the
time cannot have an output test. Not decided here.
A file system in memory. Go’s fstest.MapFS tests a reader without a
disk. A Root is a directory; D5 is the disk-backed half and is what the
examples need.
Syntax in the language. No language has it, and nothing about a file comparison needs the compiler: it is the test tool’s, as ADR-269 D13 puts it.
NAME.files/ beside NAME.stdout. It says what must
come out and not what goes in, and it kept inputs as loose files in tests/,
reachable only because the test ran in the package’s directory - which D1
removes.
Expectations written by hand only. Every intended change of output becomes editing files to match a run; that is what D3 replaces, and the reason every tool compared has a bless mode.
NAME.in/ and
NAME.out/, and tests/examples.rs holds no expected output of its own.NAME.in/, not loose in tests/.std.contracts carries fs::scratch.