Issue #549: Doc tests built once: each block a package of its own, all of them modules of one crate - measure first

tools, note · opened 2026-10-08 · updated 2026-10-08 · 0 comments

Where this comes from

Three places in this repository check Nikaia programs that live outside a package’s own tests. Each one carries its own loop:

crates/nikaia/tests/examples.rs runs nikaia test once per package in examples/.
crates/nikaia/tests/specification.rs, through specbook, takes every nika block of Part I-III and lowers it.
tests/guide/ (a Nikaia package) builds every nika block of guide/ as a project of its own and runs it with nikaia run.

nikaia test checks one package (Part III 13.2). It has no doc tests, and no way to take several packages in one call. Every caller therefore builds the loop itself, and every block or package is its own Cargo build. One package is one crate today (ADR-286 D19, project.rs cargo_workspace), so 14 guide blocks mean 14 rustc and link runs for 14 small programs.

The owner’s view is that compiling each block alone is the wrong default, because it costs time for no gain. There should be a way to build them together with the same isolation, and a switch to build them one by one.

The idea

The isolation is the language’s own: a package is a directory and a namespace (Part I 9.1). Two blocks that both declare main or Item cannot share a package. So each block becomes a package of its own: a directory, its own names, its own ledger. Blocks marked // file: src/X.nika join the package of the block that follows, as guide/syntax.md’s module example already writes.

The Rust side keeps those packages apart without a crate each. For a doc-test build, each package lowers to a mod of one crate instead of a crate of its own. Its main becomes an entry the test binary dispatches to. A Rust module is a namespace as separate as a crate for everything a package can declare, so the separation drawn by directories survives. What it saves is the per-crate rustc and link work.

At run time each block is still alone: one process and one empty directory per block, which is how nikaia test already runs test blocks and output tests (14.1).

Further rules:

A block that a page marks as refused (NK…) is settled by the checker and never reaches rustc.
Blocks whose build settings differ (user_parallelism, target, dependencies) form separate groups, one crate each.
Anything the lowering makes crate-global (crate attributes, extern symbols such as examples/c-library’s, assets) moves the block to a build of its own, automatically.
--separately builds every package as its own crate, as today. If the merged build fails in rustc, the blocks are rebuilt one by one to name the culprit; that failure is a C.1 defect in any case.

Prior art:

Rust 2024 merges doctests into one crate, one module each, runs each in its own process, and keeps standalone_crate as the opt-out.
Go compiles a package’s Example functions into its one test binary.
Python’s doctest shares one namespace per docstring; that is rejected here as a default, because a block must stay copyable on its own.

To measure (before anything is built)

Count, never time (callgrind, --branch-sim=yes), release builds:

Where one block’s build goes: the instructions in Nikaia’s front end (parse, check, ledger, lowering) against Cargo, rustc and the link, for the 14 projects of tests/guide/ run one by one as today.
A merged build by hand: the lowered Rust of the same 14 blocks, each wrapped in mod block_N with main renamed, plus a dispatching main, compiled as one crate with the same settings. The output of each block must match first, byte for byte.
The same comparison for examples/’ packages, to see whether “several packages, one crate” is worth having beyond doc tests.

If (1) shows the front end dominates, or (2) saves little, this stops here with the numbers. Otherwise it becomes a decision: where doc tests are declared (the manifest naming Markdown files, or a nikaia test argument), the block markers, and whether specbook and examples.rs move onto it.

Back to Open Issues