Status: Accepted Related: ADR-004 (the lowering D4’s transpiler is), ADR-296 (builds the grammar protocol on D2), ADR-021 D2 (records the toolchain D1 names), issue #579 (D5’s first tool, the Rust-to-Nikaia converter)
Nikaia compiles through the Rust toolchain and hosts foreign syntax inside its
own (dsl, grammar). Both choices constrain the toolchain before any language
feature is designed: the first decides which compiler a release is tied to,
the second rules out any parser that starts from Rust’s own lexer.
rust-toolchain.toml names the channel and its components; it is the single
source of truth and the channel is never repeated elsewhere.
Stable is affordable because nothing here asks for more: no -Z flag is passed
in any build, no crate in the workspace declares a #![feature(…)], and the
Rust that Stage 0 emits uses no unstable feature either. So a clone builds,
tests and lints on the toolchain the file names, and that is the one leg CI
requires to be green.
What a toolchain costs a user: a stable toolchain with rustfmt and clippy
installs in about 12 s and weighs about 600 MiB, and the compiler it builds
links libc/libgcc and nothing else, so an installation is an ordinary one.
winnow-grammar compiles an EBNF-like grammar into winnow combinators at
compile time, and parses &str/&[u8] directly.
The alternative was syn-grammar, which parses a Rust TokenStream and so
inherits Rust’s lexer. That is disqualifying three times over: syntax that is
not a valid Rust token sequence cannot be expressed at all; an embedded SQL or
assembly block would have to survive Rust tokenization before the grammar saw
it, which makes hosting a foreign DSL impossible; and trivia is discarded before
the grammar runs, which rules out a formatter or an LSP.
Parsing raw text makes the toolchain scannerless. ADR-296 promotes that from an implementation detail to a language guarantee.
The features that guarantee rests on:
| Feature | What it carries |
|---|---|
&str/&[u8] input, lexical vs. syntactic rules |
scannerless parsing; whitespace control without a separate lexer |
cut operator (=>) |
commit points — an error instead of silent backtracking |
span binding (@) over LocatingSlice |
source positions, and so ADR-300’s diagnostics |
fold(rule, init, step) |
an entry rule that reduces instead of collecting, so a large input parses in constant space (ADR-296) |
string interner (Symbol) |
identifiers with cheap equality and one copy per spelling |
its own ParseError with a rule stack |
the diagnostics contract, Part III Appendix C |
recover(rule, sync) |
fault-tolerant parsing for tooling |
Not provided, and deferred in ADR-296 D7: auto-generated AST types and regex terminals.
winnow-grammar is pinned by git revision, and moves together with winnow.Pinned in Cargo.lock, not by semver. The pin and the workspace winnow
version are advanced in one step: the crate tracks winnow closely, and its
diagnostics engine replaced winnow::error::ContextError with its own
ParseError — a breaking change for anything that matches on error text.
Stage 2 must handle the standard library and user code. That Stage 0 emits source rather than machine code is relied on by ADR-296, ADR-300 and ADR-014, and is ADR-004 D1’s single lowering.
D1 holds for tools too. A tool of this repository that needs what only a Rust
front end knows - which method a call resolves to, the type of a local, the
width of an intermediate integer - asks rust-analyzer, a component of the
stable toolchain (rustup component add rust-analyzer), through its index dump
(rust-analyzer lsif). The part that reads the dump and acts on it is written
in Nikaia, as every helper tool is. Where rust-analyzer cannot tell a type, the
tool says so at that place; it never guesses.
The reason is that the answer has to be exact and cheap to obtain: rust-analyzer resolves names, macros and traits as an IDE does, on the toolchain the repository already names, as one relocatable binary of about 45 MiB.
Rejected: a pinned nightly with rustc_driver for the tool (it would be exact,
but every machine that runs the tool would need the nightly and rustc-dev,
about 1.5 GiB, and the API is unstable; the withdrawal of 0.0.9 was about the
compiler, not about tools, so this was a cost, not a rule); a Rust front end of
our own, macros and constants included (a parser sees spelling, not types, and
types need name resolution, inference and trait resolution, which is rustc’s
front end again); syn (spelling only, and ADR-290 D13).
A release is a toolchain, not a crate: the compiler, the toolchain it is built and tested against, and the pinned parser backend are one artifact. Advancing any of them is a release-sized change, and ADR-021 makes that observable by hashing the toolchain into the build-cache key.