ADR-007: The Scannerless Grammar Protocol, and Two Ways to Bind a Hole

Status: Superseded by ADR-296 — its D1-D8 are D1-D8 there Related: ADR-001 D2 (the parser backend this rests on), ADR-009 (frames and folds over the same protocol), ADR-011 (the lowering as built), ADR-026 (what D3’s compile-time half still waits on)

1. Context

Nikaia hosts foreign syntax inside its own: dsl sql { … }, dsl x86 { … }, a grammar construct that is part of the language rather than a library. That one requirement decides the parser architecture, and it decides it before anything else is designed.

The parser backend was chosen independently (ADR-001 D2) and arrived at the same answer: winnow-grammar parses &str/&[u8] rather than a token stream. Two lines of reasoning reaching the same place is the strongest evidence available that it is right.

What the backend already provides, and what it does not:

Needed	Status
scannerless parsing, no global lexer	built — lexical (UPPERCASE) vs. syntactic (lowercase) rules replace the lexer/parser split
commit points, spelled =>	built — the cut operator
error recovery	built — recover(rule, sync)
zero-copy tokens	built, by two mechanisms — see D8
auto-generated AST types	not built — action blocks are required (D7)
regex terminals compiled to DFAs	not built — fixed built-in terminals instead (D7)

2. Decisions

D1 — Scannerless parsing is a language guarantee, not an implementation detail.

The core parser maintains no global token stream. A grammar consumes the raw character stream, and whitespace handling is a per-rule property.

A global lexer cannot tokenize embedded languages with conflicting delimiters: JSON’s } against a Nikaia block end, a Rust lifetime 'a against a character literal. Any host language that wants to embed foreign syntax has to give up the global lexer.

Consequence, accepted: Nikaia can never adopt a token-stream frontend without breaking dsl.

D2 — A DSL block ends explicitly, with } eod.

let query = dsl mysql {
    SELECT name FROM users WHERE age >= :target_age
} eod

Given D1 the core parser cannot find the end by counting braces — in foreign syntax a } may be a string character, an operator, or absent. It has two options: run the DSL parser to find the end, or scan for a marker. The marker also buys parallelism: having found the boundary, the core parser skips the body and continues with the rest of the file while the DSL parser works.

Accepted limitation. A body containing the literal } eod must be loaded with dsl G from "file" instead. That is better than a configurable or heuristic terminator, which would reintroduce exactly the ambiguity D1 removes.

D3 — dsl is an expression, with one syntax for compile time and runtime.

const CONFIG: Json::Value = dsl Json from "config.json"   // compile time; the build fails on error
let data = dsl Json from user_input                       // runtime; throws ParseError
let query = dsl mysql { … } eod                           // inline body

Which mode applies follows from the context — const or let — not from a different spelling. The predecessor promised dual-mode parsing and delivered two unrelated mechanisms, from … with … for constants and a generated Parser::parse() for runtime; a promise of two modes is only real if they share a syntax.

Built: the runtime half (Expr::DslFrom). The compile-time half is not, and const does not exist at item level, so the context this discriminates on has no representation yet. What has to be settled first — what a build may read, and what may run while it reads — is ADR-026, deliberately open.

D4 — A hole binds in one of two ways, and the grammar author says which.

• meta::capture(id) — immediate: resolved from the surrounding scope at compile time.
• meta::parameter(name, type) — deferred: a named runtime parameter.

One mechanism cannot serve both. Scope binding is correct for assembly, where $v = in(reg) val must mean this val, here. It is actively wrong for a prepared statement: baking values into the statement definition destroys the reuse that makes it worth having.

This is the same immediate/deferred axis the language already uses for lambda capture (@immediate/@detached) and for teardown (task::scope vs. parked cleanup, ADR-006 D3). Three independent tracks landed on it, so it is recorded as a language principle in Part II 10.5.

D5 — A DSL with deferred parameters generates a shadow type.

// dsl mysql { … :id … :active } eod  ==>  SqlAst[Signature = {id: i32, active: bool}]

impl SqlParser {
    pub fn execute(self; ...args: Self::dsl) -> Result[Row] { … }
}

stm.execute(; target_id: 501)

DSL parameters are configuration, so they are named arguments after the ; — the Subject ; Config protocol applies to metaprogramming without exception. The struct is monomorphised per DSL string and stack-allocated.

Without a generated type the compiler cannot check that every :name in the body is supplied at the call site, which is the entire safety argument for embedding SQL rather than concatenating strings.

Risk, stated. This is the least proven decision here: it requires synthesising a type from content produced by a user-defined parser, which nothing in the toolchain does today. Specified now so the protocol is coherent, and flagged as the piece most likely to need revision.

Built: the type, the check and the spread. A body with :name holes generates a struct carrying those names, one per distinct parameter list; the call site builds it from what stands after its ; and passes it by value; and ...args: Self::dsl is the parameter that receives it. A :name the call does not pass is NK1112 and a name the statement does not have is NK1113, both against the .nika line (ADR-012). crates/nikaia/src/dsl.rs holds the analysis and tests/dsl_parameters.rs compiles and runs the result.

Three things about how it is built follow from ADR-011 D2 rather than from this record, and are worth naming because the spelling above reads as though they were settled here:

• The field types come from the call site, not from the body. Nothing in … WHERE id = :id … says that :id is an i32; the sketch above writes {id: i32, active: bool} because meta::parameter(name, type) (D4) is where a grammar says so, and no grammar reaches this compiler. So the struct is generic in every field and monomorphised where it is built. An emitter that wrote i32 would be guessing.
• One struct per parameter list, not per DSL string. Two statements that take :id and :active share the type. Per-string identity needs the emitter to know which statement a call site’s receiver came from, and nothing observable turns on it: a struct literal names its fields, so the call site needs only the names it passes.
• Self::dsl also names the result, so a driver can hand the bound parameters back. Extending the spelling to the result position is a choice this record did not make; it is what a driver written in Nikaia today can do with the value at all.
• A hole is found by scanning, which the grammar was meant to do. a::b and 12:30 are not holes; SELECT ':id' is read as one, because knowing that it is inside a string means knowing where SQL’s strings begin. That is meta::parameter’s job in D4, and its consequence here is a refused call naming a parameter that is not really one — not a program that is silently wrong.

Not built: the driver’s own half. args.values() (Part II, 10.5) needs a value type for the parameters, which is meta::parameter(name, type) again, and SqlAst[Signature = {…}] is a spelling the parser has no rule for. So the type that carries the names exists and the library that reads them by name does not: a driver receives the struct and passes it on, which is exactly as far as a generic parameter reaches without a trait this record does not define.

D6 — Hardware instructions leave the language core.

unsafe asm with its register-constraint table and clobber is removed. Assembly is a library-provided DSL:

use std::backend::x86

dsl x86 { $v = in(reg) val
          $r = out(reg) result
          mov $r, $v } eod

Register constraints in the core couple the language to register machines. The targets WebAssembly, which is a stack machine with no registers to constrain. A core construct that cannot be given meaning on a first-class target is a defect in the core, not in the target. As DSLs, backend::x86 and backend::wasm coexist, each validating its own operands.

This is D4’s immediate capture: unlike SQL, the assembly DSL resolves names from the current scope and injects at the call site.

Consequence. unsafe loses its only specified use. It stays reserved for future FFI work rather than being removed from the grammar.

D7 — Auto-AST and regex terminals are goals, not behaviour.

• Auto-AST — generating structs for labelled sequences and enums for alternatives where no -> { … } is given. Desirable, since it removes most boilerplate from a grammar; the backend requires action blocks today.
• Regex terminals — the intent, efficient terminal matching directly on the byte stream, is met by the backend’s built-in terminals (ident, string, float, hex_digit1, …). The specific mechanism is not adopted.

Spec text must not promise either as available.

D8 — Zero-copy is two mechanisms, and the grammar picks per rule.

• Identifiers and keywords → interned Symbol. Short, highly repetitive, compared far more often than read. Interning deduplicates them and makes equality an integer comparison.
• Bulk text (string literals, comments, doc text, DSL bodies) → tethered slice. Long, rarely repeated, rarely compared. Interning would waste arena space; a tether costs one shared handle for the whole buffer (ADR-005 D4, ADR-008 D4).

They are complementary, not competing. The guarantee is stated positively — the source buffer cannot be freed while a token points into it — and why the negative form is wrong is ADR-008 §4.

3. Rejected alternatives

• A configurable or inferred DSL terminator — reintroduces the delimiter ambiguity scannerless parsing exists to remove (D2).
• Keeping unsafe asm in the core with a WASM special case — the special case would be a second, register-free syntax, which is a DSL with extra steps plus a permanent fork in the core (D6).
• One binding mechanism instead of two — see D4’s prepared-statement argument.

4. Consequences

Part II Chapter 10 is the scannerless protocol; Part III Chapter 16 documents the backend DSL pattern rather than a core construct; Chapters 15 and 17 adopt } eod and parameter holes.

Two changes were made upstream in winnow-grammar and are in use: a fix to the interner’s handle round-trip, without which interned identifiers were unusable; and fold(rule, init, step), a repetition that threads an accumulator instead of collecting, which is what lets a 13 GB input parse in constant space (ADR-009 D2).

Still to raise upstream: auto-AST, and error recovery that accumulates diagnostics instead of discarding the skipped input.

This site is open source. Improve this page.