span(p), here) and Whether a Line Break Precedes It (same_line); /// Is a Rule, and No Grammar Declares a State or an extern ruleStatus: Accepted
Date: October 10, 2026 (#557)
Answers: what the grammar language needs so that the compiler’s own grammar
(parser/mod.rs, 318 rules) can be written in it
(ADR-294, #125)
Related:
ADR-135 D3 (a /// run belongs to the item after it),
ADR-307 D4 (a [ or ( after a line break does not continue the
expression before it),
ADR-285 (u32),
Part II 10.8 (the grammar’s vocabulary)
Which elements does the grammar language add, if any, so that the compiler’s grammar needs nothing the Nikaia language cannot say?
The compiler’s grammar is written in Rust with the same macro the Nikaia
grammar language lowers to. It uses three things Part II 10.8 does not offer:
the byte range a rule matched (@=, 52 rules, 78 uses), a declared parser
state (state Trivia, which records whether the whitespace just skipped held
a newline and collects /// doc comments), and hand-written rules
(extern rule). An element not on the page of 10.8 is not in the language.
| Language / tool | Its answer | Source |
|---|---|---|
| Go | the scanner inserts a semicolon after a line’s last token, so a newline is a token the grammar sees | from memory, not checked |
| JavaScript | automatic semicolon insertion looks at whether a line terminator precedes a token | from memory, not checked |
| Kotlin, Swift | a newline ends a statement unless the next token cannot begin one; the parser asks whether a newline precedes a token | from memory, not checked |
Rust (rustc_parse) |
doc comments are tokens, attached to the item that follows; one before nothing is an error | from memory, not checked |
| PEG tools (pest, nom, winnow) | a user state or a custom parser function; a span from the position before and after | from memory, not checked |
| Question | Option | Why it lost / won |
|---|---|---|
| the span of a match | a rule-level @= as in Rust |
lost: a new rule header form for what an element says |
s:span(p) and here |
won: the form of text(p), which is on the page already |
|
| a line break before a token | a user-declared parser state | lost: the type needs Default, which declared types do not derive, and each author carries the by-position idempotence that backtracking asks of state |
a built-in same_line |
won: the generated parser already runs WS and knows what it skipped |
|
/// |
a doc buffer in the parser’s state | lost: it is the state above |
| a rule | won: /// is text a rule matches |
|
| a rule only Rust can write | extern rule |
lost: the number rules are already Nikaia, a lexer rule plus an action that may fail (10.8) does the same |
span(p) yields the byte range p matched; here yields a position and consumes nothingspan(p) is p with the Span it covered as its value, as text(p) has the
text. here is the position as a u32. A Span is two u32 (ADR-294) and
is offered by std::text.
same_line matches nothing and fails where the whitespace skipped just before it held a line breakThe generated parser records, by position, what its last WS skip held, so the
answer is a fact about the input and survives backtracking: a branch that fails
has written what a branch that succeeds would have.
extern ruleA rule reads its input with the elements of 10.8; a number or any other
input that needs a decision is a lexical rule whose action returns a value or
fails with fail("…").
/// is a ruleA grammar that has doc comments writes a doc rule ("///" until(line_ending),
one or more) and uses it where docs may stand. Its WS skips // and not
///. The compiler’s grammar refuses a /// where no item, field or variant
follows, as rustc does, and it no longer ignores one.
/// is a parse error.span(p) is useful to any grammar that builds a tree with positions, and
same_line to any that is newline-sensitive.Span is declared in std::text and how tools/ast.nika’s uses move
to it.