Bytes as Readily as Text; What Becomes Text Is Checked Where It Becomes TextStatus: Accepted
Date: October 18, 2026
Answers: whether a grammar can parse input that is not text, and what a
view cut from such input is
Related: ADR-016 D1 (a view of a mapping is text because the
mapping was checked), §4 (fusing the check into the parse, deferred);
winnow-grammar ADR 25 (input bytes;, the backend this lowers to);
Part I 2.6 (text and Bytes), Part II 10.7, 10.8
grammar Record {
entry rule record -> Entry = len:dec[i64](digit+) ":" body:bytes(any*) { … }
}
let raw = fs::read("data.bin", root) // Bytes
let e = Record::record(raw) // refused today: an entry takes textAn entry takes text, so a format that is not UTF-8 - a binary record, a
length-prefixed field, a file whose encoding the grammar decides - cannot be
described. A program has to prove the whole input to be text before the first
rule runs, and the one place that may do so (fs::read_to_string, fs::map)
refuses the file instead.
| Option | Why it lost | See |
|---|---|---|
A grammar declares its input (input bytes in the grammar) |
the same grammar is wanted over both; the argument already says which it is | D1 |
A Bytes input is checked whole, then parsed as text |
it is what a text input already is, and it still refuses a binary format | D2 |
A view cut from Bytes is ref String unchecked |
a ref String that is not UTF-8 is undefined behaviour in code written safely (ADR-016 D1) |
D2 |
A separate bytes_grammar |
two vocabularies for a difference of one input type | D1 |
String or a Bytes; the argument decides.The same grammar is parsed over either. Record::record(raw) with a Bytes
parses its bytes; with a String it parses the text. Nothing in the grammar
says which, and a grammar called with both is compiled for both. Every element
of 10.8 means the same over both inputs; the differences are D2 and D3.
Bytes is checked as UTF-8 where it becomes text.Over a Bytes input, an element that yields text (until, text, string,
raw_ident, intern, ident) checks the bytes it covered,
and one that yields a scalar (any, char) checks what it decodes. What is
matched by comparison - a literal, digit, hex_digit, the whitespace, a
frame boundary, dec[T] - needs no check: no byte it accepts can be anything
but what was compared. A view that passed the check is a view of the input, as
over text; nothing is copied.
Over a String the same elements check nothing, because the text was already
checked when it was made.
The parse stops there. No enclosing alternative is retried, because none that
reads the same bytes as text can succeed, and the error must name the byte
rather than the last alternative tried. The entry throws ParseError as for
any other failure, with the message invalid UTF-8 at byte N: 0xXX (the byte in hex), where N counts
from the start of the whole input, in pieces of a par_fold too.
bytes(p) hands back the bytes p covered.A view of the input’s bytes, as b[a..<b] is (Part I 2.6), unchecked, over
either input. It is how a grammar keeps a field that is not text.
Over a Bytes, text that an element yields and the program uses only to look
up or insert a key of a map of text is checked as UTF-8 only where the map
does not hold it. Bytes equal to a key the map holds are that key’s bytes,
which were checked when it went in, so the check would answer the same. A
name that is not UTF-8 equals no key, so it misses and is checked at its own
offset as D3 says: the error and what the program printed before it are
unchanged. In 1BRC the check per row cost 115.6 instructions of 338.1 when the
mapping was parsed as Bytes; on a miss it runs once a station.
Decided by the supervising session on the owner’s direction (October 20, 2026: decide what is needed for the goal), from #664’s measurement.
ref String. ADR-016 D1’s reason stands unchanged.Bytes (fs::map without the check),
and a type for text not yet checked that a map lookup could check only on a
miss. They are about speed, not about what a program can say, and are
measured in their own issue before anything is decided.