Status: Accepted; D5 added October 19, 2026 (the check moves from fs::map to where the mapping becomes text).
Closes: UTF-8 validation being serial, the next cost after ADR-015
Related: ADR-283 (& is a view marker — which is why D1 is
not optional), ADR-296 D19 (nothing but the
driver), ADR-296 D10 (chunk count does not change results)
fs::map validates the file as UTF-8 before the parser sees it, serially. At
105 MB that is about a third of the parallel run; at the benchmark’s real 13 GB
it would be seconds before a row is read. Two fair questions follow: why is the
check there at all, and why is it that slow?
&str.fs::map hands the parser a &str over the mapped pages, and every slice the
grammar cuts out of it is a &str into those same pages. Removing the check
does not make the program slightly less careful; it makes it
from_utf8_unchecked over bytes nobody controls, and every view after it
undefined behaviour in code the user wrote safely.
Not an argument from principle. With the check removed, the flagship prints
{Hamburg=12.0/12.0/12.0, \xffBad=3.0/3.0/3.0}for a file with one stray byte — a station name that is a &str, is not UTF-8,
and is formatted without complaint on the far side of a Deref that promises
otherwise. The check is load-bearing.
What may eventually skip it is an assertion the user makes, in the shape the
language already has three times over (@frame(…, unchecked),
fs::map(…; trusted: false), dsl backend::x86) — a hatch the source opens and
the ledger records, never a default the runtime picks. §4 says why there is
none.
fs::map cuts the mapping into one piece per core and validates in parallel. A
cut walks forward off a continuation byte, at most three times, so every piece
begins where a character begins: no sequence is split, and no piece can accept a
truncation the piece before it left behind. A failing piece contributes
chunk_start + valid_up_to, and the answer is the minimum — so a rejected
file names the same byte however many cores looked at it, which is
ADR-296 D10’s promise applied to validation.
Below 1 MiB it runs in one piece: waking the pool costs more than the check.
The serial validator is the oracle it must agree with, verdict and offset.
user_parallelism does not reach into std, and does not here either.The obvious objection: user_parallelism = no means Parallelism::Off, so should
fs::map validate on one core there? No — and it could not, because
ADR-296 D19 says nothing but the DSL driver reads the switch.
What the switch governs is the fold: whether the input is cut into pieces whose results are merged. The UTF-8 check has no result to merge. It returns the same verdict and the same offset on one core and on sixty-four (D2), so how many cores ran it is not observable in the program — it is a library function doing the fastest correct thing.
The switch bounds the user’s code, not every function a program calls (ADR-037 D2).
checked-text checks a piece with simdutf8: its fast check answers the
common case, that the bytes are text, and only a piece that fails is checked
again by its compat half, which finds the offset std::str::from_utf8 would.
The verdict and the offset are the serial oracle’s (D2); a test holds one piece
to it for every one- and two-byte sequence and a set of three- and four-byte
ones, each at the start of a piece, inside a long ASCII run and across the
64-byte blocks the vector check reads.
It was rejected while the check was 10 ms of a 140 ms program. On 1BRC since
(benches/brc, Nikaia 0.0.424), it was 35.5 of 390 instructions a row and
0.89 of its 4.28 mispredictions, the largest single thing left against the
hand-written loop that does not validate at all; with simdutf8 the program
runs 365 a row and 3.42, the same output and the same refusal of a stray byte,
at the same offset. On the clock, pinned to one core and unpinned alike, it is
the same or faster (0.332 → 0.305 s, 0.305 → 0.278 s); on an all-ASCII twin
of the file it is the same (0.301 → 0.302 s), which is the case where the
standard library’s own fast path was already good. The dependency is one
crate, behind checked-text’s simd feature, so the crate that holds the
unsafe of text still has none without its features.
fs::map hands back the pages as a Bytes and checks nothing. m.text() is
the mapping as text: the first call checks the whole of it as D2-D4 say and
throws io::IoError::NotText with the earliest bad byte; every later call hands
back the same view without checking again. A cut, m[a..<b].text(), checks
that cut and nothing else, as any Bytes’s does (ADR-320 D7). A grammar run
over a mapping checks the text it reads, and a byte that is not UTF-8 is an
error at its position.
D1 is untouched: nothing becomes a &str unchecked. What moves is where the
check is paid. A program that reads only some of a file as text, a binary
format with text fields, or a parser that needs a few names out of a gigabyte,
pays for the bytes it reads as text and not for the file. A program that reads
all of it as text pays what it paid before, once. And the fused check §4
deferred, each piece checked on the thread that parses it, becomes the
grammar’s to do, because the grammar is handed bytes.
The price is one line: a program that reads a mapping as text writes
m.text(), and NotText comes from there instead of from fs::map.
On 8 million rows (105 MB, about 1.5 % of bytes multi-byte) and 4 cores, the chunked check divides by the core count (about 3.9×), and what is left to gain by removing the check entirely is 10 ms — 2 % sequential, 7 % parallel — against the output in D1. That is the answer to “why not switch it off”.
The check costs a larger share of the time than of the instructions because it is a second pass over the memory, which the parse then walks again. And it is slow on this data because the standard validator’s ASCII fast path drops to the per-character path at every multi-byte station name — about 10 GB/s on an ASCII twin of the file, under 2 GB/s on the real one.
\xff above (D1).trusted: false —
ADR-010’s hatch is about hashing, not encoding, and the two
must not share a keyword: one says “I accept a slower hash”, the other says “I
accept undefined behaviour if I am wrong”.fs::map_bytes, beside a
checking fs::map (D5): two functions for one mapping, and a program that
wants a few fields as text still pays for the whole file through the one it
picked.fs::map that picks bytes or text (D5): a result whose type
depends on an option’s value.