charStatus: Accepted; D10’s form and D11 added October 18, 2026; D12 added October 19, 2026; D13 added October 20, 2026. A hash a program can call is not decided here.
Date: October 5, 2026
Answers: #453, what a
program needs to work on text by hand: a search from a position, one element
at a position, and what one character is.
Related: ADR-016 D1 (text is valid UTF-8, checked where it
enters); ADR-283 (a mapping, its views, what a Bytes offers);
ADR-285 D1 (a count and a position are an i64), Part I 2.4 (a
literal’s type comes from its use); ADR-287 (comptime);
ADR-288 D11 (what the language offers is the specification’s
decision); ADR-279 D8 (a value that is a word by its kind is
copied).
Text is UTF-8: a is one byte, ä two, € three. The element at position
5 can mean the byte at byte 5, the scalar starting there, or the fifth thing a
reader sees, and those differ as soon as the text is not ASCII. A reader sees
one ä whether it is stored as one scalar (U+00E4) or as a and a combining
mark (U+0061 U+0308), and one family emoji that is five scalars.
The language had len() counting bytes, s[a..<b] cutting at byte positions,
and char - one character, a Unicode scalar value - which a reader takes for
what they see. None of it said its unit.
| storage | an element by number | a character | |
|---|---|---|---|
| Rust | UTF-8 | none on str; as_bytes()[i] |
char, a scalar; graphemes in a crate |
| Go | bytes | s[i] is a byte |
rune, a scalar |
| Swift | UTF-8, ASCII flag, short strings inline | none; String.Index, views utf8, unicodeScalars |
Character, a grapheme, checked in the compiler and segmented again by the runtime |
| Python | Latin-1, UCS-2 or UCS-4 by content | s[i], a scalar |
no type of its own |
| Java | Latin-1 or UTF-16 by content | charAt(i), a UTF-16 unit |
char, a UTF-16 unit |
Rust and Go keep scalars in the compiler and graphemes in a library; Swift puts graphemes in both, and its compiler and runtime have disagreed about emoji sequences - on Apple’s systems the runtime is the operating system’s, so one binary segments differently on two versions.
One representation. A file mapped into memory is text without a copy, and a name cut out of it is a view into it; a representation chosen by content, as Python’s and Java’s, would recode every input to get there.
char. A scalar is scalar, and a byte is u8.char means a byte in C, a UTF-16 unit in Java, a scalar in Rust and a
grapheme in Swift, and every reader brings one of them. scalar is a Unicode
scalar value - a code point that is not a surrogate (U+D800-DFFF), which valid
UTF-8 never holds - and is a word, copied as a number is. It says what is
counted and nothing about an encoding.
s.len() and no s[i].s.bytes, s.scalars and s.graphemes. A number about text means nothing
until its unit is said, and which unit a program means - ten bytes of a fixed
field, ten scalars, ten things on a screen - is the program’s meaning, which no
compiler can guess. Where no number is involved there is no unit to say:
split, lines, trim, starts_with, contains, ==, find, a grammar and
for need no view. s.is_empty() stays: empty is empty in every unit.
s.bytes[i] and s.bytes[a..<b] index and cut, and s.bytes.len() counts, in
constant time. s.scalars and s.graphemes are sequences: take(n),
skip(n), count(), for. Reaching the tenth grapheme walks nine; a bracket
would hide that, and take(10) says it.
Replaced by ADR-374: a position is a text::Pos, a text is cut
by before, after and between.
let sep = line.find(";") ?? return
let name = line[..<sep]
let rest = line[sep + 1..]find’s result, a grammar’s match and such a position moved by the byte length
of a known text are positions in that text; the compiler keeps where a
position came from, as it keeps where a value came from for trust, and lets it
cut the text it came from without a view. That is the compiler doing the work:
the program never says a unit it did not choose. A number the program makes up
needs a view (NK1214), and so does a position from one text used on another.
A cut inside a scalar stops the program, in the language’s words.
find searches from a position, on text and on bytes.Changed by ADR-374 D2: on text find and rfind give a
text::Match? and from: is a text::Pos; on bytes this stands.
s.find(needle; from: i64) -> i64? on text, and b.find(byte: u8; from: i64)
-> i64? on Bytes and on s.bytes. null is not there; a sentinel number
would be one more thing to forget before a cut. On text from may be any byte
position: UTF-8 is self-synchronising, so a match of a valid needle always
starts on a scalar. A one-byte needle is a byte search of the kind memchr does,
which is what makes a parser written by hand as fast as a grammar’s search.
Bytes is read as a run of bytes.b[i] -> u8, b[a..<b], b.len(), b.find(…), and b.text() to make text of
it with the UTF-8 check. A buffer whose content cannot be read is not a buffer.
Its checks are a list’s, and a proof drops them as it drops a list’s.
'…' is a u8 or a scalar, by its use; its content limits which.'…' |
u8 |
scalar |
|---|---|---|
one ASCII character: 'a', ';', '\n' |
yes | yes |
'\xNN' above \x7F |
yes | no |
one scalar that is not ASCII: 'ä' |
no | yes |
A literal no use types is a scalar. A surrogate is refused. Several scalars
that normalise (NFC) to one are that one, with a warning (NK1213): an editor
that stores é as e and a combining mark has written what its user meant as
one character, and a scalar cannot hold two. The normalisation is the standard
library’s own, and NFC composition does not change between Unicode versions
(Unicode’s stability policy), so the meaning does not either. Anything else is
refused (NK1212), with the text literal to write instead: "👍🏽".
s.graphemes segments by Unicode’s rules (UAX #29) and hands out each grapheme
as a ref String, compared with a text literal. The rules change with Unicode
versions, so they live in one place - std, released with the program - and the
compiler’s meaning of a program never depends on them. A grapheme literal
checked while the program builds follows from comptime once std’s
segmentation is written in Nikaia: the check runs std’s code, not a copy.
== and a hash compare bytes; normalising is written.a == b is true where the bytes are, and a hash agrees with it. A precomposed
and a decomposed ä differ until the program writes
s.normalized(text::Form::Nfc) (or Nfd, Nfkc, Nfkd); the form is always
written, because NFKC also makes fi equal to fi. Equality that normalised
would cost every comparison that is not ASCII.
The form is an argument, not an option after ;: an option has a default, and
no form is one. It is a variant of text::Form written with its path, as every
variant is, so a form can be held in a value and passed on.
s.bytes is a Bytes, a view wherever a view suffices.s.bytes reads the text’s own bytes and is typed Bytes, the type fs::read
and a socket hand out, so a function over bytes takes either and b[i],
b[a..<b], len, find, text() and for are one set of reads (D7). As a
String is a view wherever a view suffices (ADR-282 D2), so is
this Bytes: it is the text’s buffer, borrowed, and costs no copy. A copy is
made only where the program keeps it beyond the text it came from, as for a
String.
Bytes reads a word at a position, in the order it names.b.u32_le(i) and b.u64_le(i) read the 4 or 8 bytes from i as an unsigned
number, least significant byte first; b.u32_be(i) and b.u64_be(i) read
them most significant first. Each reads what b[i..<i + n] reads and is checked
as that cut is: outside, it stops the program, and a proof drops the check
(D7). A signed i64 is the u64 read through bits_as_i64(), which keeps its
bits (ADR-285 D42), and a signed 32-bit word the u32 through bits_as_i32(); as widens only where the value fits (Part I 2.2), so a u32 becomes an
i64 with as. The widths are the
language’s unsigned ones (Part I 2.2): there is no u16, so a two-byte field is
two byte reads.
A file format, a protocol and a hash read words, and a program that writes eight byte reads, seven shifts and seven ors for one is asking the optimiser to find the load it meant. The order is in the name, because a word read in the machine’s own order gives another number on another machine. Reading a word is a read of bytes, not a construct for scanning: ADR-296 D12 keeps the grammar’s scans portable and word-at-a-time, and does not keep a program from writing its own.
let mut tail: Vec[u8] = Vec()
for b in data[cut..] { tail.push(b) }
for _ in 0..<8 { tail.push(0) }
let padded = Bytes(tail)A word read stays what D12 makes it, checked as its cut, also at the end of a
mapping. std offers no read that fills with zeros and guarantees no margin
behind a mapping. A program whose loop reads whole words up to its last row
copies the last bytes into a buffer of its own with zeros behind them and runs
the same loop over that buffer.
Why. The copy is a few bytes once per input, and it is what the fastest
programs in Rust and Java write by hand. A read that fills with zeros is four
more methods beside the four of D12, each to be documented, for a case one
loop per program has. A margin behind a mapping is a promise only some
platforms keep without work, holds for a mapping and not for a Bytes that
came from elsewhere, and would have to be carried in the type a function
takes, against fs::Mapped being a Bytes. Rejected: zero-filled reads
(u64_le_filled); a margin fs::map guarantees; reading past the end
unchecked.
Decided October 20, 2026.
s[i] as a byte (Go): it reads as a character and is not one.s[5; scalar]: the unit is right, but a
bracket promises constant time, and scalar and grapheme cannot keep it.String.Index): it stops mixing units
by making every position opaque; D5 stops it by knowing where a position came
from, and numbers stay numbers.char kept as a scalar (Rust): the name keeps misleading.s.bytes (Swift’s String.UTF8View): a second
name for a run of bytes, and every function over bytes written twice or behind
an interface.s.bytes as a copy (Go’s []byte(s)): an allocation on every read of a
field, which is what #453 measured against.s.nfc() (Rust’s unicode-normalization): short,
but the form cannot be a value.b.read[u64](i; order: little) (D12): one name for four,
but a type argument and an option on every read, and nothing else in std
is read that way.u64.from_le(b[i..<i + 8]) (Rust’s
from_le_bytes, D12): it needs a type for exactly n bytes, which the
language does not have, and says the length twice.memcpy into a uint64_t, D12): the
same bytes are another number on another machine.form: nfc, Swift’s
.nfc): a rule for every enum, not this one’s to make.s.scalars keeps an ASCII flag for a constant-time walk on ASCII
text.