Open and answer this issue on GitHub.
language, libraries, performance, ready · opened 2026-10-05 · updated 2026-10-19 · 13 comments
Can a Nikaia program make a hot loop over text as fast as tuned Rust, when the author is willing to write more? Today the answer is no. The fastest Nikaia 1BRC is the one that writes the least: the grammar in examples/1brc.nika. Every hand-tuned variant was worse, because two building blocks are missing.
Callgrind, 1 M rows, per row, the same output as naive in every case. The method is benches/brc/README.md’s. The compiler is 0.0.478.
| variant | instructions | mispredictions |
|---|---|---|
examples/1brc.nika: grammar and HashMap |
300.1 | 1.58 |
| grammar, own open-addressing table, FNV-1a over every byte of the name | 330.5 | 2.66 |
| grammar, own table, hash of the length and the first 8 bytes at most | 309.3 | 1.49 |
one hand-written loop over text.bytes(), hash computed while reading the name, own table |
428.0 | 4.39 |
Rust, tuned |
272.2 | 3.70 |
Where it goes:
bytes(), one byte per turn; nikaia-std’s hash and tuned’s read eight at a time. Every byte of the name was 30 instructions a row and a misprediction at the loop’s varying end.; or \n from a position, the loop above spends 275 instructions a row in itself. The grammar searches with memchr inside its runtime, which is why it beats hand work.text.find(b';', from) → i64?, or an iterator of positions, backed by memchr.hash::of(name) with std’s byte hash, so a table of one’s own does not pay a byte loop.text.byte_at(i)), which a tuned parser walks back with.std functions or as grammar features. A grammar already has the search; is it enough to say “write a grammar”?unsafe, UTF-8 boundaries respected when a slice is cut, bounds checked or proved.memchr/as_bytes, Go’s bytes.IndexByte, Zig’s std.mem.indexOfScalar, Java’s String.indexOf(int, from).The decision basis (introduction, today’s state, other languages, options, a recommendation) is owed before this is asked.
Written in 9d7ddc11 (0.0.496):
NK1212-NK1214, char under NK1135) say what.The question started as what does a hand-tuned text loop need, measured above. In the round with the owner it became what one character is in the language as a whole:
Bytes that can be read, were agreed first.s.len() counts bytes, s[a..<b] cuts at bytes and stops in Rust’s words inside a scalar, and the specification said none of it.String.Index, graphemes checked in the compiler and segmented again in the runtime. The owner chose views that name their unit, with graphemes kept out of the compiler (Rust’s and Go’s split).scalar;s.scalars and s.graphemes are sequences;'é' is normalised with a warning rather than refused;== and a hash compare bytes, and normalising is written with its form.char. The types are scalar and u8.s.bytes indexes and cuts; s.scalars and s.graphemes are sequences. There is no s.len() and no s[i]; s.is_empty() stays.s.find(needle; from:) -> i64?, a grammar’s match, either moved by the byte length of a literal. A made-up number needs a view (NK1214).'…' is a u8 or a scalar by its use (NK1212). A decomposed one that normalises to one scalar is taken, with NK1213.Bytes is read as a run: b[i], b[a..<b], len, find, text().std’s, by UAX #29 in std’s Unicode version.== and a hash compare bytes; s.normalized(; form: nfc|nfd|nfkc|nfkd) is written.Still open: a hash a program can call.
The order matters: each step keeps the tree building.
scalar as a type beside char. Make char an alias the checker accepts for now, then move std.contracts, text::digit_value, the grammar builtins (any, digit, hex_digit, char) and the self-hosted tools to scalar. After that, char is NK1135 with the help naming scalar and u8.'…' typed by its use as u8 or scalar, as a number is (Part I 2.4), defaulting to scalar; '\xNN' above \x7F only as a u8; surrogates refused; NK1212. NK1213 normalises through std’s NFC.s.bytes (index, cut, len, find, for), s.scalars and s.graphemes (take, skip, count, for). Graphemes go through std, e.g. unicode-segmentation behind nikaia-std, with its Unicode version stated.find: s.find(needle; from: i64 = 0) -> i64? on text, with a one-byte needle through memchr; b.find(byte; from:) on Bytes and s.bytes. This replaces the fall-through to Rust’s str::find (#452).Bytes: b[i] -> u8, b[a..<b], b.find, b.text() (#452’s raw[0]).find, a grammar match, plus a literal’s byte length) and lets it cut that text. A made-up number or another text’s position is NK1214, with the view in the help. A cut inside a scalar stops the program with the language’s own message, not Rust’s not a char boundary.s.len() and s[n] on text. Move the self-hosted tools off chars().collect() + index, onto views or find.s.normalized(; form:) in std.'\u{D800}' refused;'é' taken with NK1213;line[..<sep] after find compiles, and s[0..<10] is NK1214;ä stops with the language’s message;"ä" == "a\u{308}" false, and true after normalized(; form: nfc);s.graphemes.count() of a family emoji is 1;find from inside a scalar finds the next match;find measured again against the table above.Generated by Claude Code
Added to step 5 (Bytes), from #489 (closed as a duplicate of this issue):
b.to_text() passes the checker and fails in rustc, because the ledger describes only Bytes::len and Bytes::is_empty. Step 5’s ledger entries close that hole. b.text() checks UTF-8 and fails with NotText (ADR-320 D7, Part I 2.6).b.text() over the bytes of a local socket pair, compiled and run, prints the text.NotText.tests/contracts.rs compiles its program instead of only lowering it.Fixes #453, #452, #489.Generated by Claude Code
s.find(needle; from: i64 = 0) -> i64? gives the byte position of the first match at or after from, or null.
How it is built
std.contracts has a str::find entry, so from: is checked as any option is.find on text, and the emitter writes it as nikaia_std::search::find(&s, &needle, from). Rust’s inherent str::find (one argument, a usize) would win over anything a library adds. This ends #452’s fall-through to str::find.What it does
from before the start searches from the start. One past the end finds nothing.str::find already runs for it (memchr-style).Tests
tests/text_find.rs: from the start, from a position, nothing after the last match, inside ä, and a cut at the position (line[0..<sep]).nikaia-std/src/search.rs.Not covered by this step
b.find(byte; from:) on Bytes and on s.bytes belongs to step 5, with the Bytes reads.find, line[..<sep] / line[sep + 1..] (a cut with an open end), does not parse.Generated by Claude Code
Bytesb[i] and b[a..<b]: index::Get is now implemented for Bytes. An index outside stops the program, as a list’s does.b.find(byte; from:) -> i64?: a byte search from a position.b.text(): a view of the text, or io::IoError::NotText where the bytes are not UTF-8. The Rust method went from Option to Result.std.contracts now describes Bytes::text and Bytes::find. #489’s hole is closed: a b.text() that the checker passed is described now, so rustc is no longer the first to hear of it.Tests
tests/bytes.rs: a file read as Bytes (index, cut, find twice, text), and a non-UTF-8 file caught as NotText.tests/sockets.rs: the bytes a server reads from a socket are made text, compiled and run, giving server read ping.tests/contracts.rs: the #472 test now compiles its program, not only lowers it.Not covered: invalid UTF-8 over a socket. A program cannot write raw bytes to a socket yet, so this is shown with a file; it is the same Bytes::text.
Note: a byte is written as a number for now (b.find(59)). ';' as a u8 is step 2.
[..<b], [..b] and [a..] now parse. [a..] is lowered to the language below’s a.., so the base is read once. Part I 2.6’s example after find now parses.
scalar instead of char, '…' typed by its use, and the s.bytes / s.scalars / s.graphemes views.b.find on s.bytes, which waits on step 3.Generated by Claude Code
8ff69c9, 0.0.660 9127dae)Step 1: there is no char.
char in a type position is NK1135. Its help names scalar and u8.char reads as scalar (ty.nika).scalar: std.contracts, std::text, the self-hosted tools and the test programs. The ADRs keep the word they were written with.nikaia lower-std regenerated text.rs and package.rs, and lowering again changes nothing.undeclared_type.rs char_is_refused_with_both_words.Step 2, first half: a '…' is a u8 where a byte is wanted.
u8 (an annotation, an argument, a comparison with a byte), it is written b'a' below. Everywhere else it is a scalar.'é' where a u8 is wanted is NK1103.b[i] on a Bytes is now a u8 in the checker too, and b[a..<b] a Bytes. Before, bytes[0] == 'a' reached rustc as a mismatch.tests/language/src/byte_literals.nika (both settings), and literals.rs.Still open in step 2:
NK1212, and NK1213 with NFC. This needs std’s normalisation, so it lands with step 8.'\xNN' above \x7F, which today’s lexer does not read.let c = 'a' whose later use is a byte. Today it is a scalar, because only a use at the literal itself decides. Numbers have open bindings (ADR-285 D23) and literals do not yet.Two things the records do not say, before steps 3 and 8 can be built:
s.bytes has. It indexes, cuts, has len/find, is walked by for, and s.bytes[0..<10].text() makes text again. A copy into a Bytes would allocate on every use, which defeats #453’s own measurement. So it needs a view type of its own, a borrowed run of the text’s bytes. Its name decides what a program writes in an annotation (let b: ? = s.bytes) and in a parameter.nfc is in s.normalized(; form: nfc). Every other variant in the specification is written in full (supervisor::Strategy::OneForOne, fs::Root::Anywhere). A bare nfc needs either a rule that a variant may be written bare where its type is known, or the form written as text::Form::Nfc.Generated by Claude Code
Written in 5fdce58 (0.0.1030): ADR-320 D10 (amended) and D11, and Part I 2.6.
s.bytes is a Bytes. It is the type fs::read and a socket hand out, so there is one set of reads and one parameter type. Like a String (ADR-282 D2), it is a view wherever a view suffices: borrowed from the text’s buffer, with no copy unless it is kept beyond the text. Rejected: a view type of its own (Swift’s UTF8View), a copy every time (Go).s.normalized(text::Form::Nfc) (or Nfd, Nfkc, Nfkd). The form is a positional argument, not an option after ;, because no form is a sensible default. As a variant, it is written with its full path. The owner’s proposal; rejected: s.nfc() (the form cannot be a value), a bare nfc (a rule for every enum).s.bytes lowers to s.as_bytes() (&[u8]) where a view suffices, and is typed Bytes in the checker. Its reads are D7’s, the same as for a Bytes from fs::read. Kept beyond the text, it is copied, as for a String.text::Form (Nfc, Nfd, Nfkc, Nfkd) in std, and s.normalized(form: text::Form) -> String.Generated by Claude Code
Step 3, part one is built: 620afac, version 0.0.1031 (92a2151).
s.bytes on text is typed Bytes. The checker records each such node in a new fact, text_bytes, and the emitter lowers it to s.as_bytes(), with no copy.Bytes parameter the body only reads lowers to &[u8], as a read-only String is &str, so one function takes both fs::read bytes and s.bytes.&[u8] in a new trait, nikaia_std::bytes::ByteReads (text(), find(byte, from)), and Bytes calls it. This also fixes b[a..<b].text(), .find() on a cut, and a cut passed to a Bytes parameter, which used to fail in rustc.s.bytes is copied with Bytes::from(..). Returning it from a ref String parameter stays refused (NK1131).crates/nikaia/tests/text_bytes.rs, together with bytes, text_find, grammar_bytes and sysroot.Still missing for step 3:
s.bytes is written directly. let k = s.bytes followed by return k, or pushing it into a list, still fails in rustc with “mismatched types”. The copy has to follow the value: wherever a let bound to s.bytes escapes, it gets Bytes::from(..), as an escaping String view does. Test: return through a let, push into a Vec[Bytes], store through a let into a field.Bytes has no .clone() contract, so a program cannot copy explicitly. Add it, with a test.Generated by Claude Code
Step 3’s s.bytes part is built: 2089126, version 0.0.1032 (6341d31). s.scalars and s.graphemes from step 3 are not built yet.
let k = s.bytes binds the view. Wherever the value is kept (a push, a field including the P { b } shorthand, an argument the callee keeps, a let mut, ??, a return), the checker records it in kept_text_bytes, and the emitter wraps that one node in Bytes::from(..).v.push(s.bytes) followed by another read of s.bytes is no longer refused as NK2105.Bytes::clone(). On a Bytes it shares the buffer. On s.bytes, on a name bound to it, or on a read-only parameter, it makes a Bytes of its own (ByteCopy::copy_of).text_bytes (6), bytes, grammar_bytes, sysroot and handed_over pass.Found on the way:
Bytes is filed as #665: let k = p.items; return k fails in rustc.fn back(s: String) -> Bytes { return s.bytes }, and a body that pushes s.bytes, still take s as an owned String, so callers allocate. The derived keeps analysis (crates/nikaia/src/contracts/keep.rs) counts s.bytes kept as s kept, although only a copy of the bytes is kept. This belongs to the performance work in #664.Generated by Claude Code
Step 3 is finished: e08b9e5, version 0.0.1033 (ac754ce).
s.scalars is a Seq[scalar] that can also be walked from the back. It lowers to s.chars().s.graphemes is a Seq[ref String], segmented by UAX #29. It goes through nikaia_std::graphemes::TextGraphemes, which uses unicode-segmentation 1.13.3 (Unicode 17.0.0). The version is exported as nikaia_std::graphemes::UNICODE_VERSION.take, skip, count, for and collect work as on any sequence, and a grapheme compares with a text literal.text_scalars and text_graphemes.crates/nikaia/tests/text_scalars_graphemes.rs:
"aä€".scalars.count() is 3;take and skip;"a\u{308}";== "ä";for over graphemes.rev() on s.graphemes stays refused, because Part I counts only scalars among the sequences that can be walked from the back.What remains:
NK1212, NK1213 with NFC, '\xNN' above \x7F, and a let typed by its later use;Generated by Claude Code
Step 8 and the NFC half of step 2 are built in 13c70fe, version 0.0.1034 (14b72d3).
text::Form (Nfc, Nfd, Nfkc, Nfkd) is a std enum. It can be held in a let, passed on and compared.s.normalized(form) -> String is described as str::normalized in std.contracts. It uses unicode-normalization 0.1.25 (Unicode 17.0.0), exported as nikaia_std::text::NORMALIZATION_UNICODE_VERSION.'…' of several scalars:
NK1213. The help offers 'é' or '\u{e9}'.NK1212, and the help gives the text literal, e.g. "👍🏽".nikaia_std::text::normalized.'\u{…}' and 'e\u{301}' did not parse at all.\u{D800} is NK1184 in every literal, text or '…', as the compiler did already. Part I 2.5 and Part III’s table now say so, and NK1212 no longer lists it (14b72d3).crates/nikaia/tests/text_normalized.rs (6), plus literals, text_literals, text_bytes, text_scalars_graphemes, sysroot, build_time and contracts.Still open:
NK1212 for a non-ASCII scalar where a u8 is wanted. Today that is NK1103.'\xNN' above \x7F, which the lexer does not read yet.let typed by its later use.Generated by Claude Code
Step 2 is built in 5fdee1e, version 0.0.1035 (38111d4). The rule is in Part I 2.6 and in Part III’s NK1200 entry.
'\xNN' above \x7F is read in a '…'. Where a u8 is wanted it is a byte (b'\xff'). Where it stays a scalar it is NK1212, and the help gives '\u{ff}' and a u8 annotation.'é' where a u8 is wanted is NK1212 instead of NK1103: “'é' is a scalar of 2 bytes, and a u8 is one”, with "é".bytes in the help.let c = ';' is typed by its uses, as an open number is (ADR-285 D23).
u8 parameter, b.find(c), an annotated let), it is a u8.u8 and a scalar are NK1200, with let c: u8 = … in the help.literals.rs (a_high_byte_escape_is_only_a_byte, a_character_binding_is_typed_by_its_uses) and two run tests in byte_literals.nika.Still open, small:
let c = 'é' handed to a u8 parameter is NK1102, not NK1212, because only literals that can be a byte are open.let mut of a '…' stays a scalar.Remaining steps: 6 (positions, NK1214), 7 (no s.len() / s[n] on text) and 9 (the remaining tests, and 1BRC by hand measured again).
Generated by Claude Code
Step 6 is built in ff7f50c (version 0.0.1037, e77d11f; the commit subject says 0.0.1036, which #659 had taken in the meantime).
What counts as a position of a text (check_core_places_text_position, kept per immutable let in CheckState.text_positions):
t.find(…), with or without from:;[..<p] or [p..]. Part I 2.6 now says so;x ?? fallback, where the fallback leaves (return, break, continue, throw) or is another position in the same text;if, each of whose branches is one such position;"lit".len(), or plus the byte length of the literal that find searched for (sep + 1 after find(";")).Refused with NK1214:
The help names s.bytes[a..<b].text() and s.graphemes.take(n). Cuts of s.bytes and of Bytes take any number.
A cut inside a scalar stops the program with the language’s own words, through TextCut in nikaia-std/src/index.rs: “byte position N is inside a scalar of the text: cut it at a position it handed out, or through .bytes”.
Fixed on the way: in a for over a Bytes, the loop variable had no type, so c == '\n' was refused in rustc. It is a u8 now, with a test in text_bytes.rs.
Moved to the new rule:
tools/*.nika;examples/k-nucleotide, whose output is byte-identical;Tests: text_positions (7), plus the text, slice and indexing tests, and tests/build-time 41/41.
In progress: crates/rs2nika (about 157 cuts) is being moved off made-up cuts. Until that lands, the rs2nika CI job is red whenever it runs.
Still open:
let mut positions are not tracked.+ needle.len() for a needle held in a variable is not counted. ADR-320 D5 says “a known text”, Part I says “a text literal”, and the build follows Part I.Generated by Claude Code
For a table of one’s own: #434 measured what each variant of a hash table costs, and for which data. The variants are a stored hash per entry, keys compared a word at a time, and no removal. The defaults stay as they are because the compiler cannot know the data. Whoever tunes by hand does know it, and needs exactly these means: a hash of a text slice the program can call (point 2 above), a hasher chosen per map (#581), and a Map trait a table of one’s own can implement (#593). The numbers are in the closing comment of #434.
Generated by Claude Code