build_time_dependencies can hang: two grammar-action tests wait on each other’s cargoOpen and answer this issue on GitHub.
ready · opened 2026-10-08 · updated 2026-10-08 · 4 comments
CI run 692 (develop at 0.0.673, 80d359d), job rust_floor: scripts/check-floor.sh ran the release suite on the floor toolchain. tests/build_time_dependencies.rs hung until the job’s 90-minute timeout.
13:11:50 test the_build_time_program_links_no_c_library ... ok
13:12:43 test a_grammar_action_calls_a_dependency_at_build_time has been running for over 60 seconds
13:12:43 test a_grammar_action_calls_another_file_of_its_package has been running for over 60 seconds
14:35:32 ##[error]The operation was canceled.The orphans killed at the end were the test binary, two nikaia processes (one per test), three cargo processes, and one more nikaia under a cargo. That last one is the rustc wrapper, lowering a package.
The same job passed at 0.0.666 (run 685), and bootstrap_compiler passed this test in run 692 itself. So it is intermittent, not a broken test.
Why it matters beyond CI: a hung job holds every later run on develop in the queue, because pushes do not cancel a run in progress. Nothing could reach main for 90 minutes.
Nested cargo builds that take locks in different orders:
cargo builds a test’s project. Its wrapper lowers the package, and a grammar action or comptime there starts an inner cargo on a target directory shared by the whole machine: grammar_run::shared_target(), and the build-time bundle under the user cache since 0.0.645.comptime_run::bundle relies on cargo’s lock on that directory so that two builds wait for each other instead of colliding.build_time_dependencies’s two grammar-action tests concurrently in a loop, release mode (cargo test --release -p nikaia --test build_time_dependencies a_grammar_action). Capture ps and the Blocking waiting for file lock lines when it stalls.CARGO_HOME lock scope, or take a lock of nikaia’s own around the shared target that is never held while waiting on an outer cargo.A first reproduction attempt, debug build on the stable toolchain, the two tests together five times:
So those two lines alone are the cold build of the shared targets, not a hang. What CI hit was still running 80 minutes later, in release on the floor toolchain with a cold cache. That makes step 1 a cold, release, floor-toolchain run: rustup toolchain install 1.88, a fresh NIKAIA_CACHE_DIR, and the two tests together in a loop.
Generated by Claude Code
The floor reproduction did not reproduce it: Rust 1.88, --release, a fresh NIKAIA_CACHE_DIR/XDG_CACHE_HOME, the whole build_time_dependencies file. All six tests passed in 82 s; four of them ran past 60 s on the cold builds and then finished. The same CI job did not hang in run 706 either (0.0.689).
So it is rare. Neither a cold cache nor the floor toolchain is enough on its own. What remains specific to run 692 is a CI runner, which has fewer cores than this machine, with every other test file of the suite running before it in the same cargo test process tree. Reproducing it needs a loop in that environment, or a reading of the lock order (step 2) without a reproduction.
Generated by Claude Code
rust_floor is switched off (1132cae, owner’s decision): until Nikaia is stable the suite runs on the current Rust only, and the oldest supported Rust becomes a goal again later. The job stays in .github/workflows/nikaia.yml behind if: false, and promote no longer waits for it.
This hang no longer holds develop up. The suspected cause, nested cargo builds on a target directory the whole machine shares, can still meet a user who builds two projects at once, so the issue stays open for that.
Generated by Claude Code
It hung again in run 707 (0.0.690, job 113375175899). The last lines before the cancel were:
Running tests/language.rs (target/floor/release/deps/language-…)
running 2 tests
test the_build_time_package_passes_its_own_tests has been running for over 60 seconds
test the_language_package_passes_its_own_tests has been running for over 60 secondsSo both hang in tests/language.rs. Two nikaia child processes and nested cargo processes were still alive at cleanup. That suggests the two package test runs block each other, for example on a shared cargo target or build-directory lock, when they run in parallel on the floor toolchain.
On the owner’s decision, rust_floor is switched off for now (commit 1132cae, if: false), and promote no longer waits on it. This issue stays open for when the floor job comes back.
Generated by Claude Code