x86_64-linux builds hang at random, defunct processes in `ps`

[booxter]

For some packages, I get builds hang without making progress. For example, if I build python312Packages.llama-cpp-python, the last messages for the build I see are:

llama-cpp-python-x86_64-linux> -- Check for working C compiler: /nix/store/gj9lra51hwhxnhz05jqk5lh03wipamv0-gcc-wrapper-14-20241116/bin/gcc - skipped
llama-cpp-python-x86_64-linux> -- Check for working C compiler: /nix/store/gj9lra51hwhxnhz05jqk5lh03wipamv0-gcc-wrapper-14-20241116/bin/gcc - skipped
llama-cpp-python-x86_64-linux> -- Detecting C compile features
llama-cpp-python-x86_64-linux> -- Detecting C compile features
llama-cpp-python-x86_64-linux> -- Detecting C compile features - done
llama-cpp-python-x86_64-linux> -- Detecting C compile features - done
llama-cpp-python-x86_64-linux> -- Detecting CXX compiler ABI info

And if I inspect the list of processes, I see:

1039 ?        Ss     0:00 /run/binfmt/rosetta /nix/store/ki4if6b0w5bqv8dc5lrjp8xm7wjy9dlf-bash-5.2p37/bin/bash -e /nix/store/vj1c3wf9c11a0qs6p3ymfvrnsdgsdcbq-source-stdenv.sh
   1045 ?        Ss     0:00 /run/binfmt/rosetta /nix/store/ki4if6b0w5bqv8dc5lrjp8xm7wjy9dlf-bash-5.2p37/bin/bash -e /nix/store/vj1c3wf9c11a0qs6p3ymfvrnsdgsdcbq-source-stdenv.sh
   1159 ?        S      0:00 /run/binfmt/rosetta /nix/store/wz0j2zi02rvnjiz37nn28h3gfdq61svz-python3-3.12.9/bin/python3.12 /nix/store/bzv2mj9f8hy89azfg44pggnn8mina9x5-python3.12
   1161 ?        S      0:00 /run/binfmt/rosetta /nix/store/1031nlccnpdp7ljbjm4ja0q7gqadsl3m-python3-3.13.2/bin/python3.13 /nix/store/dk10yg5zz30yjn5wjz9ciy4bggnqm7av-python3.13
   1164 ?        S      0:00 /run/binfmt/rosetta /nix/store/1031nlccnpdp7ljbjm4ja0q7gqadsl3m-python3-3.13.2/bin/python3.13 /nix/store/gxwbfwdiygi92k2l8lj757fr4k66kh81-python3.13
   1165 ?        S      0:00 /run/binfmt/rosetta /nix/store/wz0j2zi02rvnjiz37nn28h3gfdq61svz-python3-3.12.9/bin/python3.12 /nix/store/h9y0b0g4h0gqd699pdyxss7xq8gg0kvh-python3.12
   1172 ?        S      0:00 /run/binfmt/rosetta /nix/store/j3f4j0i2h5d7h3sbfnlgva8nk0cydvwq-cmake-3.31.5/bin/cmake -S. -B/build/tmp7hogwqg0/build -DCMAKE_BUILD_TYPE:STRING=Rele
   1173 ?        S      0:00 /run/binfmt/rosetta /nix/store/j3f4j0i2h5d7h3sbfnlgva8nk0cydvwq-cmake-3.31.5/bin/cmake -S. -B/build/tmp34uqs_rf/build -DCMAKE_BUILD_TYPE:STRING=Rele
   1321 ?        Z      0:00 [ld] <defunct>
   1340 ?        Z      0:00 [ninja] <defunct>

This doesn't happen to all builds, only to some. (AFAIU these are always builds involving actual compilation. Regular python packages build fine for what I see.)

[booxter]

I think this report may describe the same situation for other packages: #2 (comment)

[booxter]

If there's more information to collect from the machine, let me know how because I don't know how to get root access to it to e.g. check dmesg or system journal.

[booxter]

@dwt FYI

[dwt]

Yeah, that looks quite like what I experienced.

[cpick]

We'll debug the hangs in this issue since they look similar.

From @dwt in the other issue:

Here's an example of a hang:

[builder@rosetta-builder:]$ nix run nixpkgs#pstree -- -u nixbld1
-+= 00001 root /run/current-system/systemd/lib/systemd/systemd
-+= 00925 root nix-daemon --daemon
-+= 06779 root nix-daemon 6774
-+= 07327 nixbld1 /run/binfmt/rosetta /nix/store/11ciq72n4fdv8rw6wgjgasfv4mjs1jrw-bash-5.2p37/bin/bash -e /nix/store/vj1c3wf9c11a0qs6p3ymfvrnsdgsdcbq-source-stdenv.sh /nix/store/shkw4qm9qcw5sc5n1k5jznc8
-+- 07852 nixbld1 /run/binfmt/rosetta /nix/store/01dapiw4pjrc9s0166dcyg9xkw0rv01z-cmake-3.31.5/bin/cmake .. -GNinja -DCMAKE_FIND_USE_SYSTEM_PACKAGE_REGISTRY=OFF -DCMAKE_FIND_USE_PACKAGE_REGISTRY=OFF -
--- 07997 nixbld1 (ninja)
-+= 00001 root /run/current-system/systemd/lib/systemd/systemd
-+= 00938 root nix-daemon --daemon
-+= 00940 root nix-daemon 935
-+= 00994 nixbld1 /run/binfmt/rosetta /nix/store/11ciq72n4fdv8rw6wgjgasfv4mjs1jrw-bash-5.2p37/bin/bash -e /nix/store/vj1c3wf9c11a0qs6p3ymfvrnsdgsdcbq-source-stdenv.sh /nix/store/shkw4qm9qcw5sc5n1k5jznc83
ny02r39-default-builder.sh
-+- 01462 nixbld1 /run/binfmt/rosetta /nix/store/01dapiw4pjrc9s0166dcyg9xkw0rv01z-cmake-3.31.5/bin/cmake .. -GNinja -DCMAKE_FIND_USE_SYSTEM_PACKAGE_REGISTRY=OFF -DCMAKE_FIND_USE_PACKAGE_REGISTRY=OFF -D
CMAKE_EXPORT_NO_PACKAGE_REGISTRY=ON -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTING=OFF -DCMAKE_INSTALL_LOCALEDIR=/nix/store/ks9jcd9iqk3ibxlai07qhsjwsavkndim-mumble-1.5.735/share/locale -DCMAKE_INSTALL_LIBEXECDIR=/
nix/store/ks9jcd9iqk3ibxlai07qhsjwsavkndim-mumble-1.5.735/libexec -DCMAKE_INSTALL_LIBDIR=/nix/store/ks9jcd9iqk3ibxlai07qhsjwsavkndim-mumble-1.5.735/lib -DCMAKE_INSTALL_DOCDIR=/nix/store/ks9jcd9iqk3ibxlai07qhsj
wsavkndim-mumble-1.5.735/share/doc/mumble -DCMAKE_INSTALL_INFODIR=/nix/store/ks9jcd9iqk3ibxlai07qhsjwsavkndim-mumble-1.5.735/share/info -DCMAKE_INSTALL_MANDIR=/nix/store/ks9jcd9iqk3ibxlai07qhsjwsavkndim-mumble
-1.5.735/share/man -DCMAKE_INSTALL_INCLUDEDIR=/nix/store/ks9jcd9iqk3ibxlai07qhsjwsavkndim-mumble-1.5.735/include -DCMAKE_INSTALL_SBINDIR=/nix/store/ks9jcd9iqk3ibxlai07qhsjwsavkndim-mumble-1.5.735/sbin -DCMAKE_
INSTALL_BINDIR=/nix/store/ks9jcd9iqk3ibxlai07qhsjwsavkndim-mumble-1.5.735/bin -DCMAKE_INSTALL_NAME_DIR=/nix/store/ks9jcd9iqk3ibxlai07qhsjwsavkndim-mumble-1.5.735/lib -DCMAKE_POLICY_DEFAULT_CMP0025=NEW -DCMAKE_
FIND_FRAMEWORK=LAST -DCMAKE_STRIP=/nix/store/ii75mhh7sxl11167m1b86p0qrjsjyjmd-gcc-wrapper-14-20241116/bin/strip -DCMAKE_RANLIB=/nix/store/ii75mhh7sxl11167m1b86p0qrjsjyjmd-gcc-wrapper-14-20241116/bin/ranlib -DC
MAKE_AR=/nix/store/ii75mhh7sxl11167m1b86p0qrjsjyjmd-gcc-wrapper-14-20241116/bin/ar -DCMAKE_C_COMPILER=gcc -DCMAKE_CXX_COMPILER=g++ -DCMAKE_INSTALL_PREFIX=/nix/store/ks9jcd9iqk3ibxlai07qhsjwsavkndim-mumble-1.5.
735 -D g15=OFF -D CMAKE_CXX_STANDARD=17 -D CMAKE_UNITY_BUILD=ON -D server=OFF -D overlay=OFF -D bundle-qt-translations=OFF -D bundled-json=OFF -D bundled-gsl=OFF -D update=OFF -D oss=OFF -D warnings-as-errors=
OFF -Dspeechd:BOOL=FALSE -Dpulseaudio:BOOL=TRUE -Dpipewire:BOOL=TRUE -Djackaudio:BOOL=FALSE -Dalsa:BOOL=TRUE -DPROTOC_EXE=/nix/store/ivbs25cdr9mhfrzvamiq824c6xadxsn8-protobuf-29.3/bin/protoc -DProtobuf_PROTOC_
EXE=/nix/store/ivbs25cdr9mhfrzvamiq824c6xadxsn8-protobuf-29.3/bin/protoc -DProtobuf_PROTOC_EXECUTABLE=/nix/store/ivbs25cdr9mhfrzvamiq824c6xadxsn8-protobuf-29.3/bin/protoc
--- 01503 nixbld1 (ninja)
[builder@rosetta-builder:~]$ nix run nixpkgs#pstree -- -u nixbld1
-+= 00001 root /run/current-system/systemd/lib/systemd/systemd
-+= 00937 root nix-daemon --daemon
-+= 00956 root nix-daemon 953
-+= 01010 nixbld1 /run/binfmt/rosetta /nix/store/11ciq72n4fdv8rw6wgjgasfv4mjs1jrw-bash-5.2p37/bin/bash -e /nix/store/vj1c3wf9c11a0qs6p3ymfvrnsdgsdcbq-source-stdenv.sh /nix/store/shkw4qm9qcw5sc5n1k5jznc8
-+- 01495 nixbld1 /run/binfmt/rosetta /nix/store/01dapiw4pjrc9s0166dcyg9xkw0rv01z-cmake-3.31.5/bin/cmake .. -GNinja -DCMAKE_FIND_USE_SYSTEM_PACKAGE_REGISTRY=OFF -DCMAKE_FIND_USE_PACKAGE_REGISTRY=OFF -
--- 01711 nixbld1 (ninja)
This is after a restart of the rosetta builder via a command like this on the host between each nix-build:

sudo launchctl kill -2 system/org.nixos.rosetta-builderd

Then:

Well, it seems not so stable as I'd hoped, as I'm getting hangs in various places, not in one reproducible one.

For example, now it hangs in:

-+= 00001 root /run/current-system/systemd/lib/systemd/systemd
-+= 00937 root nix-daemon --daemon
-+= 00939 root nix-daemon 932
-+= 00993 nixbld1 /run/binfmt/rosetta /nix/store/11ciq72n4fdv8rw6wgjgasfv4mjs1jrw-bash-5.2p37/bin/bash -e /nix/store/vj1c3wf9c11a0qs6p3ymfvrnsdgsdcbq-source-stdenv.sh /nix/store/shkw4qm9qcw5sc5n1k5jznc83
ny02r39-default-builder.sh
-+- 01459 nixbld1 /run/binfmt/rosetta /nix/store/01dapiw4pjrc9s0166dcyg9xkw0rv01z-cmake-3.31.5/bin/cmake .. -GNinja -DCMAKE_FIND_USE_SYSTEM_PACKAGE_REGISTRY=OFF -DCMAKE_FIND_USE_PACKAGE_REGISTRY=OFF -D
CMAKE_EXPORT_NO_PACKAGE_REGISTRY=ON -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTING=OFF -DCMAKE_INSTALL_LOCALEDIR=/nix/store/ks9jcd9iqk3ibxlai07qhsjwsavkndim-mumble-1.5.735/share/locale -DCMAKE_INSTALL_LIBEXECDIR=/
nix/store/ks9jcd9iqk3ibxlai07qhsjwsavkndim-mumble-1.5.735/libexec -DCMAKE_INSTALL_LIBDIR=/nix/store/ks9jcd9iqk3ibxlai07qhsjwsavkndim-mumble-1.5.735/lib -DCMAKE_INSTALL_DOCDIR=/nix/store/ks9jcd9iqk3ibxlai07qhsj
wsavkndim-mumble-1.5.735/share/doc/mumble -DCMAKE_INSTALL_INFODIR=/nix/store/ks9jcd9iqk3ibxlai07qhsjwsavkndim-mumble-1.5.735/share/info -DCMAKE_INSTALL_MANDIR=/nix/store/ks9jcd9iqk3ibxlai07qhsjwsavkndim-mumble
-1.5.735/share/man -DCMAKE_INSTALL_INCLUDEDIR=/nix/store/ks9jcd9iqk3ibxlai07qhsjwsavkndim-mumble-1.5.735/include -DCMAKE_INSTALL_SBINDIR=/nix/store/ks9jcd9iqk3ibxlai07qhsjwsavkndim-mumble-1.5.735/sbin -DCMAKE_
INSTALL_BINDIR=/nix/store/ks9jcd9iqk3ibxlai07qhsjwsavkndim-mumble-1.5.735/bin -DCMAKE_INSTALL_NAME_DIR=/nix/store/ks9jcd9iqk3ibxlai07qhsjwsavkndim-mumble-1.5.735/lib -DCMAKE_POLICY_DEFAULT_CMP0025=NEW -DCMAKE_
FIND_FRAMEWORK=LAST -DCMAKE_STRIP=/nix/store/ii75mhh7sxl11167m1b86p0qrjsjyjmd-gcc-wrapper-14-20241116/bin/strip -DCMAKE_RANLIB=/nix/store/ii75mhh7sxl11167m1b86p0qrjsjyjmd-gcc-wrapper-14-20241116/bin/ranlib -DC
MAKE_AR=/nix/store/ii75mhh7sxl11167m1b86p0qrjsjyjmd-gcc-wrapper-14-20241116/bin/ar -DCMAKE_C_COMPILER=gcc -DCMAKE_CXX_COMPILER=g++ -DCMAKE_INSTALL_PREFIX=/nix/store/ks9jcd9iqk3ibxlai07qhsjwsavkndim-mumble-1.5.
735 -D g15=OFF -D CMAKE_CXX_STANDARD=17 -D CMAKE_UNITY_BUILD=ON -D server=OFF -D overlay=OFF -D bundle-qt-translations=OFF -D bundled-json=OFF -D bundled-gsl=OFF -D update=OFF -D oss=OFF -D warnings-as-errors=
OFF -Dspeechd:BOOL=FALSE -Dpulseaudio:BOOL=TRUE -Dpipewire:BOOL=TRUE -Djackaudio:BOOL=FALSE -Dalsa:BOOL=TRUE -DPROTOC_EXE=/nix/store/ivbs25cdr9mhfrzvamiq824c6xadxsn8-protobuf-29.3/bin/protoc -DProtobuf_PROTOC_
EXE=/nix/store/ivbs25cdr9mhfrzvamiq824c6xadxsn8-protobuf-29.3/bin/protoc -DProtobuf_PROTOC_EXECUTABLE=/nix/store/ivbs25cdr9mhfrzvamiq824c6xadxsn8-protobuf-29.3/bin/protoc
--- 01537 nixbld1 (ld)
-+= 00001 root /run/current-system/systemd/lib/systemd/systemd
-+= 00932 root nix-daemon --daemon
-+= 01145 root nix-daemon 1142
-+= 01336 nixbld1 /run/binfmt/rosetta /nix/store/11ciq72n4fdv8rw6wgjgasfv4mjs1jrw-bash-5.2p37/bin/bash -e /nix/store/vj1c3wf9c11a0qs6p3ymfvrnsdgsdcbq-source-stdenv.sh /nix/store/shkw4qm9qcw5sc5n1k5jznc83
ny02r39-default-builder.sh
-+- 05132 nixbld1 /run/binfmt/rosetta /nix/store/01dapiw4pjrc9s0166dcyg9xkw0rv01z-cmake-3.31.5/bin/cmake .. -GNinja -DCMAKE_FIND_USE_SYSTEM_PACKAGE_REGISTRY=OFF -DCMAKE_FIND_USE_PACKAGE_REGISTRY=OFF -D
CMAKE_EXPORT_NO_PACKAGE_REGISTRY=ON -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTING=OFF -DCMAKE_INSTALL_LOCALEDIR=/nix/store/kxldkscjwnk3zff9v0h5lgg929g53qbx-mumble-overlay-1.5.735/share/locale -DCMAKE_INSTALL_LIBE
XECDIR=/nix/store/kxldkscjwnk3zff9v0h5lgg929g53qbx-mumble-overlay-1.5.735/libexec -DCMAKE_INSTALL_LIBDIR=/nix/store/kxldkscjwnk3zff9v0h5lgg929g53qbx-mumble-overlay-1.5.735/lib -DCMAKE_INSTALL_DOCDIR=/nix/store
/kxldkscjwnk3zff9v0h5lgg929g53qbx-mumble-overlay-1.5.735/share/doc/mumble -DCMAKE_INSTALL_INFODIR=/nix/store/kxldkscjwnk3zff9v0h5lgg929g53qbx-mumble-overlay-1.5.735/share/info -DCMAKE_INSTALL_MANDIR=/nix/store
/kxldkscjwnk3zff9v0h5lgg929g53qbx-mumble-overlay-1.5.735/share/man -DCMAKE_INSTALL_INCLUDEDIR=/nix/store/kxldkscjwnk3zff9v0h5lgg929g53qbx-mumble-overlay-1.5.735/include -DCMAKE_INSTALL_SBINDIR=/nix/store/kxldk
scjwnk3zff9v0h5lgg929g53qbx-mumble-overlay-1.5.735/sbin -DCMAKE_INSTALL_BINDIR=/nix/store/kxldkscjwnk3zff9v0h5lgg929g53qbx-mumble-overlay-1.5.735/bin -DCMAKE_INSTALL_NAME_DIR=/nix/store/kxldkscjwnk3zff9v0h5lgg
929g53qbx-mumble-overlay-1.5.735/lib -DCMAKE_POLICY_DEFAULT_CMP0025=NEW -DCMAKE_FIND_FRAMEWORK=LAST -DCMAKE_STRIP=/nix/store/3mr688gqdc74xw2k82zv1lhdsgwmqw6p-gcc-wrapper-14-20241116/bin/strip -DCMAKE_RANLIB=/n
ix/store/3mr688gqdc74xw2k82zv1lhdsgwmqw6p-gcc-wrapper-14-20241116/bin/ranlib -DCMAKE_AR=/nix/store/3mr688gqdc74xw2k82zv1lhdsgwmqw6p-gcc-wrapper-14-20241116/bin/ar -DCMAKE_C_COMPILER=gcc -DCMAKE_CXX_COMPILER=g+

• -DCMAKE_INSTALL_PREFIX=/nix/store/kxldkscjwnk3zff9v0h5lgg929g53qbx-mumble-overlay-1.5.735 -D g15=OFF -D CMAKE_CXX_STANDARD=17 -D CMAKE_UNITY_BUILD=ON -D server=OFF -D client=OFF -D overlay=ON -DPROTOC_EXE=/n
  ix/store/ivbs25cdr9mhfrzvamiq824c6xadxsn8-protobuf-29.3/bin/protoc -DProtobuf_PROTOC_EXE=/nix/store/ivbs25cdr9mhfrzvamiq824c6xadxsn8-protobuf-29.3/bin/protoc -DProtobuf_PROTOC_EXECUTABLE=/nix/store/ivbs25cdr9m
  hfrzvamiq824c6xadxsn8-protobuf-29.3/bin/protoc
  --- 06187 nixbld1 (pkg-config)

[axman6]

I'm also running into issues with this, where building x86_64-linux packages seem to be just stopping, and I haven't found a way to recover. It seems to always be in the configure phase, but often not in exactly the same spot each time. Let me know if there's any info I can provide to help, this seems to be breaking pretty reliably for me at the moment. (I currently have a build which has three jobs stuck endlessly in the configurePhase, which usually only takes a few seconds).

An example package that is failing for me in a flake is :b inputs.nixpkgs.legacyPackages.x86_64-linux.pkgsStatic.hdf5-cpp (from nixpkgs rev 3016b4b15d13f3089db8a41ef937b13a9e33a8df, if that's helpful)

[booxter]

The only workaround I found for hanging jobs is killing the builder with kill. :(

[dwt]

I have reverted to restarting the builder, that seems to help sometimes. To be honest I haven't experienced this in the last two months, so I was hoping that Apple may have fixed something here.

I have this scripts to deal with that:

❯ cat stop-rosetta-builder -p
#!/bin/sh

sudo launchctl kill -15  system/org.nixos.rosetta-builderd

[dwt]

@cpick We could add this script as an executable script to the rosetta-builder flake, so you can do something like nix run rosetta-builder#stop. That might be useful in a variety of situations actually?

[cpick]

@cpick We could add this script as an executable script to the rosetta-builder flake, so you can do something like nix run rosetta-builder#stop. That might be useful in a variety of situations actually?

Yeah, I think this is a good idea.

In general, I get occasional reports about this issue, but have never been able to reproduce it myself so don’t have much to go on.

[dwt]

@cpick I've added #38 to add the stop script.