From 2d75aa75a72f9c6547cbfa9dabcf56f3fd826a80 Mon Sep 17 00:00:00 2001 From: Alexander Groleau Date: Thu, 24 Aug 2023 10:34:37 -0500 Subject: [PATCH] add packaging techniques recipe --- source/recipes/packaging.md | 235 ++++++++++++++++++++++++++++++++++++ 1 file changed, 235 insertions(+) create mode 100644 source/recipes/packaging.md diff --git a/source/recipes/packaging.md b/source/recipes/packaging.md new file mode 100644 index 000000000..e421a696f --- /dev/null +++ b/source/recipes/packaging.md @@ -0,0 +1,235 @@ +# Packaging: Tools & Techniques +One of Nix's primary use-cases is in addressing common difficulties encountered while packaging software, like *managing dependencies*. + +In the long term, Nix helps tremendously in alleviating that stress, but when *first* (re)packaging existing software with Nix, it's common to encounter missing dependencies preventing builds from succeeding. + +This page covers some tools and techniques for the following: +- tracking down these missing dependencies and including them in your derivations, +- determining whether to use `some_pkg` versus `some_pkg.dev` for missing headers or development libraries, and +- adding platform-specific dependencies. + + +## Dependency Discovery +If you're packaging software you wrote, you should already know what dependencies are needed. + +In this case, you can find the Nix package names for these dependencies by checking the [NixOS Package Search](https://search.nixos.org/packages) site. + +For packaging software other people wrote, check the upstream repository for this information (could be in a Makefile or requirements.txt or something similar), or in the case of prebuilt binaries, check system requirements and other documentation from the upstream. + +Especially for proprietary software, prebuilt binaries will usually come with a list of additional required packages. + +### Helpful Tools +There are a few programs that can be used to find a binary executable's dependencies. + +`ldd` allows for viewing an executable file's shared object dependencies, and is an easy way to determine which libraries are needed to run an existing binary: + +```console +$ ldd ./result/bin/hello + linux-vdso.so.1 (0x00007ffc8938e000) + libc.so.6 => /nix/store/3n58xw4373jp0ljirf06d8077j15pc4j-glibc-2.37-8/lib/libc.so.6 (0x00007f9461d28000) + /nix/store/3n58xw4373jp0ljirf06d8077j15pc4j-glibc-2.37-8/lib/ld-linux-x86-64.so.2 => /nix/store/3n58xw4373jp0ljirf06d8077j15pc4j-glibc-2.37-8/lib64/ld-linux-x86-64.so.2 (0x00007f9461f10000) +``` + +In this case, `hello` requires `linux-vdso`, `libc`, and `ld`. + +The first of these is part of the Linux kernel, while the latter two are part of the `stdenv` for Linux hosts. + +Usually the output is more interesting: + +```console +$ ldd /nix/store/sspk2i8rcj5h82zm2gd30xih5il2bw7l-emacs-28.2/bin/emacs + linux-vdso.so.1 (0x00007ffdb8592000) + libXcursor.so.1 => /nix/store/29zrpf3c54d5zs1hj7bdzf97l9m2ay6h-libXcursor-1.2.0/lib/libXcursor.so.1 (0x00007f7b245b3000) + libtiff.so.6 => /nix/store/j7zyv8fn0g2kb4bpa9li7i7akpkp07jw-libtiff-4.5.0/lib/libtiff.so.6 (0x00007f7b2452d000) + libjpeg.so.62 => /nix/store/yxgca61a6rwhdhpbh966r8v0hzvfsyzj-libjpeg-turbo-2.1.5.1/lib/libjpeg.so.62 (0x00007f7b2447e000) + libpng16.so.16 => /nix/store/m9dqylpdc52yw6mg6sq88w6zwfs6dw87-libpng-apng-1.6.39/lib/libpng16.so.16 (0x00007f7b24445000) + ... (87 lines omitted) +``` + +In this example, `ldd` output many dependencies on the left, along with their present link location in the Nix store on the right. + +Users of Linux systems other than NixOS will see different but similar output. + +In both cases, the Nix package name of the dependency listed on the left can be found by searching the [NixOS Package Search](https://search.nixos.org/packages) site. + +:::{note} +`ldd` is not available on Darwin systems, but the `otool -L` command serves a similar purpose in that context. +::: + +## Including Headers and Libraries +Because everything is a derivation in Nix, there is no distinction between headers, libraries, and anything else *from the perspective of package management*: everything is available in the same way from nixpkgs. + +Of course, how you can actually *use* dependencies is another matter, and highly context-dependent. + +Some derivations, sometimes called "split" derivations, contain multiple outputs like `lib`, `man`, and `bin`, respectively representing the runtime libraries, the manual pages, and compiled binaries. + +### Library Include Dirs +When building a package from source, you often need to make subdirectories of some library packages available to the build system. + +There are several ways to do this, but the best two options involve adding the path to the relevant library directories to `$PATH` or another environment variable, and either: +- (re-) exporting this via a bash command within `mkDerivation`'s `buildPhase` attribute, or +- letting Nix make the variable available in the build environment by declaring the environment as a new attribute in `mkDerivation`. + +There are valid reasons to one or the other of these, but here we'll use the latter, after a brief interlude to examine some useful functions from nixpkgs. + +#### Utility Functions +`lib.strings` in nixpkgs contains a family of handy functions for constructing Unix-style colon-separated path strings, used by build tools to search for dependencies. + +A description of each is given below, along with sample `nix repl` output showing their use. + +You'll often want to reach for these when packaging your software. + +:::{note} +The last positional argument of all of these functions is a list containing strings representing filesystem paths, packages, or both. +::: + +##### `makeLibraryPath` +This function returns a string containing the paths to the library subdirectory of everything in the given list. + +```console +nix-repl> makeLibraryPath [ "/usr" "/usr/local" ] +=> "/usr/lib:/usr/local/lib" +nix-repl> makeLibraryPath [ pkgs.openssl pkgs.zlib ] +=> "/nix/store/zqhfl96qd58z9whx865sx1njnm2kxryi-openssl-3.0.5/lib:/nix/store/35236f66yj5n3ams20w873fkddlfvghm-zlib-1.2.12/lib" +``` + +##### `makeBinPath` +Similar to `makeLibraryPath`, `makeBinPath` constructs a search path for the `/bin` subdirectory of everything in the given list. + +```console +nix-repl> makeBinPath ["/usr" "/usr/local"] +=> "/usr/bin:/usr/local/bin" +nix-repl> makeBinPath [pkgs.openssl pkgs.zlib] +=> "/nix/store/cr1998fn6rjmhrhwylqzq021zhn54zhp-openssl-3.0.5-bin/bin:/nix/store/35236f66yj5n3ams20w873fkddlfvghm-zlib-1.2.12/bin" +``` + +##### `makeSearchPath` +This function takes an additional string argument, which contains the name of a subdirectory to append to the paths of each element in the list. + +```console +nix-repl> makeSearchPath "bin" [ pkgs.openssl pkgs.zlib ] +=> "/nix/store/cr1998fn6rjmhrhwylqzq021zhn54zhp-openssl-3.0.5-bin/bin:/nix/store/35236f66yj5n3ams20w873fkddlfvghm-zlib-1.2.12/bin" +nix-repl> makeSearchPath "bin" [ pkgs.openssl pkgs.zlib ] +"/nix/store/cr1998fn6rjmhrhwylqzq021zhn54zhp-openssl-3.0.5-bin/bin:/nix/store/35236f66yj5n3ams20w873fkddlfvghm-zlib-1.2.12/bin" +``` + +##### `makeSearchPathOutput` +In addition to the subdirectory name argument of `makeSearchPath`, `makeSearchPathOutput` also takes a string containing the name of an output attribute of a derivation, appending the subdirectory to the path of that output. + +This function and `makeSearchPath` are especially handy when working with split derivations. + +```console +nix-repl> makeSearchPathOutput "dev" "bin" [ pkgs.openssl pkgs.zlib ] +"/nix/store/iwigjmzk8nailq789i0sd35cc9lyqwml-openssl-3.0.5-dev/bin:/nix/store/45nwws8pg1qdr2s14vf2na1wwddj3bca-zlib-1.2.12-dev/bin" +``` + +## Platform-specific Dependencies +Every hardware and software platform comes with unique quirks, some of which impact the experience of building software targeting those platforms, necessitating additional dependencies not required elsewhere. Some examples: +- many compilers on aarch64-darwin and x86_64-darwin must include [Frameworks from Apple's SDKs](https://developer.apple.com/library/archive/documentation/MacOSX/Conceptual/BPFrameworks/Concepts/WhatAreFrameworks.html) +- tasks in computationally-intensive areas like machine learning, numerical computing, and signal processing see significant speedup from hardware acceleration, with specialty devices requiring certain drivers +- producing self-contained, statically-compiled binaries often involves the use of different toolchains, like those targeting musl libc. + +Determining _which_ of these dependencies are needed can occasionally be a challenge, as non-Linux platforms — particularly _newer_ ones like aarch64-darwin — tend to be less well-documented, and sometimes only supported experimentally. + +### Darwin +As previously mentioned, Nix packages on macOS (aarch64-darwin, x86_64-darwin) often require functions from Apple's SDK Frameworks, available from Nixpkgs as `pkgs.darwin.apple_sdk.frameworks`. + +There are other important differences, however. + +- The Darwin `stdenv` uses `clang` instead of `gcc` (though `gcc` is still available for Darwin systems). + +When referring to the compiler within a derivation, `$CC` or `cc` will work in both cases. + +Some builds hardcode `gcc` or `g++` in their build scripts, which can usually be fixed by adding `makeFlags = [ "CC=cc" ];`, or by patching the build scripts directly. + +```nix +stdenv.mkDerivation { + name = "libfoo-1.2.3"; + # ... + buildPhase = '' + $CC -o hello hello.c + ''; +} +``` + +- On Darwin, libraries are linked using absolute paths, and are resolved by their `install_name` at link time. + +Sometimes packages won’t set this correctly, which causes the library lookups to fail at runtime. + +This can be fixed by adding extra linker flags pointing to the correct library paths, or by running `install_name_tool -id` during the `fixupPhase`. + +```nix +stdenv.mkDerivation { + name = "libfoo-1.2.3"; + # ... + makeFlags = lib.optional stdenv.isDarwin "LDFLAGS=-Wl,-install_name,$(out)/lib/libfoo.dylib"; +} +``` + +- Even if the libraries are linked using absolute paths and resolved correctly via their `install_name`, tests can sometimes fail to run binaries, because the `checkPhase` runs before the libraries are installed. + +This can usually be solved by running the tests after the `installPhase`, or by using `DYLD_LIBRARY_PATH`. + +More information about this variable can be found in the *dyld(1)* manpage. + +```console +dyld: Library not loaded: /nix/store/7hnmbscpayxzxrixrgxvvlifzlxdsdir-jq-1.5-lib/lib/libjq.1.dylib +Referenced from: /private/tmp/nix-build-jq-1.5.drv-0/jq-1.5/tests/../jq +Reason: image not found +./tests/jqtest: line 5: 75779 Abort trap: 6 +``` + +```nix +stdenv.mkDerivation { + name = "libfoo-1.2.3"; + # ... + doInstallCheck = true; + installCheckTarget = "check"; +} +``` + +- Some packages assume Xcode is available and use `xcrun` to resolve build tools like `clang`. + +This causes errors like `xcode-select: error: no developer tools were found at '/Applications/Xcode.app'`, even though the build doesn’t actually depend on Xcode. + +```nix +stdenv.mkDerivation { + name = "libfoo-1.2.3"; + # ... + prePatch = '' + substituteInPlace Makefile \ + --replace '/usr/bin/xcrun clang' clang + ''; +} +``` + +The `xcbuild` package can be used to build projects that *do* depend on Xcode, but the replacement is not fully compatible with Xcode and can occasionally cause issues. + +`x86_64-darwin` uses the 10.12 SDK by default, but some software is not compatible with that version of the SDK. + +In that case, the 11.0 SDK used by aarch64-darwin is available for use on x86_64-darwin. + +To use it, reference `apple_sdk_11_0` instead of `apple_sdk` in your derivation and use `pkgs.darwin.apple_sdk_11_0.callPackage` instead of `pkgs.callPackage`. + +On Linux, this will have the same effect as `pkgs.callPackage`, so you can use `pkgs.darwin.apple_sdk_11_0.callPackage` regardless of platform. + +### Including Platform-specific Dependencies +The Nix standard library contains a few helper functions useful for conditionally-including platform-specific dependencies. + +Two of the most commonly used are `lib.optionals` and `lib.optionalString`, both of which take a boolean argument in the first position, followed by a string in the case of `lib.optionalString`, or any kind of attribute in the case of `lib.optionals`. + +Here's an example using both of these to optionally include some packages and dynamically-linked libraries within a derivation: + +```nix +... + +frameworks = pkgs.darwin.apple_sdk.frameworks; +darwinInputs = lib.optionals pkgs.stdenv.isDarwin (with pkgs; [ libiconv frameworks.Security ]); +nixLdFlags = lib.optionalString pkgs.stdenv.isDarwin "-F${frameworks.CoreServices}/Library/Frameworks -framework CoreServices -L${pkgs.libiconv}/lib"; + +buildInputs = (with pkgs; [ gcc cmake ]) ++ darwinInputs; + +NIX_LD_FLAGS = nixLdFlags; + +... +```