Fix nix build for c2rust

[Chobbes]

This commit adds a number of improvements to the nix build for c2rust the main highlights are:

• A c2rust package is provided, allowing nix users to install and run c2rust itself, instead of just getting a development environment from nix.
• Update LLVM / clang to version 18.
• Test cases pass in the c2rust dev environment now.

Some fixes include...

• Removing oxalica and just using fenix. Mixing rust overlays in nix was causing issues.
• Use the nix tinycbor package to build c2rust. The c2rust cmake files add an external dependency on tinycbor, which causes sandboxed nix builds to fail because they cannot fetch the package. This involves patching the CMakeLists.txt file, and using pkg-config to find the system tinycbor library. Note: in the dev shell cmake will still fetch the external tinycbor.
• Use the C2RUST_TINYCBOR_PATH environment variable, if it exists, to find tinycbor in c2rust-ast-exporter's build.rs
• Use bindgenHook to make sure Rust's bindgen can find header files.
• Update the test_translator.py script to include nix flags in compiler_commands.json Messing with bindgenHook, disable checks so package builds.

[kkysen]

Thanks for the improvements here! To reiterate what I've told some other people, we don't have the resources to maintain and review any of the Nix-related code here, but we're happy to accept fixes. I've reviewed the parts of this that touch the main c2rust files (build.rs and test_translator.py), but I haven't looked at the nix-specific files.

Also, I'm not sure if you've seen @RossSmyth's #1285, so I just wanted to make sure the two of you are aware of each other's nix changes and can coordinate anything necessary.

[Chobbes]

@kkysen thanks for the review! I have pushed the fixes for most of the things you brought up, and I left a couple of them unresolved in order to get your opinion. The main one is the use of the environment variable for the test script --- I think this is actually the right thing to do here because nix will automatically set the environment variable for you, which I think is better than having an argument to the test script that you have to know to set on nix systems. The other is a stylistic concern, which I'm happy to defer to your judgement entirely on.

Yes, I had gotten this to work under nix a week or so ago with all of the test cases passing, and just got the time to clean this up on Friday. Only then did I see @RossSmyth's #1285, which seems to have had activity from between me getting things working locally and cleaning things up and making my PR. I commented on that PR when I noticed, but haven't heard back yet.

[Rua]

I tried this out and got this:

$ nix run github:Chobbes/c2rust/nix-fixed -- transpile -o . --overwrite-existing compile_commands.json
i_sound.c:27:10: fatal error: 'errno.h' file not found

Building c2rust succeeds, but while running the transpile it can't find C's header files.

[Chobbes]

I tried this out and got this:

$ nix run github:Chobbes/c2rust/nix-fixed -- transpile -o . --overwrite-existing compile_commands.json
i_sound.c:27:10: fatal error: 'errno.h' file not found

Building c2rust succeeds, but while running the transpile it can't find C's header files.

Yes! This is expected (though maybe not ideal) and the compile_commands.json needs to have the paths to search for the libraries. For example:

"arguments": [ "cc", "-D_FORTIFY_SOURCE=0", "-I/nix/store/y3lzh1bym1q2h0d09byc6lxsiya3izqm-clang-wrapper-18.1.8/resource-root/include","-I/nix/store/74qjr01q87nwfl0dbsr1s45p8crw3q1f-glibc-2.40-66-dev/include", "-I/nix/store/qnwxpk0in4bm43q2qnykvkjxa9qhqd0z-gcc-14.3.0/include/c++/14.3.0", "-I/nix/store/qnwxpk0in4bm43q2qnykvkjxa9qhqd0z-gcc-14.3.0/include/c++/14.3.0/x86_64-unknown-linux-gnu", "-I/nix/store/pqy858gyg7d8c9p7k587023sardl4wfh-compiler-rt-libc-18.1.8-dev/include", "-I/nix/store/pqy858gyg7d8c9p7k587023sardl4wfh-compiler-rt-libc-18.1.8-dev/include", "-I/nix/store/3yb3a900z5r0x678d8pp46gwab8bbki5-clang-18.1.8-dev/include", "-I/nix/store/3yb3a900z5r0x678d8pp46gwab8bbki5-clang-18.1.8-dev/include", "-I/nix/store/fhdpk4fpgpdkkf8pdl6px5lmnp42z3fz-llvm-18.1.8-dev/include", "-I/nix/store/fhdpk4fpgpdkkf8pdl6px5lmnp42z3fz-llvm-18.1.8-dev/include", "-I/nix/store/z1l05nn4xyaxv25f9pvi7bkmw6jmb48c-ncurses-6.5-dev/include", "-I/nix/store/z1l05nn4xyaxv25f9pvi7bkmw6jmb48c-ncurses-6.5-dev/include", "-I/nix/store/qjh11kgxv245166f27017hwx2fvmwh8p-zlib-1.3.1-dev/include", "-I/nix/store/qjh11kgxv245166f27017hwx2fvmwh8p-zlib-1.3.1-dev/include", "-I/nix/store/7qggxrwirl36a4cgab3q5kzbira2gbqi-tinycbor-0.6.1/include", "-I/nix/store/7qggxrwirl36a4cgab3q5kzbira2gbqi-tinycbor-0.6.1/include", "-I/nix/store/b97gjl5zygaf6vzxsa1lf7jih8q4xi00-openssl-3.5.1-dev/include", "-I/nix/store/b97gjl5zygaf6vzxsa1lf7jih8q4xi00-openssl-3.5.1-dev/include", "-I/nix/store/pqy858gyg7d8c9p7k587023sardl4wfh-compiler-rt-libc-18.1.8-dev/include", "-I/nix/store/pqy858gyg7d8c9p7k587023sardl4wfh-compiler-rt-libc-18.1.8-dev/include", "-I/nix/store/3yb3a900z5r0x678d8pp46gwab8bbki5-clang-18.1.8-dev/include", "-I/nix/store/3yb3a900z5r0x678d8pp46gwab8bbki5-clang-18.1.8-dev/include", "-I/nix/store/fhdpk4fpgpdkkf8pdl6px5lmnp42z3fz-llvm-18.1.8-dev/include", "-I/nix/store/fhdpk4fpgpdkkf8pdl6px5lmnp42z3fz-llvm-18.1.8-dev/include", "-I/nix/store/z1l05nn4xyaxv25f9pvi7bkmw6jmb48c-ncurses-6.5-dev/include", "-I/nix/store/z1l05nn4xyaxv25f9pvi7bkmw6jmb48c-ncurses-6.5-dev/include", "-I/nix/store/qjh11kgxv245166f27017hwx2fvmwh8p-zlib-1.3.1-dev/include", "-I/nix/store/qjh11kgxv245166f27017hwx2fvmwh8p-zlib-1.3.1-dev/include", "-I/nix/store/7qggxrwirl36a4cgab3q5kzbira2gbqi-tinycbor-0.6.1/include", "-I/nix/store/7qggxrwirl36a4cgab3q5kzbira2gbqi-tinycbor-0.6.1/include", "-I/nix/store/b97gjl5zygaf6vzxsa1lf7jih8q4xi00-openssl-3.5.1-dev/include", "-I/nix/store/b97gjl5zygaf6vzxsa1lf7jih8q4xi00-openssl-3.5.1-dev/include", "-c", "variable_arrays.c" ],

You can pull these from the BINDGEN_EXTRA_CLANG_ARGS and NIX_CFLAGS_COMPILE environment variables. This is done for the compile_commands.json in the test suite:

            nix_compile_flags = os.environ.get("BINDGEN_EXTRA_CLANG_ARGS", "") + os.environ.get("NIX_CFLAGS_COMPILE", "")
            nix_compile_flags = nix_compile_flags.replace("-isystem ", "-I").replace("-idirafter ", "-I")
            nix_compile_flags = ['"{}"'.format(flag) for flag in nix_compile_flags.split() if flag.startswith("-I")]
            nix_compile_flags = ", ".join(nix_compile_flags) + ","
            self.clang_extra_flags += nix_compile_flags

I'm not sure if there's a better solution in general, there's some discussions about this problem in nix:

• https://discourse.nixos.org/t/using-nix-infrastructure-to-reliably-generate-compile-commands-json/21961/10

I'm not super familiar with the whole compile_commands.json thing, it seems like it makes sense to specify where the libraries are in nix there, but maybe there's a way to provide a wrapper for c2rust that would be more convenient.

[Rua]

I added the flags in NIX_CFLAGS_COMPILE to the JSON (replacing with -I as needed like in the script) and that got it a bit further. But it can't find stddef.h now. I guess that file is in another location not listed?

Also, BINDGEN_EXTRA_CLANG_ARGS appears to be empty on my system.

[Chobbes]

I added the flags in NIX_CFLAGS_COMPILE to the JSON (replacing with -I as needed like in the script) and that got it a bit further. But it can't find stddef.h now. I guess that file is in another location not listed?

Also, BINDGEN_EXTRA_CLANG_ARGS appears to be empty on my system.

You might want to also add the -I flags for clang --print-resource-dir. You'll need the BINDGEN_EXTRA_CLANG_ARGS from the dev shell. You can also set NIX_DEBUG=1 and compile your file and it should print all of the nix paths and you can use that.

[Rua]

Ok, finally figured it out! Turns out I was almost there, I just needed to change this arg

-I/nix/store/qnwxpk0in4bm43q2qnykvkjxa9qhqd0z-gcc-14.3.0/lib/gcc/x86_64-unknown-linux-gnu/14.3.0/include-fixed

to say include at the end instead.

[Chobbes]

Ok, finally figured it out! Turns out I was almost there, I just needed to change this arg

-I/nix/store/qnwxpk0in4bm43q2qnykvkjxa9qhqd0z-gcc-14.3.0/lib/gcc/x86_64-unknown-linux-gnu/14.3.0/include-fixed

to say include at the end instead.

Fantastic! Yeah, I'm not sure, but it might make sense to be able to add some flags for the includes to c2rust directly, and then we can make a wrapper like the nix clang wrapper that automatically includes some of the system libraries. I figured for now it makes sense to just explicitly include all of the stuff because thats sort of what it seems like you're supposed to do, but it's definitely not the most convenient and the default compile_commands.json generated by the c2rust translator tests leaves some things implicit, I think.

[kkysen]

A couple little things. I thought I sent this earlier but I guess I forgot to click submit.

[Rua]

Is this ready to be merged now? I'm on Nix, and have a few PRs going that are really awkward to work with without these fixes! :D

[Rua]

@Chobbes When running some of the tests, I get errors saying it can't find stdlib.h. Is that something you can fix?

EDIT: I think it's in this function somewhere, which generates compile commands for the tests. Since your changes added Nix-specific options to the compile commands elsewhere, I figure they need to be added here too?
https://github.com/immunant/c2rust/blob/master/c2rust-transpile/src/lib.rs#L246-L283

[kkysen]

Thanks! LGTM now!

[kkysen]

Is this ready to be merged now? I'm on Nix, and have a few PRs going that are really awkward to work with without these fixes! :D

Yes, I think so.

@Chobbes When running some of the tests, I get errors saying it can't find stdlib.h. Is that something you can fix?

Unless something is broken. If it's all good, I can merge.

[Rua]

It seems like it might be a Nix-specific issue relating to the tests. The PR as it stands now is an improvement for sure, and I'd be happy to see it merged now, but hopefully Chobbes will look into this problem afterwards.

[Chobbes]

@Rua I can't look today, but I can look on Monday. All of the tests were passing for me with cargo in the dev shell, though. How are you running the tests?

[Rua]

$ nix develop
$ cargo clean
$ cargo test -p c2rust-transpile --release
   [Compiling stuff...]
    Finished release [optimized] target(s) in 45.39s
     Running unittests src/lib.rs (target/release/deps/c2rust_transpile-109e8649cb12d2ca)

running 5 tests
test renamer::tests::forgets ... ok
test renamer::tests::scoped ... ok
test renamer::tests::simple ... ok
test rust_ast::validate_repr ... ok
test c_ast::tests::test_compare_src_locs_ord ... ok

test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running tests/snapshots.rs (target/release/deps/snapshots-61d349fc3a38e06a)

running 1 test
/home/rua/code/c2rust/c2rust-transpile/tests/snapshots/arrays.c:1:10: fatal error: 'stdlib.h' file not found
    1 | #include <stdlib.h>
      |          ^~~~~~~~~~
1 error generated.
Error while processing /home/rua/code/c2rust/c2rust-transpile/tests/snapshots/arrays.c.
error: multiple input filenames provided (first two filenames are `+nightly-2022-08-08` and `tests/snapshots/arrays.rs`)

test transpile_all ... FAILED

failures:

---- transpile_all stdout ----
Transpiling arrays.c
thread 'transpile_all' panicked at 'assertion failed: status.unwrap().success()', c2rust-transpile/tests/snapshots.rs:125:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace


failures:
    transpile_all

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.19s

error: test failed, to rerun pass '-p c2rust-transpile --test snapshots'

[Chobbes]

@Rua thanks! I guess I had been running the ./scripts/test_translator.py script directly after running cargo build --release, and I'm not sure how that all plugs into cargo test. That's probably where the differences are arising, though.

[Chobbes]

@Rua ah, yeah, okay, I didn't know about these cargo test cases. It looks like these rely on the create_temp_compile_commands function in c2rust-transpile/src/lib.rs file to create the compile_commands.json file. I think this needs to be modified in a similar way to the python test script, or maybe there's a better way to wrap things so we don't need to do that. I'm not sure what the best approach is.

[kkysen]

@Rua ah, yeah, okay, I didn't know about these cargo test cases. It looks like these rely on the create_temp_compile_commands function in c2rust-transpile/src/lib.rs file to create the compile_commands.json file. I think this needs to be modified in a similar way to the python test script, or maybe there's a better way to wrap things so we don't need to do that. I'm not sure what the best approach is.

Feel free to remove that create_temp_compile_commands in snapshots.rs and replace it with a deliberately created compile_commands.json. We've been meaning to do that for a few things (easier to compile as a crate so we can use dependencies and so we can suppress C warnings), so if that makes it easier for you, too, win-win.