Fixing cpptrace JIT frame support

[jeaye]

Hey Jeremy. :) It's been over a year since you tackled #226 to help with jank and here to follow up and say that this is unfortunately no longer working on LLVM 22+. I've asked some LLVM folks about this, to see what may have changed, but I'm not optimistic about getting a clear answer. So instead I've been trying to break this apart myself. I'm hoping that these pieces I've gathered will make some sense to you, if you have the time and interest to take a look. :)

Recap and overview

As a recap, we added a hook into the jank runtime to call this function after giving any new code or object files to LLVM's JIT runtime:

  static void register_jit_stack_frames()
  {
    if(auto *entry = cpptrace::detail::__jit_debug_descriptor.relevant_entry)
    {
      cpptrace::register_jit_object(entry->symfile_addr, entry->symfile_size);
    }
  }

I have a single .jank file which I'm running. It JIT compiles a clojure_core_foo_1_0 function which throws and then calls it immediately. I've upgraded to cpptrace::try_catch and I'm using that to catch the exception and print the stacktrace.

((fn foo [] (throw 'meow)))

The trace comes back with the very first frame not having a symbol:

Stack trace (most recent call first):
#0  0x00007ffff3286092
#1  0x0000555555e3790a in jank::runtime::obj::jit_function::call() const at jit_function.cpp:66
#2  0x00005555557e4b5c in jank::runtime::oref<jank::runtime::object>::call() const at oref.hpp:542
#3  0x00005555565d4d93 in jank::evaluate::eval(jtl::ref<jank::analyze::expr::call>) at evaluate.cpp:336
#4  0x0000555556623ee5 in auto jank::evaluate::eval(jtl::ref<jank::analyze::expression>)::$_0::operator()<jtl::ref<jank::analyze::expr::call> >(jtl::ref<jank::analyze::expr::call>) const at evaluate.cpp:271
#5  0x00005555565d3c3a in auto jank::analyze::visit_expr<jank::evaluate::eval(jtl::ref<jank::analyze::expression>)::$_0>(jank::evaluate::eval(jtl::ref<jank::analyze::expression>)::$_0 const&, jtl::ref<jank::analyze::expression>) at visit.hpp:55
#6  0x00005555565d3a46 in jank::evaluate::eval(jtl::ref<jank::analyze::expression>) at evaluate.cpp:271
#7  0x00005555566fd320 in jank::runtime::context::eval_string(jtl::immutable_string const&, jank::read::source_position const&) const at context_dynamic.cpp:77
#8  0x00005555566fcee0 in jank::runtime::context::eval_string(jtl::immutable_string const&) const at context_dynamic.cpp:150
#9  0x00005555566fcdbf in jank::runtime::context::eval_file(jtl::immutable_string const&) at context_dynamic.cpp:52
#10 0x000055555570c179 in jank::run() at main.cpp:78
#11 0x000055555570c01c in main::$_0::operator()(int, char const**) const at main.cpp:486
#12 0x000055555570b7eb in main::$_0::__invoke(int, char const**) at main.cpp:390
#13 0x00005555566ee92b in jank_init_dynamic::$_0::operator()() const at c_api_dynamic.cpp:45
#25 0x00005555566ee205 in jank_init_dynamic at c_api_dynamic.cpp:34
#26 0x000055555570b770 in main at main.cpp:384

However, we can see from the 0x00007ffff3286092 address space that it's different from the normal 0x0000555555e3790a range for AOT compiled functions.

Troubleshooting

I've put a breakpoint on __jit_debug_register_code to ensure it's being called. It is. I have hacked in a simple print to show if we're registering any entries for this JIT compiled foo function and we are. We're registering just one entry, which makes sense for the one JIT function we're adding.

    void register_jit_object(const char* ptr, std::size_t size) {
        std::printf("register 0x%lx, %zu\n", ptr, size);
        detail::register_jit_object(ptr, size);
    }

Here's the output I get from running the .jank file:

register 0x7fffe7884000, 113296
Uncaught exception: meow

<stack trace from above>

So I wrote that entry out to an object file, starting at 0x7fffe7884000, for 113296 bytes. Inside, we can find the clojure_core_foo_1_0 symbol.

❯ objdump -t /tmp/jit_debug_obj.o  | grep clojure
00007ffff3286000 g     F .ltext	0000000000000093 clojure_core_foo_1_0

We can see from this output that the symbol is at 00007ffff3286000 and is 0000000000000093 bytes. That perfectly matches the first range I'm seeing from llvm-dwarfdump.

❯ llvm-dwarfdump --debug-info /tmp/jit_debug_obj.o | head -40
/tmp/jit_debug_obj.o:	file format elf64-x86-64

.debug_info contents:
0x00000000: Compile Unit: length = 0x000045df, format = DWARF32, version = 0x0005, unit_type = DW_UT_compile, abbr_offset = 0x0000, addr_size = 0x08 (next unit at 0x000045e3)

0x0000000c: DW_TAG_compile_unit
              DW_AT_producer	("clang version 23.0.0git")
              DW_AT_language	(DW_LANG_C_plus_plus_14)
              DW_AT_name	("<<< inputs >>>")
              DW_AT_str_offsets_base	(0x00000008)
              DW_AT_stmt_list	(0x00000000)
              DW_AT_comp_dir	("/home/jeaye/projects/jank/compiler+runtime")
              DW_AT_low_pc	(0x0000000000000000)
              DW_AT_ranges	(indexed (0x0) rangelist = 0x00000010
                 [0x00007ffff3286000, 0x00007ffff3286093) ### Right here.

Summary of findings

When I presented all of this to Claude to see if it thought it was an LLVM issue or cpptrace issue, Claude suggested this is a cpptrace issue.

The throw PC falls exactly inside both the symbol table entry's range and the first DWARF range. Both objdump and llvm-dwarfdump are correctly resolving this unlinked ET_REL object's addresses — objdump by combining the symbol's section-relative st_value with the (already-patched-by-ELFDebugObjectPlugin) section sh_addr, and llvm-dwarfdump by applying the pending ELF relocations against .debug_addr/.debug_rnglists on the fly as it reads.

So: LLVM is emitting correct debug info, ELFDebugObjectPlugin is patching section addresses correctly, and the object is genuinely resolvable by tooling that knows how to process ET_REL relocations at read time. The remaining gap is squarely in cpptrace's own resolver not doing that relocation step when it reads the object you hand it via register_jit_object.

I'm still investigating why this would have changed, but as jank is much more mature now than it was a year ago, I intend on continuously testing that this doesn't regress, once we can get it working again. That way any future issues become clear immediately.

I've attached the object file I've been using, in case that's helpful (had to gzip it to upload).

jit_debug_obj.o.gz

If it's easier to chat on Discord, feel free to reach out! Thanks for any time or attention to this. ❤️

[jeaye]

I vibed a patch with copilot and it works, but I make no claim on its quality and so I would rather just show a diff rather than make a PR. Hopefully this provides the guidance needed in order for you to make a quality fix.

The overview:

This looks like a cpptrace compatibility bug with newer LLVM ORC JIT debug objects, not an LLVM debug-info bug. LLVM’s object is valid, but cpptrace assumes .text and unadjusted symbol semantics that don’t hold for this ET_REL JIT object.

The changes:

1. elf::get_pc_ranges() accepts executable sections instead of only a section literally named .text, so JIT objects with sections like .ltext are registered.
2. ELF symbol loading normalizes ET_REL symbol values by adding the target section’s sh_addr, so live JIT PCs can match symbol table entries.
3. The JIT libdwarf fallback now sets the frame’s raw_address and uses the live JIT PC for symbol lookup, instead of subtracting the section base.

diff --git a/src/binary/elf.cpp b/src/binary/elf.cpp
index 655fa9f..7695635 100644
--- a/src/binary/elf.cpp
+++ b/src/binary/elf.cpp
@@ -151,23 +151,17 @@ namespace detail {
 
     Result<std::vector<elf::pc_range>, internal_error> elf::get_pc_ranges() {
         std::vector<pc_range> vec;
-        auto header_info_ = get_header_info();
-        if(header_info_.is_error()) {
-            return header_info_.unwrap_error();
-        }
-        auto& header_info = header_info_.unwrap_value();
-        auto strtab_ = get_strtab(header_info.e_shstrndx);
-        if(strtab_.is_error()) {
-            return strtab_.unwrap_error();
-        }
-        auto& strtab = strtab_.unwrap_value();
         auto sections_res = get_sections();
         if(!sections_res) {
             return sections_res.unwrap_error();
         }
         const auto& sections = sections_res.unwrap_value();
         for(const auto& section : sections) {
-            if(string_view(strtab.data() + section.sh_name) == ".text") {
+            if(
+                section.sh_size != 0
+                && section.sh_addr != 0
+                && (section.sh_flags & SHF_EXECINSTR) != 0
+            ) {
                 vec.push_back(
                     pc_range{to<frame_ptr>(section.sh_addr), to<frame_ptr>(section.sh_addr + section.sh_size)}
                 );
@@ -250,6 +244,7 @@ namespace detail {
             return internal_error("ELF file header size mismatch {}", file->path());
         }
         header_info info;
+        info.e_type = byteswap_if_needed(file_header.e_type);
         info.e_phoff = byteswap_if_needed(file_header.e_phoff);
         info.e_phnum = byteswap_if_needed(file_header.e_phnum);
         info.e_phentsize = byteswap_if_needed(file_header.e_phentsize);
@@ -294,6 +289,7 @@ namespace detail {
             section_info info;
             info.sh_name = byteswap_if_needed(section_header.sh_name);
             info.sh_type = byteswap_if_needed(section_header.sh_type);
+            info.sh_flags = byteswap_if_needed(section_header.sh_flags);
             info.sh_addr = byteswap_if_needed(section_header.sh_addr);
             info.sh_offset = byteswap_if_needed(section_header.sh_offset);
             info.sh_size = byteswap_if_needed(section_header.sh_size);
@@ -409,6 +405,11 @@ namespace detail {
         // page 32: symtab spec
         static_assert(Bits == 32 || Bits == 64, "Unexpected Bits argument");
         using SymEntry = typename std::conditional<Bits == 32, Elf32_Sym, Elf64_Sym>::type;
+        auto header_info_ = get_header_info();
+        if(header_info_.is_error()) {
+            return std::move(header_info_).unwrap_error();
+        }
+        const auto& header_info = header_info_.unwrap_value();
         auto sections_ = get_sections();
         if(sections_.is_error()) {
             return std::move(sections_).unwrap_error();
@@ -438,6 +439,13 @@ namespace detail {
                     normalized.st_shndx = byteswap_if_needed(entry.st_shndx);
                     normalized.st_value = byteswap_if_needed(entry.st_value);
                     normalized.st_size = byteswap_if_needed(entry.st_size);
+                    if(
+                        header_info.e_type == ET_REL
+                        && normalized.st_shndx != SHN_UNDEF
+                        && normalized.st_shndx < sections.size()
+                    ) {
+                        normalized.st_value += sections[normalized.st_shndx].sh_addr;
+                    }
                     // on arm I've observed zero-size symbols that overlap with symbols we care about
                     // this interferes with some symbol lookup - that could be fixed by enhancing the logic there but
                     // also it's easy to just exclude zero-size symbols here
diff --git a/src/binary/elf.hpp b/src/binary/elf.hpp
index 02c68df..855ede7 100644
--- a/src/binary/elf.hpp
+++ b/src/binary/elf.hpp
@@ -22,6 +22,7 @@ namespace detail {
         bool is_64;
 
         struct header_info {
+            uint16_t e_type;
             uint64_t e_phoff;
             uint32_t e_phnum;
             uint32_t e_phentsize;
@@ -36,6 +37,7 @@ namespace detail {
         struct section_info {
             uint32_t sh_name;
             uint32_t sh_type;
+            uint64_t sh_flags;
             uint64_t sh_addr;
             uint64_t sh_offset;
             uint64_t sh_size;
diff --git a/src/symbols/symbols_with_libdwarf.cpp b/src/symbols/symbols_with_libdwarf.cpp
index 5f83345..5cda14a 100644
--- a/src/symbols/symbols_with_libdwarf.cpp
+++ b/src/symbols/symbols_with_libdwarf.cpp
@@ -91,8 +91,9 @@ namespace libdwarf {
         auto object_res = lookup_jit_object(dlframe.raw_address);
         // TODO: At some point, dwarf resolution
         if(object_res) {
-            frame.frame.symbol = object_res.unwrap().object
-                .lookup_symbol(dlframe.raw_address - object_res.unwrap().base).value_or("");
+            frame.frame.raw_address = dlframe.raw_address;
+            frame.frame.object_address = dlframe.raw_address - object_res.unwrap().base;
+            frame.frame.symbol = object_res.unwrap().object.lookup_symbol(dlframe.raw_address).value_or("");
         }
     }
     #endif

[jeremy-rifkin]

Hi @jeaye thanks for reaching out and for the thorough issue report! I'll have to re-familiarize myself with the elf parsing code and learn about what ORC is doing differently with its elf definitions now, but the patch looks relatively clean. I have regretfully found little time for personal projects as of late but I will try to find some time to go through this in the near future and get it merged for you