Issues with `_NT_TIB` object in Volshell

[the-rectifier]

While working at #1816, I encountered an issue in Volshell when parsing the _NT_TIB structures. I don't know if this is for this specific object combination, but as @eve-mem mentioned, this deserves its own issue.

Currently, if we dereference and cast _KTHREAD.Teb into _NT_TIB (first entry of _TEB is _NT_TIB), it results in a malformed object:

(layer_name_Process4988) >>> dt(nt_tib)
symbol_table_name1!_NT_TIB (56 bytes) @ 0x82bf405000:
   0x0 :   ExceptionList            *symbol_table_name1!_EXCEPTION_REGISTRATION_RECORD     N/A
   0x8 :   StackBase                *symbol_table_name1!void                               N/A
  0x10 :   StackLimit               *symbol_table_name1!void                               N/A
  0x18 :   SubSystemTib             *symbol_table_name1!void                               N/A
  0x20 :   FiberData                *symbol_table_name1!void                               N/A
  0x20 :   Version                  symbol_table_name1!unsigned long                       N/A
  0x28 :   ArbitraryUserPointer     *symbol_table_name1!void                               N/A
  0x30 :   Self                     *symbol_table_name1!_NT_TIB                            N/A
(layer_name_Process4988) >>>

To reproduce, try the following script (change the pid argument accordingly):

kernel = self.context.modules[self.config["kernel"]]
kernel_layer = self.context.layers[kernel.layer_name]
proc = gp(pid=4988)

proc_layer_name = proc.add_process_layer()
proc_layer = self.context.layers[proc_layer_name]

cl(proc_layer_name)

thread_list = list(
    proc.ThreadListHead.to_list(
        f"{kernel.symbol_table_name}!_ETHREAD", "ThreadListEntry"
    )
)

active_thread_list = [t for t in thread_list if t.ExitTime.QuadPart < 0]
main_thread = active_thread_list[0]
nt_tib = main_thread.Tcb.Teb.dereference().cast(f'{kernel.symbol_table_name}!_NT_TIB')

assert(main_thread.Tcb.Teb == nt_tib.vol.offset)

# this works
# StackBase is @ NtTib + 8
layer_read = proc_layer.read(nt_tib.vol.offset + 8, 8)
print(layer_read.hex())

# this fails
nt_tib.StackBase

As you will see, accessing the attribute .StackBase will cause a PagedInvalidAddressException while directly reading from the layer will be fine.

Full run:

Volshell (Volatility 3 Framework) 2.26.2
Readline imported successfully  PDB scanning finished
Running code from file:///home/canopus/dev/diss/volatility3/inter.py

000032bf82000000
Traceback (most recent call last):
  File "<input>", line 28, in <module>
  File "/home/canopus/dev/diss/volatility3/volatility3/framework/objects/__init__.py", line 985, in __getattr__
    member = template(context=self._context, object_info=object_info)
  File "/home/canopus/dev/diss/volatility3/volatility3/framework/objects/templates.py", line 96, in __call__
    return self.vol.object_class(
           ~~~~~~~~~~~~~~~~~~~~~^
        context=context, object_info=object_info, **arguments
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/canopus/dev/diss/volatility3/volatility3/framework/objects/__init__.py", line 168, in __new__
    value = cls._unmarshall(context, data_format, object_info)
  File "/home/canopus/dev/diss/volatility3/volatility3/framework/objects/__init__.py", line 407, in _unmarshall
    cls._get_raw_value(
    ~~~~~~~~~~~~~~~~~~^
        context, data_format, object_info.layer_name, object_info.offset
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/canopus/dev/diss/volatility3/volatility3/framework/objects/__init__.py", line 425, in _get_raw_value
    data = context.layers.read(layer_name, offset, length)
  File "/home/canopus/dev/diss/volatility3/volatility3/framework/interfaces/layers.py", line 635, in read
    return self[layer].read(offset, length, pad)
           ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^
  File "/home/canopus/dev/diss/volatility3/volatility3/framework/layers/linear.py", line 45, in read
    for offset, _, mapped_offset, mapped_length, layer in self.mapping(
                                                          ~~~~~~~~~~~~^
        offset, length, ignore_errors=pad
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ):
    ^
  File "/home/canopus/dev/diss/volatility3/volatility3/framework/layers/intel.py", line 307, in mapping
    for offset, size, mapped_offset, mapped_size, map_layer in self._mapping(
                                                               ~~~~~~~~~~~~~^
        offset, length, ignore_errors
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ):
    ^
  File "/home/canopus/dev/diss/volatility3/volatility3/framework/layers/intel.py", line 363, in _mapping
    chunk_offset, page_size, layer_name = self._translate(offset)
                                          ~~~~~~~~~~~~~~~^^^^^^^^
  File "/home/canopus/dev/diss/volatility3/volatility3/framework/layers/intel.py", line 515, in _translate
    return self._translate_swap(self, offset, self._bits_per_register // 2)
           ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/canopus/dev/diss/volatility3/volatility3/framework/layers/intel.py", line 462, in _translate_swap
    return super()._translate(offset)
           ~~~~~~~~~~~~~~~~~~^^^^^^^^
  File "/home/canopus/dev/diss/volatility3/volatility3/framework/layers/intel.py", line 162, in _translate
    entry, position = self._translate_entry(offset & self.page_mask)
                      ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/canopus/dev/diss/volatility3/volatility3/framework/layers/intel.py", line 217, in _translate_entry
    raise exceptions.PagedInvalidAddressException(
    ...<5 lines>...
    )
volatility3.framework.exceptions.PagedInvalidAddressException: Page Fault at entry 0x0 in table page directory pointer

Code complete

Call help() to see available functions

Volshell mode        : Windows
Current Layer        : layer_name
Current Symbol Table : symbol_table_name1
Current Kernel Name  : kernel

(layer_name_Process4988) >>>

[ikelos]

Volatility is functioning as expected:

You construct a process layer that you then proceed to never use:

proc_layer_name = proc.add_process_layer()
proc_layer = self.context.layers[proc_layer_name]

You extract a process from the kernel layer because the process object was constructed on the kernel layer (see get_process if needed) and it has no idea what that pointer is or does, and then you tell it it's a doubly linked list of pointers still in the kernel layer, because why would it be anywhere else, so you've now got pointers that apparently point into the kernel layer, and that's why they dereference incorrectly.

thread_list = list(
    proc.ThreadListHead.to_list(
        f"{kernel.symbol_table_name}!_ETHREAD", "ThreadListEntry"
    )
)

Volatility can't know which layer you wanted those pointers to point into, structures (compile time information) have no way of saying which layer they point into (runtime information) so they need to be told by user when they don't point to the layer they were constructed on.

Instead, you need to tell volatility that you know which layer those pointers point into, and you can do that by providing the layer parameter to the to_list method. (Unfortunately this feature was made a long time ago, so it should really be called layer_name rather than layer, but it takes a string and operates as if it were called layer_name). It will construct the object of the list on that layer.

As such, swapping out for the following code will give you exactly what you expected and everything works just fine:

thread_list = list(
    proc.ThreadListHead.to_list(
        f"{kernel.symbol_table_name}!_ETHREAD", "ThreadListEntry", layer=proc_layer_name
    )
)

As such I'm going to mark this as closed, but if there's anything you think is still outstanding, feel free to reopen it. 5:)

[the-rectifier]

Hey @ikelos, thanks for replying! I tried adding the layer argument, and it actually straightens out everything.

I had no idea objects were so 'convoluted' together. So let me get this clear. By omitting the layer argument, each _ETHREAD in the list is created on the kernel layer. Thus, every cast/dereference will still produce objects on the kernel layer(?).

This is why directly reading from the proc_layer is fine, but accessing it fails.

This was my initial thought as well, which I tried remediating by changing to the proc_layer (I thought accessing attributes would automatically use the current layer, hence the cl(proc_layer_name) in the beginning.)

[ikelos]

Volshell's curent layer is only used by volshell functions. Normal plugins don't have a default value for a layer and need to be told it at every point (or assume it to be the same layer as the parent).

You can cast an object onto a different layer. You need to think about the layer_name (which layer the data is read from, for instance the physical layer in the case of scanners) and the native_layer_name (which layer it "natively" lives in, and therefore which layer things it references live in). It's mostly relevant for structures that cross reference in some way, but that's the reason the native_layer concept was introduced. Unless the native_layer differs from the layer_name or there's an explicit cast, the chain of layers down through substructures will stay the same.

Since to_list is often used with pointers (and those pointers may have different native_layers to the ones they were constructed on) we provide the layer parameter, rather than having to cast each one individually each time. It should keep everything as everyone expects.

I think casting at the ThreadListHead level is right? I think casting it at the main_thread.Tcb.Teb.dereference() point would only work because there isn't a pointer in that chain that needed to be on a different layer. If there were it'd start reading random (possibly inaccessible) memory from the wrong layer...

[the-rectifier]

This is really informative, thank you! I will amend the code in the mentioned PR as well!

[eve-mem]

Thanks for looking into this @ikelos and @the-rectifier - I finally got time to sit down and you've already worked it out. Thank you <3