Use P/Invoke to call lstat instead of shelling out to stat to retrieve the inode (#7453)

## Summary of changes

Adds a P/Invoke path for retrieving the inode on linux and only later
falls back to using `stat -c %i <path>`

## Reason for change

Calling `stat` will fail if `stat` isn't available (as in chiseled
containers, for example). It also results in additional spans being
created when we're running in a version-mismatch scenario(because the
`Process` invocation is traced)

## Implementation details

Based my initial PoC on P/Invoking into `System.Native`, which is a shim
shipped with recent .NET that handles a bunch of marshalling issues etc
and returns [this
type](https://github.com/dotnet/runtime/blob/main/src/libraries/Common/src/Interop/Unix/System.Native/Interop.Stat.cs).
That's [how recent .NET
versions](https://github.com/dotnet/runtime/blob/main/src/libraries/System.Private.CoreLib/src/System/IO/FileStatus.Unix.cs#L498)
handle various bits like this.

However, System.Native only gives the correct inode value in .NET 7+
(from my local testing) and isn't available in early .NET Core. I then
tried the "raw" route, trying to P/Invoke directly into `libc`. I also
investigated making the syscall directly, but libc papers over
differences in old kernels etc, so it seemed the easiest route. However,
older versions of `libc` don't export the `lstat` or `fstatat` symbols
as public symbols, and instead they're just macros.

Finally settled on a similar approach to System.Native, i.e. we P/Invoke
into Datadog.Trace.Native which then makes the `lstat` call.

## Test coverage

Should _mostly_ be covered by existing tests (and smoke tests). I added
an additional unit test to make sure the `stat -c %i` call and the
P/Invoke give the same value, otherwise we should be covered.

I also rebased on top of [this
branch](#7446) which is
where the error appeared - and did a [full-installer
run](https://dev.azure.com/datadoghq/dd-trace-dotnet/_build/results?buildId=186579&view=results),
and it fixed all the issues, so we should be good 🤞 And we still have
the existing fallback if not

Note that I explicitly disabled the process spans in the
version-conflict tests, as whether we get the spans or not is dependent
on the underlying host system, which is a bit gross.

## Other details

~~AI assisted in various places, including generating the structs.~~
None of that worked in the end. Great job AI.

I made the failure an error as it _shouldn't_ fail in normal execution,
and we'd like to report if it does, however there are some cases where
the P/Invoke calls aren't re-written so we know this won't work. In case
of failure, we explicitly check for that, and log the error as debug
instead.

To get everything to compile easily I split the `ContainerMetadata`
implementation for .NET FX as it's not supported in that scenario
anyway.

Also, to be able to unit test the P/Invoke work, I copied the native
libs into the test directories.

Finally, it's worth us looking into [this
method](https://learn.microsoft.com/en-us/dotnet/api/system.runtime.interopservices.nativelibrary.setdllimportresolver?view=net-9.0)
to allow P/Invokes to happen in the native runner, given we know where
the native libs are, they're just not re-written. Something to think
about in a separate PR I think

Author: andrewlock

tracer/build/_build/Build.Steps.cs

@@ -680,18 +680,27 @@ async Task DownloadWafVersion(string libddwafVersion = null, string uncompressFo
 
                 var testBinFolder = testDir / "bin" / BuildConfiguration;
 
-                var (ext, source) = Platform switch
+                var (ext, source, libdatadog) = Platform switch
                 {
-                    PlatformFamily.Windows => ("dll", MonitoringHomeDirectory / $"win-{TargetPlatform}" / "datadog_profiling_ffi.dll"),
-                    PlatformFamily.Linux => ("so", MonitoringHomeDirectory / GetUnixArchitectureAndExtension().Arch / "libdatadog_profiling.so"),
-                    PlatformFamily.OSX => ("dylib", MonitoringHomeDirectory / "osx" / $"libdatadog_profiling.dylib"),
+                    PlatformFamily.Windows => ("dll", MonitoringHomeDirectory / $"win-{TargetPlatform}", "datadog_profiling_ffi.dll"),
+                    PlatformFamily.Linux => ("so", MonitoringHomeDirectory / GetUnixArchitectureAndExtension().Arch, "libdatadog_profiling.so"),
+                    PlatformFamily.OSX => ("dylib", MonitoringHomeDirectory / "osx", "libdatadog_profiling.dylib"),
                     _ => throw new NotSupportedException($"Unsupported platform: {Platform}")
                 };
 
+                var libs = new[]
+                {
+                    (libdatadog, $"LibDatadog.{ext}"),
+                    ($"Datadog.Tracer.Native.{ext}", $"Datadog.Tracer.Native.{ext}"),
+                };
+
                 foreach (var framework in frameworks)
                 {
-                    var dest = testBinFolder / framework / $"LibDatadog.{ext}";
-                    CopyFile(source, dest, FileExistsPolicy.Overwrite);
+                    foreach (var lib in libs)
+                    {
+                        var dest = testBinFolder / framework / lib.Item2;
+                        CopyFile(source / lib.Item1, dest, FileExistsPolicy.Overwrite);
+                    }
                 }
             }
         });

tracer/src/Datadog.Trace/ClrProfiler/NativeMethods.cs

@@ -243,6 +243,23 @@ public static int GetUserStrings(int arrSize, [In, Out] UserStringInterop[] arr)
             }
         }
 
+        public static bool TryGetInodeForPath(string path, out long result)
+        {
+            if (IsWindows)
+            {
+                result = -1;
+                return false;
+            }
+
+            result = NonWindows.GetInodeForPath(path);
+            if (result < 0)
+            {
+                return false;
+            }
+
+            return true;
+        }
+
         // the "dll" extension is required on .NET Framework
         // and optional on .NET Core
         // The DllImport methods are re-written by cor_profiler to have the correct vales
@@ -330,6 +347,9 @@ private static class NonWindows
 
             [DllImport("Datadog.Tracer.Native", CharSet = CharSet.Unicode)]
             public static extern int GetUserStrings(int arrSize, [In, Out] UserStringInterop[] arr);
+
+            [DllImport("Datadog.Tracer.Native")]
+            public static extern long GetInodeForPath([MarshalAs(UnmanagedType.LPWStr)]string path);
         }
     }
 }

tracer/src/Datadog.Trace/PlatformHelpers/ContainerMetadata.NetFramework.cs

@@ -0,0 +1,37 @@
+// <copyright file="ContainerMetadata.NetFramework.cs" company="Datadog">
+// Unless explicitly stated otherwise all files in this repository are licensed under the Apache 2 License.
+// This product includes software developed at Datadog (https://www.datadoghq.com/). Copyright 2017 Datadog, Inc.
+// </copyright>
+
+#nullable enable
+
+#if NETFRAMEWORK
+
+namespace Datadog.Trace.PlatformHelpers;
+
+/// <summary>
+/// Utility class with methods to interact with container hosts.
+/// </summary>
+internal static class ContainerMetadata
+{
+    /// <summary>
+    /// Gets the id of the container executing the code.
+    /// Return <c>null</c> if code is not executing inside a supported container.
+    /// </summary>
+    /// <returns>The container id or <c>null</c>.</returns>
+    public static string? GetContainerId() => null;
+
+    /// <summary>
+    /// Gets the unique identifier of the container executing the code.
+    /// Return values may be:
+    /// <list type="bullet">
+    /// <item>"ci-&lt;containerID&gt;" if the container id is available.</item>
+    /// <item>"in-&lt;inode&gt;" if the cgroup node controller's inode is available.
+    ///        We use the memory controller on cgroupv1 and the root cgroup on cgroupv2.</item>
+    /// <item><c>null</c> if neither are available.</item>
+    /// </list>
+    /// </summary>
+    /// <returns>The entity id or <c>null</c>.</returns>
+    public static string? GetEntityId() => null;
+}
+#endif

tracer/src/Datadog.Trace/PlatformHelpers/ContainerMetadata.cs

@@ -3,12 +3,17 @@
 // This product includes software developed at Datadog (https://www.datadoghq.com/). Copyright 2017 Datadog, Inc.
 // </copyright>
 
+#if !NETFRAMEWORK
+
 using System;
 using System.Collections.Generic;
+using System.Diagnostics.CodeAnalysis;
 using System.IO;
 using System.Linq;
+using System.Runtime.InteropServices;
 using System.Text.RegularExpressions;
 using System.Threading;
+using Datadog.Trace.ClrProfiler;
 using Datadog.Trace.Logging;
 using Datadog.Trace.Util;
 
@@ -117,8 +122,8 @@ public static string ExtractInodeFromCgroupLines(string controlGroupsMountPath,
             {
                 var tuple = ParseControllerAndPathFromCgroupLine(line);
                 if (tuple is not null
-                    && !string.IsNullOrEmpty(tuple.Item2)
-                    && (tuple.Item1 == string.Empty || string.Equals(tuple.Item1, "memory", StringComparison.OrdinalIgnoreCase)))
+                 && !string.IsNullOrEmpty(tuple.Item2)
+                 && (tuple.Item1 == string.Empty || string.Equals(tuple.Item1, "memory", StringComparison.OrdinalIgnoreCase)))
                 {
                     string controller = tuple.Item1;
                     string cgroupNodePath = tuple.Item2;
@@ -149,6 +154,49 @@ public static Tuple<string, string> ParseControllerAndPathFromCgroupLine(string
         }
 
         internal static bool TryGetInode(string path, out long result)
+            => TryGetInodeUsingPInvoke(path, out result)
+            || TryGetInodeUsingStat(path, out result);
+
+        // Internal for testing
+        internal static bool TryGetInodeUsingPInvoke(string path, out long result)
+        {
+            result = 0;
+
+            try
+            {
+                if (!NativeMethods.TryGetInodeForPath(path, out result))
+                {
+                    LogError(null, "Error obtaining inode using PInvoke, returned " + result);
+                    return false;
+                }
+
+                return true;
+            }
+            catch (Exception ex)
+            {
+                LogError(ex, "Error obtaining inode using PInvoke");
+                return false;
+            }
+
+            static void LogError(Exception ex, string message)
+            {
+#pragma warning disable DDLOG004 // Must use constant strings - disabled as it's an integer only, and only called twice in the app lifetime
+                if (EnvironmentHelpersNoLogging.IsClrProfilerAttachedSafe())
+                {
+                    // if it's attached, then this really is an error
+                    Log.Error(ex, message);
+                }
+                else
+                {
+                    // The profiler isn't there, so this is "normal"
+                    Log.Debug(ex, message);
+                }
+            }
+#pragma warning restore DDLOG004
+        }
+
+        // Internal for testing
+        internal static bool TryGetInodeUsingStat(string path, out long result)
         {
             result = 0;
 
@@ -159,7 +207,7 @@ internal static bool TryGetInode(string path, out long result)
             }
             catch (Exception ex)
             {
-                Log.Warning(ex, "Error obtaining inode.");
+                Log.Warning(ex, "Error obtaining inode using stat");
                 return false;
             }
         }
@@ -223,3 +271,4 @@ private static bool IsHostCgroupNamespaceInternal()
         }
     }
 }
+#endif

tracer/src/Datadog.Tracer.Native/Datadog.Tracer.Native.def

@@ -24,5 +24,6 @@ EXPORTS
     ShouldHeal
     ReportSuccessfulInstrumentation
     GetUserStrings
+    GetInodeForPath
     InitEmbeddedCallSiteDefinitions
     InitEmbeddedCallTargetDefinitions

tracer/src/Datadog.Tracer.Native/interop.cpp

@@ -268,7 +268,37 @@ EXTERN_C BOOL STDAPICALLTYPE ShouldHeal(ModuleID moduleId, int methodToken, WCHA
     return trace::profiler->ShouldHeal(moduleId, methodToken, instrumentationId, products);
 }
 
+EXTERN_C u_long STDAPICALLTYPE GetInodeForPath(const WCHAR* path)
+{
+#ifdef _WIN32
+    // We don't need to support this on Windows
+    return -1;
+#else
+    if (!path)
+    {
+        return -1;
+    }
+
+    const std::string str = ToString(path);
+    if (str.empty())
+    {
+        return -1;
+    }
+
+    // We use lstat, because that matches the behaviour of the 'stat' command
+    // i.e. don't follow symlinks
+    struct stat st {};
+    if (lstat(str.c_str(), &st) != 0)
+    {
+        return -errno;
+    }
+
+    return st.st_ino;
+#endif
+}
+
 #ifndef _WIN32
+
 EXTERN_C void *dddlopen (const char *__file, int __mode)
 {
     return dlopen(__file, __mode);

tracer/test/Datadog.Trace.ClrProfiler.IntegrationTests/VersionConflict/VersionConflict2xTests.cs

@@ -18,6 +18,11 @@ public class VersionConflict2xTests : TestHelper
         public VersionConflict2xTests(ITestOutputHelper output)
             : base("VersionConflict.2x", output)
         {
+            // We disable the process integration to prevent the `stat` process spans that
+            // _may_ be present depending on the underlying host system (cgroup v1/v2).
+            // The "do not trace" helper that normally blocks tracing these spans doesn't
+            // work in version conflict scenarios.
+            SetEnvironmentVariable("DD_TRACE_Process_ENABLED", "0");
         }
 
         [SkippableFact]

tracer/test/Datadog.Trace.Tests/PlatformHelpers/ContainerMetadataTests.cs

@@ -3,6 +3,8 @@
 // This product includes software developed at Datadog (https://www.datadoghq.com/). Copyright 2017 Datadog, Inc.
 // </copyright>
 
+#if !NETFRAMEWORK
+
 using System;
 using System.Collections.Generic;
 using System.Diagnostics;
@@ -255,5 +257,23 @@ public void Parse_TryGetInode()
             success.Should().BeTrue();
             inode.Should().BeGreaterThan(0);
         }
+
+        [SkippableFact]
+        public void Parse_TryGetInode_ShouldGetSameValueFromPinvokeAndProcess()
+        {
+            SkipOn.Platform(SkipOn.PlatformValue.Windows);
+            SkipOn.Platform(SkipOn.PlatformValue.MacOs);
+
+            string currentDirectory = Environment.CurrentDirectory;
+            bool success1 = ContainerMetadata.TryGetInodeUsingStat(currentDirectory, out long inode1);
+            bool success2 = ContainerMetadata.TryGetInodeUsingPInvoke(currentDirectory, out long inode2);
+
+            success1.Should().BeTrue();
+            success2.Should().BeTrue();
+            inode1.Should().BeGreaterThan(0);
+            inode2.Should().Be(inode1);
+        }
     }
 }
+
+#endif