Skip to content

[DTLTO][LLD][COFF] Add support for Integrated Distributed ThinLTO #148594

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Jul 20, 2025
Merged

[DTLTO][LLD][COFF] Add support for Integrated Distributed ThinLTO #148594

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
81e1c4f
[DTLTO][LLD][COFF] Add support for Integrated Distributed ThinLTO
bd1976bris Jul 14, 2025
f88476a
Merge branch 'main' into dtlto_lld_coff
tru Jul 19, 2025
b25fa9a
Use more efficient indenting for new the options in Options.td
bd1976bris Jul 19, 2025
f131ad7
Use -<option name> spelling for -thinlto- options in new tests.
bd1976bris Jul 19, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
93 changes: 93 additions & 0 deletions cross-project-tests/dtlto/link-archive-thin.test
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
REQUIRES: lld-link

## Test that a DTLTO link succeeds and outputs the expected set of files
## correctly when thin archives are present.

RUN: rm -rf %t && split-file %s %t && cd %t

## Compile bitcode. -O2 is required for cross-module importing.
RUN: %clang -O2 --target=x86_64-pc-windows-msvc -flto=thin -c \
RUN: foo.c bar.c dog.c cat.c start.c

## Generate thin archives.
RUN: lld-link /lib /llvmlibthin /out:foo.lib foo.o
## Create this bitcode thin archive in a subdirectory to test the expansion of
## the path to a bitcode file that is referenced using "..", e.g., in this case
## "../bar.o".
RUN: mkdir lib
RUN: lld-link /lib /llvmlibthin /out:lib/bar.lib bar.o
## Create this bitcode thin archive with an absolute path entry containing "..".
RUN: lld-link /lib /llvmlibthin /out:dog.lib %t/lib/../dog.o
RUN: lld-link /lib /llvmlibthin /out:cat.lib cat.o
RUN: lld-link /lib /llvmlibthin /out:start.lib start.o

## Link from a different directory to ensure that thin archive member paths are
## resolved correctly relative to the archive locations.
RUN: mkdir %t/out && cd %t/out
RUN: lld-link /subsystem:console /machine:x64 /entry:start /out:my.exe \
RUN: %t/foo.lib %t/lib/bar.lib ../start.lib %t/cat.lib \
RUN: /includeoptional:dog ../dog.lib \
RUN: -thinlto-distributor:%python \
RUN: -thinlto-distributor-arg:%llvm_src_root/utils/dtlto/local.py \
RUN: -thinlto-remote-compiler:%clang \
RUN: /lldsavetemps

## Check that the required output files have been created.
RUN: ls | FileCheck %s --check-prefix=OUTPUTS --implicit-check-not=cat

## JSON jobs description.
OUTPUTS-DAG: my.[[PID:[a-zA-Z0-9_]+]].dist-file.json

## Individual summary index files.
OUTPUTS-DAG: start.1.[[PID]].native.o.thinlto.bc{{$}}
OUTPUTS-DAG: dog.2.[[PID]].native.o.thinlto.bc{{$}}
OUTPUTS-DAG: foo.3.[[PID]].native.o.thinlto.bc{{$}}
OUTPUTS-DAG: bar.4.[[PID]].native.o.thinlto.bc{{$}}

## Native output object files.
OUTPUTS-DAG: start.1.[[PID]].native.o{{$}}
OUTPUTS-DAG: dog.2.[[PID]].native.o{{$}}
OUTPUTS-DAG: foo.3.[[PID]].native.o{{$}}
OUTPUTS-DAG: bar.4.[[PID]].native.o{{$}}


## It is important that cross-module inlining occurs for this test to show that Clang can
## successfully load the bitcode file dependencies recorded in the summary indices.
## Explicitly check that the expected importing has occurred.

RUN: llvm-dis start.1.*.native.o.thinlto.bc -o - | \
RUN: FileCheck %s --check-prefixes=FOO,BAR,START

RUN: llvm-dis dog.2.*.native.o.thinlto.bc -o - | \
RUN: FileCheck %s --check-prefixes=FOO,BAR,DOG,START

RUN: llvm-dis foo.3.*.native.o.thinlto.bc -o - | \
RUN: FileCheck %s --check-prefixes=FOO,BAR,START

RUN: llvm-dis bar.4.*.native.o.thinlto.bc -o - | \
RUN: FileCheck %s --check-prefixes=FOO,BAR,START

FOO-DAG: foo.o
BAR-DAG: bar.o
DOG-DAG: dog.o
START-DAG: start.o


#--- foo.c
extern int bar(int), start(int);
__attribute__((retain)) int foo(int x) { return x + bar(x) + start(x); }

#--- bar.c
extern int foo(int), start(int);
__attribute__((retain)) int bar(int x) { return x + foo(x) + start(x); }

#--- dog.c
extern int foo(int), bar(int), start(int);
__attribute__((retain)) int dog(int x) { return x + foo(x) + bar(x) + start(x); }

#--- cat.c
__attribute__((retain)) void cat(int x) {}

#--- start.c
extern int foo(int), bar(int);
__attribute__((retain)) int start(int x) { return x + foo(x) + bar(x); }
41 changes: 41 additions & 0 deletions cross-project-tests/dtlto/link-dtlto.c
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
// REQUIRES: lld-link

/// Simple test that DTLTO works with a single input bitcode file and that
/// --save-temps can be applied to the remote compilation.

// RUN: rm -rf %t && mkdir %t && cd %t

// RUN: %clang --target=x86_64-pc-windows-msvc -c -flto=thin %s -o dtlto.obj

// RUN: lld-link /subsystem:console /entry:_start dtlto.obj \
// RUN: -thinlto-distributor:%python \
// RUN: -thinlto-distributor-arg:%llvm_src_root/utils/dtlto/local.py \
// RUN: -thinlto-remote-compiler:%clang \
// RUN: -thinlto-remote-compiler-arg:--save-temps

/// Check that the required output files have been created.
// RUN: ls | sort | FileCheck %s

/// No files are expected before.
// CHECK-NOT: {{.}}

/// Linked ELF.
// CHECK: {{^}}dtlto.exe{{$}}

/// Produced by the bitcode compilation.
// CHECK-NEXT: {{^}}dtlto.obj{{$}}

/// --save-temps output for the backend compilation.
// CHECK-NEXT: {{^}}dtlto.s{{$}}
// CHECK-NEXT: {{^}}dtlto.s.0.preopt.bc{{$}}
// CHECK-NEXT: {{^}}dtlto.s.1.promote.bc{{$}}
// CHECK-NEXT: {{^}}dtlto.s.2.internalize.bc{{$}}
// CHECK-NEXT: {{^}}dtlto.s.3.import.bc{{$}}
// CHECK-NEXT: {{^}}dtlto.s.4.opt.bc{{$}}
// CHECK-NEXT: {{^}}dtlto.s.5.precodegen.bc{{$}}
// CHECK-NEXT: {{^}}dtlto.s.resolution.txt{{$}}

/// No files are expected after.
// CHECK-NOT: {{.}}

int _start() { return 0; }
2 changes: 1 addition & 1 deletion cross-project-tests/lit.cfg.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
config.test_format = lit.formats.ShTest(not llvm_config.use_lit_shell)

# suffixes: A list of file extensions to treat as test files.
config.suffixes = [".c", ".cl", ".cpp", ".m"]
config.suffixes = [".c", ".cl", ".cpp", ".m", ".test"]

# excludes: A list of directories to exclude from the testsuite. The 'Inputs'
# subdirectories contain auxiliary inputs for various tests in their parent
Expand Down
12 changes: 12 additions & 0 deletions lld/COFF/Config.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -192,6 +192,18 @@ struct Configuration {
// Used for /lldltocachepolicy=policy
llvm::CachePruningPolicy ltoCachePolicy;

// Used for /thinlto-distributor:<path>
StringRef dtltoDistributor;

// Used for /thinlto-distributor-arg:<arg>
llvm::SmallVector<llvm::StringRef, 0> dtltoDistributorArgs;

// Used for /thinlto-remote-compiler:<path>
StringRef dtltoCompiler;

// Used for /thinlto-remote-compiler-arg:<arg>
llvm::SmallVector<llvm::StringRef, 0> dtltoCompilerArgs;

// Used for /opt:[no]ltodebugpassmanager
bool ltoDebugPassManager = false;

Expand Down
17 changes: 17 additions & 0 deletions lld/COFF/Driver.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2088,6 +2088,23 @@ void LinkerDriver::linkerMain(ArrayRef<const char *> argsArr) {
Fatal(ctx) << "/manifestinput: requires /manifest:embed";
}

// Handle /thinlto-distributor:<path>
config->dtltoDistributor = args.getLastArgValue(OPT_thinlto_distributor);

// Handle /thinlto-distributor-arg:<arg>
for (auto *arg : args.filtered(OPT_thinlto_distributor_arg))
config->dtltoDistributorArgs.push_back(arg->getValue());

// Handle /thinlto-remote-compiler:<path>
config->dtltoCompiler = args.getLastArgValue(OPT_thinlto_compiler);
if (!config->dtltoDistributor.empty() && config->dtltoCompiler.empty())
Err(ctx) << "A value must be specified for /thinlto-remote-compiler if "
"/thinlto-distributor is specified.";

// Handle /thinlto-remote-compiler-arg:<arg>
for (auto *arg : args.filtered(OPT_thinlto_compiler_arg))
config->dtltoCompilerArgs.push_back(arg->getValue());

// Handle /dwodir
config->dwoDir = args.getLastArgValue(OPT_dwodir);

Expand Down
11 changes: 10 additions & 1 deletion lld/COFF/LTO.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -110,7 +110,16 @@ BitcodeCompiler::BitcodeCompiler(COFFLinkerContext &c) : ctx(c) {

// Initialize ltoObj.
lto::ThinBackend backend;
if (ctx.config.thinLTOIndexOnly) {
if (!ctx.config.dtltoDistributor.empty()) {
backend = lto::createOutOfProcessThinBackend(
llvm::hardware_concurrency(ctx.config.thinLTOJobs),
/*OnWrite=*/nullptr,
/*ShouldEmitIndexFiles=*/false,
/*ShouldEmitImportFiles=*/false, ctx.config.outputFile,
ctx.config.dtltoDistributor, ctx.config.dtltoDistributorArgs,
ctx.config.dtltoCompiler, ctx.config.dtltoCompilerArgs,
!ctx.config.saveTempsArgs.empty());
} else if (ctx.config.thinLTOIndexOnly) {
auto OnIndexWrite = [&](StringRef S) { thinIndices.erase(S); };
backend = lto::createWriteIndexesThinBackend(
llvm::hardware_concurrency(ctx.config.thinLTOJobs),
Expand Down
11 changes: 11 additions & 0 deletions lld/COFF/Options.td
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -270,6 +270,17 @@ def thinlto_object_suffix_replace : P<
def thinlto_prefix_replace: P<
"thinlto-prefix-replace",
"'old;new' replace old prefix with new prefix in ThinLTO outputs">;
def thinlto_distributor : P<"thinlto-distributor",
"Distributor to use for ThinLTO backend compilations. If specified, ThinLTO "
"backend compilations will be distributed">;
def thinlto_distributor_arg : P<"thinlto-distributor-arg",
"Arguments to pass to the ThinLTO distributor">;
def thinlto_compiler : P<"thinlto-remote-compiler",
"Compiler for the ThinLTO distributor to invoke for ThinLTO backend "
"compilations">;
def thinlto_compiler_arg : P<"thinlto-remote-compiler-arg",
"Compiler arguments for the ThinLTO distributor to pass for ThinLTO backend "
"compilations">;
def lto_obj_path : P<
"lto-obj-path",
"output native object for merged LTO unit to this path">;
Expand Down
37 changes: 35 additions & 2 deletions lld/docs/DTLTO.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,7 @@ during the traditional link step.

The implementation is documented here: https://llvm.org/docs/DTLTO.html.

Currently, DTLTO is only supported in ELF LLD. Support will be added to other
LLD flavours in the future.
Currently, DTLTO is only supported in ELF and COFF LLD.

ELF LLD
-------
Expand Down Expand Up @@ -40,3 +39,37 @@ The command-line interface is as follows:
Some LLD LTO options (e.g., ``--lto-sample-profile=<file>``) are supported.
Currently, other options are silently accepted but do not have the intended
effect. Support for such options will be expanded in the future.

COFF LLD
--------

The command-line interface is as follows:

- ``/thinlto-distributor:<path>``
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems that most ThinLTO tests use the -thinlto* spelling.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this comment. I have changed the tests to use the -thinlto* spelling for consistency. However, I strongly feel that for user facing content like the documentation and the new error message we should use the /thinlto* spelling for consistency with the help text. The currently help text renders like:

  /thinlto-distributor-arg:<value>
                          Arguments to pass to the ThinLTO distributor
  /thinlto-distributor:<value>
                          Distributor to use for ThinLTO backend compilations. If specified, ThinLTO backend compilations will be distributed
  /thinlto-emit-imports-files
                          Emit .imports files with -thinlto-index-only
  /thinlto-index-only:<value>
                          -thinlto-index-only and also write native module names to file
  /thinlto-index-only     Instead of linking, emit ThinLTO index files
  /thinlto-object-suffix-replace:<value>
                          'old;new' replace old suffix with new suffix in ThinLTO index
  /thinlto-prefix-replace:<value>
                          'old;new' replace old prefix with new prefix in ThinLTO outputs
  /thinlto-remote-compiler-arg:<value>
                          Compiler arguments for the ThinLTO distributor to pass for ThinLTO backend compilations
  /thinlto-remote-compiler:<value>
                          Compiler for the ThinLTO distributor to invoke for ThinLTO backend compilations

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK:)

Specifies the file to execute as the distributor process. If specified,
ThinLTO backend compilations will be distributed.

- ``/thinlto-remote-compiler:<path>``
Specifies the path to the compiler that the distributor process will use for
backend compilations. The compiler invoked must match the version of LLD.

- ``/thinlto-distributor-arg:<arg>``
Specifies ``<arg>`` on the command line when invoking the distributor.
Can be specified multiple times.

- ``/thinlto-remote-compiler-arg:<arg>``
Appends ``<arg>`` to the remote compiler's command line.
Can be specified multiple times.

Options that introduce extra input/output files may cause miscompilation if
the distribution system does not automatically handle pushing/fetching them to
remote nodes. In such cases, configure the distributor - possibly using
``/thinlto-distributor-arg:`` - to manage these dependencies. See the
distributor documentation for details.

Some LLD LTO options (e.g., ``/lto-sample-profile:<file>``) are supported.
Currently, other options are silently accepted but do not have the intended
effect. Support for such options could be expanded in the future.

Currently, there is no DTLTO command line interface supplied for ``clang-cl``,
as users are expected to invoke LLD directly.
71 changes: 71 additions & 0 deletions lld/test/COFF/dtlto/files.test
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
REQUIRES: x86

## Test that the LLD options /lldsavetemps and -thinlto-emit-imports-files
## function correctly with DTLTO we also check that index files
## (-thinlto-emit-index-files) are not emitted with DTLTO.

RUN: rm -rf %t && split-file %s %t && cd %t

RUN: sed 's/@t1/@t2/g' t1.ll > t2.ll

## Generate ThinLTO bitcode files. Note that t3.bc will not be used by the
## linker.
RUN: opt -thinlto-bc t1.ll -o t1.bc
RUN: opt -thinlto-bc t2.ll -o t2.bc
RUN: cp t1.bc t3.bc

## Generate object files for mock.py to return.
RUN: llc t1.ll --filetype=obj -o t1.obj
RUN: llc t2.ll --filetype=obj -o t2.obj

## Create response file containing shared ThinLTO linker arguments.
## -start-lib/-end-lib is used to test the special case where unused lazy
## bitcode inputs result in empty index/imports files.
## Note that mock.py does not do any compilation; instead, it simply writes
## the contents of the object files supplied on the command line into the
## output object files in job order.
RUN: echo "/entry:t1 /subsystem:console \
RUN: t1.bc t2.bc -start-lib t3.bc -end-lib /out:my.exe \
RUN: -thinlto-distributor:\"%python\" \
RUN: -thinlto-distributor-arg:\"%llvm_src_root/utils/dtlto/mock.py\" \
RUN: -thinlto-distributor-arg:t1.obj \
RUN: -thinlto-distributor-arg:t2.obj \
RUN: -thinlto-remote-compiler:fake.exe" > l.rsp

## Check that without extra flags, no index/imports files are produced and
## backend temp files are removed.
RUN: lld-link @l.rsp
RUN: ls | FileCheck %s \
RUN: --check-prefixes=NOBACKEND,NOOTHERS

## Check that with /lldsavetemps and -thinlto-emit-imports-files backend
## tempoary files are retained and no index/imports files are produced.
RUN: rm -f *.imports *.thinlto.bc
RUN: lld-link @l.rsp /lldsavetemps -thinlto-emit-imports-files
RUN: ls | sort | FileCheck %s \
RUN: --check-prefixes=BACKEND,NOOTHERS

## JSON jobs description, retained with --save-temps.
## Note that DTLTO temporary files include a PID component.
NOBACKEND-NOT: {{^}}my.[[#]].dist-file.json{{$}}
BACKEND: {{^}}my.[[#]].dist-file.json{{$}}

## Index/imports files for t1.bc.
NOOTHERS-NOT: {{^}}t1.bc.imports{{$}}
NOOTHERS-NOT: {{^}}t1.bc.thinlto.bc{{$}}

## Index/imports files for t2.bc.
NOOTHERS-NOT: {{^}}t2.bc.imports{{$}}
NOOTHERS-NOT: {{^}}t2.bc.thinlto.bc{{$}}

## Empty index/imports files for unused t3.bc.
NOOTHERS-NOT: {{^}}t3.bc.imports{{$}}
NOOTHERS-NOT: {{^}}t3.bc.thinlto.bc{{$}}

#--- t1.ll
target datalayout = "e-m:w-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128"
target triple = "x86_64-pc-windows-msvc"

define void @t1() {
ret void
}
Loading