[CI] Add GHA job to test downstream GB-25

[giordano]

No description provided.

[giordano]

ERROR: /root/.bazel/external/xla/xla/BUILD:1237:11: Compiling xla/cpu_function_runtime.cc failed: absolute path inclusion(s) found in rule '@@xla//xla:cpu_function_runtime':
the source file 'xla/cpu_function_runtime.cc' includes the following non-builtin files with absolute paths (if these are builtin files, make sure these paths are in your toolchain):
  '/root/.bazel/external/eigen_archive/Eigen/src/Core/InternalHeaderCheck.h'
  '/root/.bazel/external/eigen_archive/Eigen/src/Core/InternalHeaderCheck.h'

Wat

[wsmoses]

er, wat

[GleasonK]

I wonder if it has to do with using include <Eigen/Core> vs include "Eigen/Core", looks like we have both patterns floating around: https://github.com/search?q=repo%3Aopenxla%2Fxla%20Eigen%2FCore&type=code

Workaround in a few other similar bugs was adding those paths to system include dirs.. but that's not ideal

[wsmoses]

if we use clang we get:

1 warning generated.
INFO: From Compiling upb/mini_table/extension_registry.c:
clang: warning: argument unused during compilation: '--cuda-path=external/cuda_nvcc' [-Wunused-command-line-argument]
ERROR: /root/.bazel/external/boringssl/BUILD:133:11: Compiling err_data.c failed: (Exit 1): process-wrapper failed: error executing CppCompile command 
  (cd /root/.bazel/sandbox/processwrapper-sandbox/358/execroot/__main__ && \
  exec env - \
    GRPC_BAZEL_RUNTIME=1 \
    JULIA=/__w/_tool/julia/1.11.6/x64/bin/julia \
    PATH=/__w/.cache/bazel/bazelisk/downloads/sha256/c97f02133adce63f0c28678ac1f21d65fa8255c80429b588aeeba8a1fac6202b/bin:/__w/_tool/julia/1.11.6/x64/bin:/__w/_tool/bazelisk/1.x/x64:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
    PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=upb \
    *** \
    TMPDIR=/tmp \
  /__w/.cache/bazel/bazel/_bazel_root/install/81618c1cfcf8a55fe29d247a9003bce4/process-wrapper '--timeout=0' '--kill_delay=15' '--stats=/root/.bazel/sandbox/processwrapper-sandbox/358/stats.out' external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc -MD -MF bazel-out/k8-opt/bin/external/boringssl/_objs/crypto/err_data.pic.d '-frandom-seed=bazel-out/k8-opt/bin/external/boringssl/_objs/crypto/err_data.pic.o' -iquote external/boringssl -iquote bazel-out/k8-opt/bin/external/boringssl -isystem external/boringssl/src/include -isystem bazel-out/k8-opt/bin/external/boringssl/src/include -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -fPIC -U_FORTIFY_SOURCE '-D_FORTIFY_SOURCE=1' -fstack-protector -Wall -fno-omit-frame-pointer -no-canonical-prefixes -DNDEBUG -g0 -O2 -ffunction-sections -fdata-sections '--cuda-path=external/cuda_nvcc' -DGRPC_BAZEL_BUILD -DBORINGSSL_IMPLEMENTATION -Wa,--noexecstack -Wall -Werror '-Wformat=2' -Wsign-compare -Wmissing-field-initializers -Wwrite-strings -Wshadow -fno-common '-D_XOPEN_SOURCE=700' '-std=c11' -Wmissing-prototypes -Wold-style-definition -Wstrict-prototypes -c external/boringssl/err_data.c -o bazel-out/k8-opt/bin/external/boringssl/_objs/crypto/err_data.pic.o)
clang: error: argument unused during compilation: '--cuda-path=external/cuda_nvcc' [-Werror,-Wunused-command-line-argument]
[3,353 / 16,619] [Prepa] action 'SolibSymlink _solib_local/_U_A_Acuda_Ucublas_S_S_CcublasLt_Ushared_Ulibrary___Ulib/libcublasLt.so.12'
[3,354 / 16,619] checking cached actions
Target //:libReactantExtra.so failed to build
INFO: Elapsed time: 274.757s, Critical Path: 4.72s
INFO: 3354 processes: 8 disk cache hit, 3035 internal, 311 processwrapper-sandbox.

[giordano]

Why -Werror, why.

[giordano]

I think this is basically ready now, but the question is what to do with it? It takes 45 minutes only to recompile xla/libreactant every time (bazel cache doesn't seem to be very effective here), pushing total runtime to over 70 minutes, not exactly a quick turnaround. And we have only two runners of these, the queue would get backed up very quickly when there are more than one pull request being worked on at the same time.

[wsmoses]

can we add a new matrix of XLA commits, default empty string, with an optional hash. This would be exceptionally useful for ablation tests (including the comms).

We should be able to fix the cache issue shortly so I'm also fine for now temporary using more resources

[wsmoses]

also if it speeds things up we can elect to do the non super verbose xla dump

[giordano]

also if it speeds things up we can elect to do the non super verbose xla dump

That I've already done, it was timing out with --xla_dump_hlo_pass_re=.* 😄

[wsmoses]

Ah fair.

In any case, let's set up the xla commit part, and ablate the comm pr

[giordano]

Is 6965da3 (#1197) what you were thinking for XLA? You would change 0123456789abcdef0123456789abcdef01234567 as you need, besides uncommenting that line.

[wsmoses]

Two things:

1. we also need to comment out or delete the load of the xla commit from Jax
2. we should match and hash

[wsmoses]

Ah jk I see you do 2

[wsmoses]

We should also add a series of targets, like
Xla_hash:

• ""
• "abcd..."

To the github actions yml, and the have it use the hash if non empty

[giordano]

Uhm, not sure how you mean exactly. Do you have an example?

[giordano]

Ok, done in 09919a3 (#1197)

[giordano]

Note that we also upload the profile traces (example), which show that NCCL communication is still a sizable fraction of the whole runtime (almost 18% in this trace)

[wsmoses]

lets see how the default compares against openxla/xla#29448 [both in terms of overall runtime and also # of all-gathers / all-reduces].

cc @felixwqp @frgossen

[wsmoses]

[in a follow up we should also ablate the impact of https://github.com/EnzymeAD/Reactant.jl/pull/1496]

[wsmoses]

okay all gathers arent in the optimized code which is good, seemingly just all-reduces to go