Add GitHub Actions workflow for building CPU Aarch64 Linux wheels

[Chizkiyahu]

Summary

Fixes #13044

Release notes: build

[Chizkiyahu]

@pytorchbot label "release notes: build"

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/13946

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 1 Cancelled Job, 37 Pending, 3 Unrelated Failures

As of commit da0b1c6 with merge base 1ea9feb ():

NEW FAILURE - The following job has failed:

• pull / unittest / linux / linux-job (gh)
  exir/tests/test_quant_fusion_pass.py::TestQuantFusionPass::test_embedding_torchao

CANCELLED JOB - The following job was cancelled. Please retry:

• pull / test-binary-size-linux-gcc / linux-job (gh)

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• pull / test-moshi-linux / linux-job (gh) (trunk failure)
  ##[error]The operation was canceled.
• pull / test-openvino-linux / linux-job (gh) (trunk failure)
  ##[error]The operation was canceled.
• pull / test-setup-linux-gcc / linux-job (gh) (trunk failure)
  ##[error]The operation was canceled.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[mergennachin]

Thanks @Chizkiyahu for your initiative to enable linux aarch64

Looks like one of the dependencies (https://github.com/meta-pytorch/tokenizers/) is not able to run on linux aarch64

Is it something you can take a look at? And add test there too?

https://github.com/meta-pytorch/tokenizers/tree/main/.github/workflows

[Chizkiyahu]

@mergennachin
Thanks for you time and help
I build tokenizers on local and it works
I open PR meta-pytorch/tokenizers#125
please approve the workflow run

[Chizkiyahu]

@mergennachin
thanks for the help

I found the problem
is missing some include bdbf309

please approve awaiting workflows

[Chizkiyahu]

@mergennachin
thanks for the help
is look like that the problem is that the arm runners don't have libatomic install
libatomic is needed for build tokenizers

there is multiple options - what is the right way to do it

1. to build in docker
2. to ask pytorch/test-infra/ to install it
3. add sudo dnf install -y libatomic to the pre-build script ?

or another way ?

[mergennachin]

Thanks @Chizkiyahu for following through and debugging the issue.

Asking pytorch/test-infra is probably the long term solution, but let's just do directly in the pre-build script
Something like this:

  # In .ci/scripts/wheel/pre_build_script.sh
  if [[ "$(uname -m)" == "aarch64" ]]; then
      sudo dnf install -y libatomic
  fi

[Chizkiyahu]

@mergennachin thanks
please approve the workflow

[mergennachin]

@Chizkiyahu I have been trying a bunch of things to make the tokenizer work without much luck. Also, sorry to pollute your PR, i was meaning to upstream to a separate branch to experiment, but it ended up merging to your branch...

Not sure what to do right now. I'm a bit occupied right now, so probably won't spend much time for the next couple of days, on this. But if you have ideas, let me know. I will keep approving the CI runs.

[Chizkiyahu]

@Chizkiyahu I have been trying a bunch of things to make the tokenizer work without much luck. Also, sorry to pollute your PR, i was meaning to upstream to a separate branch to experiment, but it ended up merging to your branch...

Not sure what to do right now. I'm a bit occupied right now, so probably won't spend much time for the next couple of days, on this. But if you have ideas, let me know. I will keep approving the CI runs.

thanks a lot
I am still working on that
big thanks for the CI approving

[Chizkiyahu]

Update

1. I found a build problem with Google’s sentencepiece on the CI ARM machine. I wasn’t able to reproduce it on my local machine, so I cannot open an issue to sentencepiece.
   The problem is resolved by this patch, which fixes the linking error.
   I updated .ci/scripts/wheel/pre_build_script.sh to apply this patch.

2. Apple’s coremltools does not support aarch64 Missing Linux AArch64 wheels on PyPi apple/coremltools#1254 and fails at runtime. I’ve updated the tests to exclude it accordingly.

[Chizkiyahu]

@mergennachin
please approve ci run

[Chizkiyahu]

@mergennachin
the new commit c3b842f that Add torchcodec as temporary dependency
brake the aarch64 build because torchcodec don't have aarch64 support see pytorch/torchcodec#569

when I try to build the torchcodec for aarch64 I have 2 problem

1. in build i needs ffmpg files in s3 for the file https://github.com/pytorch/torchcodec/blob/main/src/torchcodec/_core/fetch_and_expose_non_gpl_ffmpeg_libs.cmake
2. when i use I_CONFIRM_THIS_IS_NOT_A_LICENSE_VIOLATION some tests fails

[Chizkiyahu]

update I open PR for torchcodec
pytorch/torchcodec#895

[NicolasHug]

Hi folks, Nicolas from TorchCodec here

Sorry, I don't think we'll be able to prioritize landing pytorch/torchcodec#895 for now. My understanding is that you have a dependency on TorchAudio, which meant you had to take a dependency on TorchCodec because TorchAudio now relies on TorchCodec for the load() and save() utils.

To unblock you for the coming release, I would recommend the following paths:

• you shouldn't need torchcodec to build an executorch wheel. If your are running some tests when building the wheel, maybe you can deactivate those that rely on torchaudio.load or torchaudio.save?
• use conda-forge to install torchcodec. There is aarch64 support for torchcodec on conda-forge: https://anaconda.org/conda-forge/torchcodec

[mergennachin]

@NicolasHug @Chizkiyahu this should unblock #14211

[Chizkiyahu]

@mergennachin
thanks you
please approve CI