make the torchprime E2E test work for forked repos

[zhanyong-wan]

🐛 Bug

If a PR is created via a forked torch_xla repo (e.g. if the contributor has no write permission to the original torch_xla repo), the torchprime E2E test will fail in CI. See #9338 for an example.

The error:

Error when evaluating 'secrets'. .github/workflows/build_and_test.yml (Line: 81, Col: 11): Secret TORCH_XLA_BOT_TOKEN is required, but not provided while calling., .github/workflows/build_and_test.yml (Line: 81, Col: 11): Secret GCLOUD_SERVICE_KEY is required, but not provided while calling.

The root cause is that PRs made from a fork won't have access to any secrets of the original repo.

[zhanyong-wan]

For now, I'll make the test optional. We should make it required after this bug is fixed.

[pganssle-google]

It seems that this is a deliberate choice on GH's part for security reasons and there's no way to enable it for public repos. I found two possible workarounds:

1. Use a workflow with pull_request_target: instead of pull_request in on, which pulls the head of the PR.
2. Use a workflow with workflow_run that is triggered based on the passing of another workflow. workflow_run deliberately does get access to secrets even if triggered from a workflow that doesn't have permissions to access the secrets, but it also checks out from the main branch, not from the PR branch.

Here is an example PR that shows both approaches (they are removing approach 1 in favor of approach 2). I believe the main problem here is that if you allow anyone to make a PR that has access to secrets, they can exfiltrate the secrets because you are executing their code.

We may be able to chain together workflow_run and pull_request_approved so that the E2E tests are run on forked repos only after approval (with the assumption that no one will approve a PR that is exfiltrating secrets). I can dig around a little later to also see if we can trigger the workflow automatically if the fork is from some form of trusted collaborator (however we want to define that).

Another thought is that if we can make sure that we're taking the "run the test on GCP" orchestration logic from the main branch and all the forked code is run only on the GCP runner (without access to the token), we probably don't need to wait for approval. I can look into this more later.