[ Enhancement ] Add Claude Code GHA to review PRs

[gwright99]

Background

This provides another layer of assurance in the PR review process for minimal cost and a minor increase in repository activity noise.

Tasks

• Review (Official) Claude Code GitHub Actions documentation
• Review (Official) Claude Code Security Reviews documentation
• Consult Seqera Engineering for any edgecases / gotchas to be aware of given the setup they are using.
  • Consult platform repo .claude folder.
  • Tighten PR security model

Consideration

• The repo is configured to allow non-Seqerans to interact with it via Issues and PRs (to encourage community-driven improvements).

• The introduction of Claude Code invocations means a charge will now be incurred each time Claude is invoked. As a result, we'll need to be more restrictive re: who is allowed to invoke Claude while still keeping the repo open.

  Decision: Make the CC GHA a sequence of two jobs:

      1. Check if user is member of seqeralabs;
      2. If yes, run rest of workflow.
  

[gwright99]

Proposed Implementation

After some adversarial prompting on the original example, I arrived at the following workflow.

name: Claude Bot

on:
  issue_comment:
    types: [created]
  pull_request_review_comment:
    types: [created]
  issues:
    types: [opened, assigned]
  pull_request_review:
    types: [submitted]

jobs:
  check-membership:
    if: |
      (github.actor != 'claude[bot]') &&
        ((github.event_name == 'issue_comment' && contains(github.event.comment.body, '@claude')) ||
        (github.event_name == 'pull_request_review_comment' && contains(github.event.comment.body, '@claude')) ||
        (github.event_name == 'pull_request_review' && contains(github.event.review.body, '@claude')) ||
        (github.event_name == 'issues' && contains(github.event.issue.body, '@claude'))
      )
    runs-on: ubuntu-latest
    outputs:
      is-member: ${{ steps.check.outputs.allowed }}
    steps:
      - name: Check if actor is an org member
        id: check
        uses: actions/github-script@v7
        with:
          script: |
            const org = 'MY_ORG_NAME';
            const user = context.actor;
            
            try {
              await github.rest.orgs.checkMembershipForUser({
                org,
                username: user
              });
              core.setOutput('allowed', 'true');
              console.log(`✓ User ${user} is a member of ${org}`);
            } catch (err) {
              core.setOutput('allowed', 'false');
              console.log(`✗ User ${user} is not a member of ${org}`);
            }

  claude-code-action:
    needs: check-membership
    if: needs.check-membership.outputs.is-member == 'true'
    runs-on: ubuntu-latest
    permissions:
      contents: read
      pull-requests: read
      issues: read
      id-token: write

    steps:
      - name: Checkout repository
        uses: actions/checkout@v4
        with:
          fetch-depth: 1

      - name: Run Claude PR Action
        uses: anthropics/claude-code-action@v1
        # Add your Claude Code Action configuration here
        # Example:
        # with:
        #   anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
        #   github_token: ${{ secrets.GITHUB_TOKEN }}

Caveat

ChatGPT identified a caveat where the REST endpoint GET /orgs/{org}/members/{username}/checkMembershipForUser may not reliably return membership for all cases when you call it using the default GITHUB_TOKEN in Actions:

GITHUB_TOKEN is convenient but does not have configurable read:org scope. In some orgs (or depending on API surface and GitHub changes), it cannot read private membership or certain org membership information. Community reports and the docs indicate you may need a token with read:org (or an app token with equivalent "Members" permission) to reliably check membership for all users.

Implication: a user with a private org membership could be treated as a non-member if you only use GITHUB_TOKEN for the check, or your check could fail unexpectedly.

I would prefer not to introduce a PAT since this adds a repo dependency upon the token-owning individual, and am ok with the tradeoff that non-public members of the Seqeralabs GitHub organization won't be able to invoke Claude.

Next Steps

1. Try solution above to see if it:
   1. It runs for Seqerans.
   2. Fails for non-Seqerans.
2. Solicit feedback from GHA experts in the org to make sure I'm not doing anything boneheaded security-wise.

[gwright99]

As it turns out, by not accommodating Private members, looks like I clobbered myself via a failed membership check:

Setting my Membership to Public and setting the organization back to "seqeralabs" succeeded:

[gwright99]

When testing with an account not yet affiliated with Seqera (ty @jhawkcates!), the GHA exited due to failed affiliation.

Unless Security reps (@arnaualcazar @aknownuser ) disagree, I'll consider the GHA sufficiently secured.

[gwright99]

Extra Security Consideration

Anthropic Claude Code Security Reviewer notes the following:

SecurityConsiderations

This action is not hardened against prompt injection attacks and should only be used to review trusted PRs. We recommend configuring your repository to use the "Require approval for all external contributors" option to ensure workflows only run after a maintainer has reviewed the PR.

Checking on existing configurations, it looks like the repo was already defended in the suggested manner. No further action required.

[gwright99]

Claude Bot (General) and Claude Security Bot both added in ea26495 .

Security Bot seems to be working but we won't really know until a more fulsome change makes its way through the PR process.

Pivoting back to other work for the time being.