CLI: Add gen-test-instructions command

[manzoorwanijk]

⚠️ This is a result of a session with Claude Code but the code has been reviewed and some minor changes have been made.

Here is the prompt used to generate the tool

For this repo/project, i.e. Jetpack, we have a release cycle for the Jetpack plugin (at projects/plugins/jetpack/). The stable release happens on the first Tuesday of every month, beta happens a day before the stable release and an alpha happens every Monday before that. We accumulate the changelog in projects/plugins/jetpack/CHANGELOG.md during alpha and beta releases and then combine them for the stable release. The week before the stable release is the testing week, where different people test the changes before the release. For that testing week, we need test instructions for the changes that happened during the cycle to ensure everything is alright. If you see the changelog, you can see every change being added as a bullet point with the PR number given in square brackets. Ideally, we would want a tool that uses AI which goes through the changelog, reads the entries, picks the PR number, checks the PR for instructions and then summarizes them in a consolidated form, avoiding repetitions. The instructions can be divided into sections with each section being a feature or the area of the plugin that is affected by the changes.
CLI would probably be a better solution as we would want to generate it locally and then review it before passing on. You may build a command just like the commands in tools/cli/commands

I then followed up with some more prompting to tweak things a bit like making PR numbers as hyperlinks.

Proposed changes:

A new tool at tools/gen-test-instructions.sh to automate the generation of consolidated test instructions for Jetpack plugin releases. This tool significantly reduces manual effort during testing week by automatically extracting, fetching, and consolidating test instructions from the changelog and GitHub PRs.

Two main features

1. AI-Powered Consolidation

• Uses Claude API to intelligently consolidate test instructions
• Groups related PRs by feature area
• Removes redundant test steps
• Organizes tests in logical order
• AI outputs PR numbers already formatted as markdown links
• Falls back to raw mode if API fails

2. Raw Output Mode

• Works without AI API key
• Organized by changelog sections
• Includes all PR details and links
• Good for quick reviews or when AI is not needed

Command Usage

# Generate since last stable release (auto-detected)
tools/gen-test-instructions.sh

# Generate since specific version
tools/gen-test-instructions.sh --version 15.1

# Specify output file
tools/gen-test-instructions.sh --output my-test-guide.md

# Skip AI consolidation (raw mode)
tools/gen-test-instructions.sh --skip-ai

# Use custom changelog path
tools/gen-test-instructions.sh --changelog path/to/CHANGELOG.md

# Filter by date
tools/gen-test-instructions.sh --since-date 2025-10-01

# With Anthropic API key for AI consolidation
tools/gen-test-instructions.sh --api-key sk-ant-...
# Or set environment variable
export ANTHROPIC_API_KEY=sk-ant-...
tools/gen-test-instructions.sh

Other information:

• Have you written new tests for your changes, if applicable?
• Have you checked the E2E test CI results, and verified that your changes do not break them?
• Have you tested your changes on WordPress.com, if applicable (if so, you'll see a generated comment below with a script to run)?

Jetpack product discussion

Does this pull request change what data or activity we track or use?

Testing instructions:

• Try the above commands

[github-actions]

Thank you for your PR!

When contributing to Jetpack, we have a few suggestions that can help us test and review your patch:

• ✅ Include a description of your PR changes.
  
• ✅ Add a "[Status]" label (In Progress, Needs Review, ...).
  
• ✅ Add a "[Type]" label (Bug, Enhancement, Janitorial, Task).
  
• ✅ Add testing instructions.
  
• ✅ Specify whether this PR includes any changes to data or privacy.
  
• ✅ Add changelog entries to affected projects
  

This comment will be updated as you work on your PR and make changes. If you think that some of those checks are not needed for your PR, please explain why you think so. Thanks for cooperation 🤖

---

Follow this PR Review Process:

1. Ensure all required checks appearing at the bottom of this PR are passing.
2. Make sure to test your changes on all platforms that it applies to. You're responsible for the quality of the code you ship.
3. You can use GitHub's Reviewers functionality to request a review.
4. When it's reviewed and merged, you will be pinged in Slack to deploy the changes to WordPress.com simple once the build is done.

If you have questions about anything, reach out in #jetpack-developers for guidance!

[Copilot]

Pull Request Overview

This PR introduces a new CLI tool to automate the generation of consolidated test instructions for Jetpack plugin releases. The tool extracts changelog entries, fetches PR details from GitHub, and optionally uses Claude AI to consolidate testing instructions into a structured markdown guide.

Key changes:

• Creates a Node.js-based tool that parses changelogs and integrates with GitHub API
• Implements AI-powered consolidation using Claude API for intelligent test instruction grouping
• Provides a bash wrapper script with comprehensive CLI interface and validation

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

File	Description
tools/js-tools/gen-test-instructions.mjs	Core Node.js implementation handling changelog parsing, GitHub API integration, and AI consolidation
tools/gen-test-instructions.sh	Bash wrapper providing CLI interface, input validation, and prerequisite checking

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[Copilot]

Pull Request Overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}