Add: AI-assisted payload generation to the Ability Explorer - Test Ability Screen by Arkenon · Pull Request #695 · WordPress/ai

Arkenon · 2026-06-10T11:07:58Z

What?

Adds AI-assisted payload generation to the Ability Test Runner. Users can describe a test scenario in natural language and automatically generate a valid JSON payload based on the selected ability's input schema.

Why?

Some abilities expose complex or deeply nested input schemas that are difficult to populate manually. Creating valid test payloads often requires understanding schema structures, enum values, nested objects, and validation constraints.

This makes testing abilities slower and more error-prone, especially for contributors who are unfamiliar with a specific ability's schema.

How?

ability_test_generate_with_ai.mp4

This PR introduces a new Generate with AI workflow in the Ability Test Runner:

Added a Generate with AI button to the test interface.
Added a modal that accepts a natural-language description of the desired test input.
Sends the ability's input schema and the user's description to the configured AI provider using wp_ai_client_prompt().
Parses the AI response, removes markdown code fencing if present, validates the JSON, and inserts the generated payload into the test input field.
The feature is only available when:
- An AI provider is configured.
- The selected ability exposes an input schema.

Implementation Details

Admin_Page.php
- Added ajax_generate_payload() AJAX handler.
- Retrieves the ability schema and user prompt.
- Calls the configured AI provider via wp_ai_client_prompt().
- Sanitizes and parses the response before returning JSON.
Abilities_Explorer.php
- Added generateNonce (ai_ability_explorer_generate_payload) for AJAX authentication.
GeneratePayloadModal.jsx
- New modal component for collecting user instructions and displaying errors.
index.js
- Added modal mounting/unmounting logic and integration with the Ability Test Runner UI.
index.scss
- Added styling for the modal footer and error notice area.

Use of AI Tools

AI assistance: Yes

Tool(s): ChatGPT

Model(s): GPT-5.5

Used for: Brainstorming the feature design, refining prompts, and drafting portions of documentation. Final implementation, review, testing, and code decisions were completed by me.

Testing Instructions

Navigate to Tools → Abilities Explorer.
Open an ability that exposes a non-trivial input schema.
Click Test Ability.
Click Generate with AI.
Enter a natural-language instruction such as:
- "Use the default values"
- "Search for posts with the title 'Hello'"
- "Query only the site URL information"
Click Generate.
Verify that the payload textarea is populated with valid JSON matching the schema.
Click Invoke Ability.
Verify that the ability executes successfully with the generated payload.

Note: An AI provider must be connected in the plugin settings for this feature to be available.

Screenshots or screencast

Before | After -- | -- N/A | Generate with AI button and payload generation modal

Changelog Entry

Added - AI-assisted payload generation for the Ability Test Runner.

…ility page.

github-actions · 2026-06-10T11:08:12Z

The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the props-bot label.

If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message.

Co-authored-by: Arkenon <arkenon@git.wordpress.org>
Co-authored-by: dkotter <dkotter@git.wordpress.org>

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.

Arkenon · 2026-06-19T08:51:51Z

Any feedback or suggestions on this PR?

codecov · 2026-06-23T15:11:34Z

Codecov Report

❌ Patch coverage is 1.35135% with 73 lines in your changes missing coverage. Please review.
✅ Project coverage is 79.14%. Comparing base (abcd7c9) to head (bff25bb).

Files with missing lines	Patch %	Lines
...udes/Experiments/Abilities_Explorer/Admin_Page.php	1.44%	68 Missing ⚠️
...eriments/Abilities_Explorer/Abilities_Explorer.php	0.00%	5 Missing ⚠️

Additional details and impacted files

@@              Coverage Diff              @@
##             develop     #695      +/-   ##
=============================================
- Coverage      79.68%   79.14%   -0.55%     
- Complexity      2460     2478      +18     
=============================================
  Files            104      104              
  Lines           9955    10025      +70     
=============================================
+ Hits            7933     7934       +1     
- Misses          2022     2091      +69

Flag	Coverage Δ
unit	`79.14% <1.35%> (-0.55%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

dkotter

I've not tested yet but a number of things that need cleaned up here. In addition, would be great to get test coverage here, unit tests potentially but definitely E2E tests.

While the overall idea is neat here, the overall flow I think could be improved a bit. For instance, the modal itself has a title of Generate Payload with AI but then in the textarea, you add a prompt that says Generate payload for ....

We've been trying to remove the use of with AI, so I think just Generate Payload both in the modal and the button that triggers the modal will be better. I'm also not sure about requiring a user to enter details about their desired payload. Could the generate function not just generate it's own payload without user input? I understand that gets trickier if a post is required, for example, but seems solvable. Just seems to make it less user friendly to click a generate button but then be required to manually enter in a bunch of information first

dkotter · 2026-06-23T15:13:36Z

+					onClose: close,
+					onSuccess: handleSuccess,
+					abilitySlug,
+					strings: aiAbilityExplorer.strings,


We can use the client-side __ functions instead of passing in strings from the server-side

Arkenon · 2026-06-26T13:20:18Z

@dkotter Thanks for your review.

I just linted the PHP files, and I'm not sure why that extra spacing was added, but I'll fix it throughout the codebase.

I'm currently working on the feature. I'll also look into generating the payload automatically instead of using a prompt.

I'll submit an updated PR as soon as possible.

…dPress Abilities API registrations

add: Auto create payload with button

…specified in iterable type array.

Arkenon · 2026-07-21T05:39:02Z

I have completed the changes. The Test Ability screen now only has a Generate Payload button. The AI agent generates the payload based on the input schema.

ScreenRecorderProject3.mp4

dkotter

This is looking better but still have a few things that need addressed before we proceed.

In addition to the things flagged, we need to ensure we get some E2E tests added and ideally let's get the PR description updated to match latest changes

dkotter · 2026-07-21T16:21:08Z

+			"Ability description: %s\n" .
+			"The payload must be a JSON object that conforms to the following JSON schema:\n%s\n" .
+			'Use plausible, realistic example values for every required property. If post id needed set id as 1.' .
+			'Include optional properties only when they help demonstrate the ability.',


The instruction here around optional properties contradicts what we do in normalize_schema_for_response, which forces all properties to be required. Ideally we should determine which approach we want to take and ensure we aren't providing conflicting instructions

dkotter · 2026-07-21T16:22:31Z

+			"Generate a realistic example test payload for the WordPress ability \"%s\".\n" .
+			"Ability description: %s\n" .
+			"The payload must be a JSON object that conforms to the following JSON schema:\n%s\n" .
+			'Use plausible, realistic example values for every required property. If post id needed set id as 1.' .


There's a couple issues here:

Some sites may not have a post ID of 1. We could find a real post and pass the ID in

What if the input schema wants a post ID but doesn't want it in an id field? For instance, if it asks for post_id instead? I think we need to be a little more generic on this instruction

Also need a newline break after this line otherwise it runs into the next line

dkotter · 2026-07-21T16:26:02Z

 					<button type="button" id="ability-test-clear" class="button">
 						<?php esc_html_e( 'Clear Result', 'ai' ); ?>
 					</button>
+					<?php if ( ! empty( $ability['input_schema'] ) ) : ?>


Would we want a check here to ensure we have an AI connector in place before we output this button? Or if not, we should ensure we output a better error message in that situation so users know what went wrong

dkotter · 2026-07-21T16:26:52Z

+				if ( ! button.dataset.originalText ) {
+					button.dataset.originalText = originalText;
+				}
+				button.innerHTML =


Can we avoid using innerHTML here? We should be able to build the span node and assign textContent to it

dkotter · 2026-07-21T16:27:31Z

Seems like a bunch of unnecessary reformatting on this file. Ideally we revert that to keep the changes here focused

Arkenon and others added 2 commits June 10, 2026 13:39

feat: Add Generate with AI button into the Ability Explorer - Test Ab…

6cc4d73

…ility page.

Merge branch 'WordPress:develop' into develop

32ea4d7

dkotter assigned Arkenon Jun 23, 2026

Merge branch 'develop' into develop

82cac3e

dkotter requested changes Jun 23, 2026

View reviewed changes

dkotter added this to the Future Release milestone Jun 23, 2026

Merge branch 'WordPress:develop' into develop

73cc927

Arkenon and others added 2 commits June 26, 2026 16:53

feat: implement Abilities Explorer experiment to inspect and test Wor…

b1bd247

…dPress Abilities API registrations

Merge branch 'WordPress:develop' into feature/pr-695

2815113

jeffpaul marked this pull request as draft June 26, 2026 17:32

jeffpaul added this to WordPress AI Roadmap Jun 26, 2026

github-project-automation Bot moved this to Triage in WordPress AI Roadmap Jun 26, 2026

jeffpaul moved this from Triage to In progress in WordPress AI Roadmap Jun 26, 2026

Arkenon and others added 11 commits July 13, 2026 13:47

Merge branch 'WordPress:develop' into feature/pr-695

5f6bac0

Merge branch 'WordPress:develop' into develop

4390a2e

Merge branch 'WordPress:develop' into feature/pr-695

ac20a05

remove: Modal payload generator

5f7c486

add: Auto create payload with button

Merge branch 'WordPress:develop' into feature/pr-695

809882a

feat: Lint & format files

5f635e3

lint&format files

3cd57e5

lint&format files

73e6c5b

Merge branch 'develop' of https://github.com/Arkenon/ai into develop

735375f

Merge branch 'WordPress:develop' into develop

44acc8b

fixed: normalize_schema_for_response() return type has no value type …

bff25bb

…specified in iterable type array.

Arkenon marked this pull request as ready for review July 21, 2026 05:33

Arkenon requested a review from a team July 21, 2026 05:33

dkotter requested changes Jul 21, 2026

View reviewed changes

Uh oh!

Conversation

Arkenon commented Jun 10, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What?

Why?

How?

Implementation Details

Use of AI Tools

Testing Instructions

Screenshots or screencast

Changelog Entry

Uh oh!

github-actions Bot commented Jun 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Arkenon commented Jun 19, 2026

Uh oh!

codecov Bot commented Jun 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

dkotter left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dkotter Jun 23, 2026

Choose a reason for hiding this comment

Uh oh!

Arkenon commented Jun 26, 2026

Uh oh!

Arkenon commented Jul 21, 2026

Uh oh!

dkotter left a comment

Choose a reason for hiding this comment

Uh oh!

dkotter Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

dkotter Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

dkotter Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

dkotter Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

dkotter Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

dkotter Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Arkenon commented Jun 10, 2026 •

edited by github-actions Bot

Loading

github-actions Bot commented Jun 10, 2026 •

edited

Loading

codecov Bot commented Jun 23, 2026 •

edited

Loading