Skip to content

ci(e2e): replace maestro with agent-device in expo native e2e - #9616

Draft
wobsoriano wants to merge 8 commits into
mainfrom
rob/agent-device-testing
Draft

ci(e2e): replace maestro with agent-device in expo native e2e#9616
wobsoriano wants to merge 8 commits into
mainfrom
rob/agent-device-testing

Conversation

@wobsoriano

@wobsoriano wobsoriano commented Aug 29, 2026

Copy link
Copy Markdown
Member

Description

Replaces Maestro with agent-device 0.20.10 as the engine behind the @clerk/expo native e2e jobs, adds Gradle caching to the three Android jobs, and fixes an Android <AuthView> first-mount bug the faster engine exposed (details below).

The flows are a 1:1 port. Every flows/*.yaml and flows/subflows/*.yaml has a .sh counterpart with the same name, the same assertions in the same order, and the same subflow reuse. lib.sh is a verb table with one bash function per Maestro verb (tap_on_id, wait_visible_text, input_text, run_flow, retry, ...) so a flow file reads like the YAML it replaced and platform differences stay in is_platform branches. run-flows.sh keeps the warmup launch, one session per flow, a single clean-state retry, force-stop between attempts, the step-summary timing table, and the debug artifact on failure (step log, screenshot, agent-device session state), with the password scrub extended to .ndjson and .ad files.

Two mappings are worth knowing about. Maestro's clearKeychain has no agent-device counterpart, so iOS clean state is settings clear-app-state plus xcrun simctl keychain reset before each launch. Maestro's eraseText and the iOS select-all dance are gone because agent-device's fill replaces the field value; the notVisible guard around retyping the prefilled identifier is kept. Selectors match text exactly, so tapping Continue cannot hit the Continue to <app> title, and waits treat an ambiguous match (two nodes with the same text) as visible.

Local runs against the SDK 57 fixture on an iPhone 17 Pro Max simulator and an API 34 emulator pass 7/7 on both platforms, first attempt, in about 2m40s per platform.

Cold CI comparison, every cache missed on both sides (Maestro run 33216124755 vs agent-device run 33269995365):

Maestro agent-device
iOS e2e step (flows plus engine warm-up) 10m39s 5m05s
Android e2e step (flows plus APK install) 7m23s 2m53s
Expo 57 iOS job 16m50s 9m58s
Expo 57 Android job 14m52s 13m31s

The Android job total is dominated by a 6m48s cold Gradle build and a 2m21s AVD snapshot in that run (4m22s and 1m39s in the Maestro run), which the Gradle cache added here removes on later runs: with it warm the fixture build takes 94 to 104 seconds instead of 256 to 445.

This is a second attempt after #9443, which used agent-device's --maestro compatibility engine and was closed over Android eraseText corruption and unsupported clearKeychain. Neither issue applies to the native CLI path.

Gradle caching uses gradle/actions/setup-gradle with cache-read-only: false, because this workflow only runs on pull requests and the action's default writes only from main. The first run of a PR seeds, later runs of the same PR hit.

I'd especially like a look at two things. First, sign-in-email-password.sh gained one conditional that never fires on the CI instance: when clerk-android lands on the email link screen (instances that prefer the magic link first), it taps Use another method and Sign in with your password. That is what let the suite run locally against a dev instance. Second, the faster stepping surfaced a real Android bug that Maestro's per-step latency hid, and this PR fixes it in @clerk/expo (patch changeset). If <AuthView> mounts within about two seconds of useAuth().isLoaded flipping, clerk-android has not finished loading its environment yet (configureNativeClerk waits for clerk-js to load before calling ClerkExpo.configure) and the sheet renders the logo, a bare Continue title and no form, and never recovers until it is dismissed and reopened. The first CI run of this branch reproduced it on the Android job in four of seven flows. ClerkAuthNativeView now keys the AuthView composition on Clerk.isInitialized, so the view is recreated once the SDK has loaded, the same thing a dismiss and reopen did by hand. A first attempt that instead held composition behind a spinner until loaded was dropped because the rehosted React logo view then never appeared in the accessibility tree. With the fix the Android suite passes 7/7 locally with the unchanged flows.

The first CI run also caught two iOS-only timing issues that are fixed here. The fresh simulator shows the one-time QuickPath keyboard tutorial sheet, whose own Continue button made the AuthView's Continue ambiguous, so the simulator prep now writes the com.apple.keyboard.preferences keys that mark those tutorials as seen (the same keys Chromium's and Detox's simulator setup use). And when the instance asks for an email code after the password, a bare settle returned before the next screen existed, so the subflow now waits for either the code screen or the signed-in state before deciding.

Checklist

  • pnpm test runs as expected.
  • pnpm build runs as expected.
  • (If applicable) JSDoc comments have been added or updated for any package exports
  • (If applicable) Documentation has been updated

Type of change

  • 🐛 Bug fix
  • 🌟 New feature
  • 🔨 Breaking change
  • 📖 Refactoring / dependency upgrade / documentation
  • other: CI / e2e tooling

https://claude.ai/code/session_01PioeNmoRqgukTH2PADUN1t

@changeset-bot

changeset-bot Bot commented Aug 29, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 1f99f01

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
Name Type
@clerk/expo Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@vercel

vercel Bot commented Aug 29, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
clerk-js-sandbox Ready Ready Preview Aug 29, 2026 7:22pm
swingset Ready Ready Preview Aug 29, 2026 7:22pm

Request Review

@pkg-pr-new

pkg-pr-new Bot commented Aug 29, 2026

Copy link
Copy Markdown

Open in StackBlitz

@clerk/astro

npm i https://pkg.pr.new/@clerk/astro@9616

@clerk/backend

npm i https://pkg.pr.new/@clerk/backend@9616

@clerk/chrome-extension

npm i https://pkg.pr.new/@clerk/chrome-extension@9616

@clerk/clerk-js

npm i https://pkg.pr.new/@clerk/clerk-js@9616

@clerk/electron

npm i https://pkg.pr.new/@clerk/electron@9616

@clerk/electron-passkeys

npm i https://pkg.pr.new/@clerk/electron-passkeys@9616

@clerk/eslint-plugin

npm i https://pkg.pr.new/@clerk/eslint-plugin@9616

@clerk/expo

npm i https://pkg.pr.new/@clerk/expo@9616

@clerk/expo-google-signin

npm i https://pkg.pr.new/@clerk/expo-google-signin@9616

@clerk/expo-passkeys

npm i https://pkg.pr.new/@clerk/expo-passkeys@9616

@clerk/express

npm i https://pkg.pr.new/@clerk/express@9616

@clerk/fastify

npm i https://pkg.pr.new/@clerk/fastify@9616

@clerk/hono

npm i https://pkg.pr.new/@clerk/hono@9616

@clerk/localizations

npm i https://pkg.pr.new/@clerk/localizations@9616

@clerk/nextjs

npm i https://pkg.pr.new/@clerk/nextjs@9616

@clerk/nuxt

npm i https://pkg.pr.new/@clerk/nuxt@9616

@clerk/react

npm i https://pkg.pr.new/@clerk/react@9616

@clerk/react-router

npm i https://pkg.pr.new/@clerk/react-router@9616

@clerk/shared

npm i https://pkg.pr.new/@clerk/shared@9616

@clerk/tanstack-react-start

npm i https://pkg.pr.new/@clerk/tanstack-react-start@9616

@clerk/testing

npm i https://pkg.pr.new/@clerk/testing@9616

@clerk/ui

npm i https://pkg.pr.new/@clerk/ui@9616

@clerk/upgrade

npm i https://pkg.pr.new/@clerk/upgrade@9616

@clerk/vue

npm i https://pkg.pr.new/@clerk/vue@9616

commit: 1f99f01

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant