ci(e2e): replace maestro with agent-device in expo native e2e - #9616
Draft
wobsoriano wants to merge 8 commits into
Draft
ci(e2e): replace maestro with agent-device in expo native e2e#9616wobsoriano wants to merge 8 commits into
wobsoriano wants to merge 8 commits into
Conversation
🦋 Changeset detectedLatest commit: 1f99f01 The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
@clerk/astro
@clerk/backend
@clerk/chrome-extension
@clerk/clerk-js
@clerk/electron
@clerk/electron-passkeys
@clerk/eslint-plugin
@clerk/expo
@clerk/expo-google-signin
@clerk/expo-passkeys
@clerk/express
@clerk/fastify
@clerk/hono
@clerk/localizations
@clerk/nextjs
@clerk/nuxt
@clerk/react
@clerk/react-router
@clerk/shared
@clerk/tanstack-react-start
@clerk/testing
@clerk/ui
@clerk/upgrade
@clerk/vue
commit: |
…st-password screen Claude-Session: https://claude.ai/code/session_01PioeNmoRqgukTH2PADUN1t
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Replaces Maestro with agent-device 0.20.10 as the engine behind the
@clerk/exponative e2e jobs, adds Gradle caching to the three Android jobs, and fixes an Android<AuthView>first-mount bug the faster engine exposed (details below).The flows are a 1:1 port. Every
flows/*.yamlandflows/subflows/*.yamlhas a.shcounterpart with the same name, the same assertions in the same order, and the same subflow reuse.lib.shis a verb table with one bash function per Maestro verb (tap_on_id,wait_visible_text,input_text,run_flow,retry, ...) so a flow file reads like the YAML it replaced and platform differences stay inis_platformbranches.run-flows.shkeeps the warmup launch, one session per flow, a single clean-state retry, force-stop between attempts, the step-summary timing table, and the debug artifact on failure (step log, screenshot, agent-device session state), with the password scrub extended to.ndjsonand.adfiles.Two mappings are worth knowing about. Maestro's
clearKeychainhas no agent-device counterpart, so iOS clean state issettings clear-app-stateplusxcrun simctl keychain resetbefore each launch. Maestro'seraseTextand the iOS select-all dance are gone because agent-device'sfillreplaces the field value; thenotVisibleguard around retyping the prefilled identifier is kept. Selectors match text exactly, so tappingContinuecannot hit theContinue to <app>title, and waits treat an ambiguous match (two nodes with the same text) as visible.Local runs against the SDK 57 fixture on an iPhone 17 Pro Max simulator and an API 34 emulator pass 7/7 on both platforms, first attempt, in about 2m40s per platform.
Cold CI comparison, every cache missed on both sides (Maestro run 33216124755 vs agent-device run 33269995365):
The Android job total is dominated by a 6m48s cold Gradle build and a 2m21s AVD snapshot in that run (4m22s and 1m39s in the Maestro run), which the Gradle cache added here removes on later runs: with it warm the fixture build takes 94 to 104 seconds instead of 256 to 445.
This is a second attempt after #9443, which used agent-device's
--maestrocompatibility engine and was closed over AndroideraseTextcorruption and unsupportedclearKeychain. Neither issue applies to the native CLI path.Gradle caching uses
gradle/actions/setup-gradlewithcache-read-only: false, because this workflow only runs on pull requests and the action's default writes only frommain. The first run of a PR seeds, later runs of the same PR hit.I'd especially like a look at two things. First,
sign-in-email-password.shgained one conditional that never fires on the CI instance: when clerk-android lands on the email link screen (instances that prefer the magic link first), it tapsUse another methodandSign in with your password. That is what let the suite run locally against a dev instance. Second, the faster stepping surfaced a real Android bug that Maestro's per-step latency hid, and this PR fixes it in@clerk/expo(patch changeset). If<AuthView>mounts within about two seconds ofuseAuth().isLoadedflipping, clerk-android has not finished loading its environment yet (configureNativeClerkwaits for clerk-js to load before callingClerkExpo.configure) and the sheet renders the logo, a bareContinuetitle and no form, and never recovers until it is dismissed and reopened. The first CI run of this branch reproduced it on the Android job in four of seven flows.ClerkAuthNativeViewnow keys theAuthViewcomposition onClerk.isInitialized, so the view is recreated once the SDK has loaded, the same thing a dismiss and reopen did by hand. A first attempt that instead held composition behind a spinner until loaded was dropped because the rehosted React logo view then never appeared in the accessibility tree. With the fix the Android suite passes 7/7 locally with the unchanged flows.The first CI run also caught two iOS-only timing issues that are fixed here. The fresh simulator shows the one-time QuickPath keyboard tutorial sheet, whose own
Continuebutton made the AuthView'sContinueambiguous, so the simulator prep now writes thecom.apple.keyboard.preferenceskeys that mark those tutorials as seen (the same keys Chromium's and Detox's simulator setup use). And when the instance asks for an email code after the password, a bare settle returned before the next screen existed, so the subflow now waits for either the code screen or the signed-in state before deciding.Checklist
pnpm testruns as expected.pnpm buildruns as expected.Type of change
✎
https://claude.ai/code/session_01PioeNmoRqgukTH2PADUN1t