ci(WPB-21987): shard playwright tests across multiple runners #19807

markbrockhoff · 2025-11-25T10:58:57Z

WPB-21987 [WEB/QA] POC - Use sharding to run e2e tests in parallel in CI

Pull Request

Summary

Build and run Testservice as part of CI
Shard playwright tests across multiple runners
Include regression tests when running tests
Fix script reporting test results as github comment to work with nested test suites

Security Checklist (required)

External inputs are validated & sanitized on client and/or server where applicable.
API responses are validated; unexpected shapes are handled safely (fallbacks or errors).
No unsafe HTML is rendered; if unavoidable, sanitization is applied and documented where it happens.
Injection risks (XSS/SQL/command) are prevented via safe APIs and/or escaping.

Standards Acknowledgement (required)

I have read and this PR upholds our Coding Standards and Tech Radar Choices.

Notes for reviewers

I updated the node setup within the pipeline to respect the .nvmrc so we'll always use the same node version in CI as we do locally

codecov · 2025-11-25T11:07:07Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 43.48%. Comparing base (0315fc1) to head (6343db7).

Additional details and impacted files

@@            Coverage Diff             @@
##              dev   #19807      +/-   ##
==========================================
+ Coverage   43.46%   43.48%   +0.01%     
==========================================
  Files        1296     1296              
  Lines       32570    32570              
  Branches     7232     7232              
==========================================
+ Hits        14157    14162       +5     
+ Misses      16698    16694       -4     
+ Partials     1715     1714       -1

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

github-actions · 2025-11-25T11:22:18Z

🔗 Download Full Report Artifact

🧪 Playwright Test Summary

✅ Passed: 69
❌ Failed: 9
⏭ Skipped: 6
🔁 Flaky: 1
📊 Total: 85
⏱ Total Runtime: 300.9s (~ 5 min 1 sec)

Failed Tests:

❌ I should not be able to change email of user managed by SCIM (tags: TC-60, regression)

Location: specs/AccountSettingsSpecs/accountSettings.spec.ts:87
Duration: 87619ms

Errors:

TimeoutError: locator.waitFor: Timeout 60000ms exceeded.
Call log:
  - waiting for locator('[data-uie-name="go-preferences"]') to be visible


   at pageManager/webapp/components/conversationSidebar.component.ts:64

  62 |
  63 |   async isPageLoaded() {
> 64 |     await this.preferencesButton.waitFor({state: 'visible', timeout: this.pageLoadingTimeout});
     |                                  ^
  65 |   }
  66 |
  67 |   async clickArchive() {
    at ConversationSidebar.isPageLoaded (/home/runner/work/wire-webapp/wire-webapp/test/e2e_tests/pageManager/webapp/components/conversationSidebar.component.ts:64:34)
    at /home/runner/work/wire-webapp/wire-webapp/test/e2e_tests/specs/AccountSettingsSpecs/accountSettings.spec.ts:115:46

❌ Verify links to manage and create teams are shown when logged in as team owner (tags: TC-1723, regression)

Location: specs/AccountSettingsSpecs/accountSettings.spec.ts:159
Duration: 26133ms

Errors:

Error: expect(received).toBe(expected) // Object.is equality

Expected: "https://wire-teams-dev.zinfra.io/login/"
Received: "https://wire-teams-staging.zinfra.io/login/"

  166 |
  167 |       await expect(components.conversationSidebar().manageTeamButton).toBeVisible();
> 168 |       expect(await components.conversationSidebar().manageTeamButton.getAttribute('href')).toBe(
      |                                                                                            ^
  169 |         'https://wire-teams-dev.zinfra.io/login/',
  170 |       );
  171 |     },
    at /home/runner/work/wire-webapp/wire-webapp/test/e2e_tests/specs/AccountSettingsSpecs/accountSettings.spec.ts:168:92

❌ Web: App should not lock if I switch back to webapp tab in time (during inactivity timeout) (tags: TC-2752, TC-2753, regression)

Location: specs/AppLock/AppLock.spec.ts:60
Duration: 70068ms

Errors:

Error: expect(locator).toBeVisible() failed

Locator: locator('[data-uie-name=\'applock-modal\'] [data-uie-name=\'applock-modal-header\']')
Expected: visible
Timeout: 10000ms
Error: element(s) not found

Call log:
  - Expect "toBeVisible" with timeout 10000ms
  - waiting for locator('[data-uie-name=\'applock-modal\'] [data-uie-name=\'applock-modal-header\']')


  80 |         await unrelatedPage.waitForTimeout(31_000);
  81 |         await webappPageA.bringToFront();
> 82 |         await expect(modals.appLock().appLockModalHeader).toBeVisible();
     |                                                           ^
  83 |       });
  84 |     },
  85 |   );
    at /home/runner/work/wire-webapp/wire-webapp/test/e2e_tests/specs/AppLock/AppLock.spec.ts:82:59
    at /home/runner/work/wire-webapp/wire-webapp/test/e2e_tests/specs/AppLock/AppLock.spec.ts:77:7

❌ I want to be asked to share telemetry data when I log in (tags: TC-8780, regression)

Location: specs/Authentication/authentication.spec.ts:23
Duration: 16071ms

Errors:

Error: expect(locator).toBeVisible() failed

Locator: locator('[data-uie-name="primary-modals-container"][aria-label=\'Consent to share user data\']').locator('[data-uie-name="status-modal-title"]')
Expected: visible
Timeout: 10000ms
Error: element(s) not found

Call log:
  - Expect "toBeVisible" with timeout 10000ms
  - waiting for locator('[data-uie-name="primary-modals-container"][aria-label=\'Consent to share user data\']').locator('[data-uie-name="status-modal-title"]')


  31 |       await pages.login().login(user);
  32 |
> 33 |       await expect(modals.dataShareConsent().modalTitle).toBeVisible();
     |                                                          ^
  34 |     },
  35 |   );
  36 | });
    at /home/runner/work/wire-webapp/wire-webapp/test/e2e_tests/specs/Authentication/authentication.spec.ts:33:58

❌ Verify 1on1 conversation is not created on the second end after you ignore connection request (tags: TC-365, TC-369, TC-370, TC-371, regression)

Location: specs/Connections/Connections.spec.ts:36
Duration: 50910ms

Errors:

TimeoutError: locator.click: Timeout 20000ms exceeded.
Call log:
  - waiting for getByTestId('[data-uie-name=\'primary-modals-container\']').getByTestId('do-action')


   at pageManager/webapp/modals/base.modal.ts:60

  58 |
  59 |   async clickAction() {
> 60 |     await this.actionButton.click();
     |                             ^
  61 |   }
  62 | }
  63 |
    at CancelRequestModal.clickAction (/home/runner/work/wire-webapp/wire-webapp/test/e2e_tests/pageManager/webapp/modals/base.modal.ts:60:29)
    at /home/runner/work/wire-webapp/wire-webapp/test/e2e_tests/specs/Connections/Connections.spec.ts:60:38
    at /home/runner/work/wire-webapp/wire-webapp/test/e2e_tests/specs/Connections/Connections.spec.ts:58:7

❌ Editing a message does not create unread dot on receiver side (tags: TC-690, regression)

Location: specs/Edit/edit.spec.ts:124
Duration: 48198ms

Errors:

TimeoutError: locator.hover: Timeout 20000ms exceeded.
Call log:
  - waiting for locator('[data-uie-name="item-message"][data-uie-send-status="2"]').filter({ has: locator('.content-message-wrapper[aria-label*="Robb Kiehn"]') })


   at pageManager/webapp/pages/conversation.page.ts:251

  249 |    */
  250 |   async openMessageOptions(message: Locator) {
> 251 |     await message.hover();
      |                   ^
  252 |     await message.getByTestId('message-actions').getByTestId('go-options').click();
  253 |     // The context menu containing the edit button is positioned globally as an overlay
  254 |     return this.page.getByRole('menu');
    at ConversationPage.openMessageOptions (/home/runner/work/wire-webapp/wire-webapp/test/e2e_tests/pageManager/webapp/pages/conversation.page.ts:251:19)
    at ConversationPage.editMessage (/home/runner/work/wire-webapp/wire-webapp/test/e2e_tests/pageManager/webapp/pages/conversation.page.ts:259:29)
    at /home/runner/work/wire-webapp/wire-webapp/test/e2e_tests/specs/Edit/edit.spec.ts:160:41
    at /home/runner/work/wire-webapp/wire-webapp/test/e2e_tests/specs/Edit/edit.spec.ts:158:18

❌ Verify you can sign in by email (tags: TC-3461, regression)

Location: specs/LoginSpecs/login.spec.ts:38
Duration: 18868ms

Errors:

Error: expect(locator).toHaveText(expected) failed

Locator: locator('[data-uie-name="status-name"]')
Expected: "Marietta Terry"
Timeout: 10000ms
Error: element(s) not found

Call log:
  - Expect "toHaveText" with timeout 10000ms
  - waiting for locator('[data-uie-name="status-name"]')


  47 |   await pages.login().signInButton.click();
  48 |
> 49 |   await expect(components.conversationSidebar().personalStatusName).toHaveText(`${user.firstName} ${user.lastName}`);
     |                                                                     ^
  50 |   await expect(components.conversationSidebar().personalUserName).toContainText(user.username);
  51 | });
  52 |
    at /home/runner/work/wire-webapp/wire-webapp/test/e2e_tests/specs/LoginSpecs/login.spec.ts:49:69

❌ I want to see a placeholder text as quote when original message is not available anymore (tags: TC-2994, regression)

Location: specs/Reply/reply.spec.ts:60
Duration: 29225ms

Errors:

TimeoutError: locator.click: Timeout 20000ms exceeded.
Call log:
  - waiting for locator('button[data-uie-name="go-people"]')


   at pageManager/webapp/components/conversationSidebar.component.ts:60

  58 |
  59 |   async clickConnectButton() {
> 60 |     await this.connectButton.click();
     |                              ^
  61 |   }
  62 |
  63 |   async isPageLoaded() {
    at ConversationSidebar.clickConnectButton (/home/runner/work/wire-webapp/wire-webapp/test/e2e_tests/pageManager/webapp/components/conversationSidebar.component.ts:60:30)
    at connectWithUser (/home/runner/work/wire-webapp/wire-webapp/test/e2e_tests/utils/userActions.ts:152:42)
    at /home/runner/work/wire-webapp/wire-webapp/test/e2e_tests/test.fixtures.ts:162:26
    at /home/runner/work/wire-webapp/wire-webapp/test/e2e_tests/test.fixtures.ts:90:9
    at /home/runner/work/wire-webapp/wire-webapp/test/e2e_tests/specs/Reply/reply.spec.ts:64:40

❌ I should not be able to send a reply after I got removed from the conversation (tags: TC-3014, regression)

Location: specs/Reply/reply.spec.ts:288
Duration: 29772ms

Errors:

TimeoutError: locator.click: Timeout 20000ms exceeded.
Call log:
  - waiting for locator('[data-uie-name="conversation-list-header"] [data-uie-name="go-create-group"]')


   at pageManager/webapp/pages/conversationList.page.ts:104

  102 |
  103 |   async clickCreateGroup() {
> 104 |     await this.createGroupButton.click();
      |                                  ^
  105 |   }
  106 |
  107 |   getConversationLocator(conversationName: string) {
    at ConversationListPage.clickCreateGroup (/home/runner/work/wire-webapp/wire-webapp/test/e2e_tests/pageManager/webapp/pages/conversationList.page.ts:104:34)
    at createGroup (/home/runner/work/wire-webapp/wire-webapp/test/e2e_tests/utils/userActions.ts:84:34)
    at /home/runner/work/wire-webapp/wire-webapp/test/e2e_tests/specs/Reply/reply.spec.ts:296:24

Flaky Tests:

⚠️ Verify the conversation is not unarchived when there are new messages in this conversation (tags: TC-99, regression)

Location: specs/ArchiveSpecs/archive.spec.ts:63

Attempt 1
Result: ❌ Failed
Duration: 48968ms

Errors:

TimeoutError: locator.click: Timeout 20000ms exceeded.
Call log:
  - waiting for locator('[data-uie-name="go-recent-view"]')
    - locator resolved to <button id="tab-1" role="tab" title="All" type="button" aria-selected="true" data-uie-status="active" data-uie-name="go-recent-view" class="conversations-sidebar-btn active">…</button>
  - attempting click action
    2 × waiting for element to be visible, enabled and stable
      - element is visible, enabled and stable
      - scrolling into view if needed
      - done scrolling
      - <div tabindex="0" role="dialog" aria-modal="true" class=" css-1ocmba1" data-uie-name="modal-template-acknowledge">…</div> from <div class="css-18o04gs">…</div> subtree intercepts pointer events
    - retrying click action
    - waiting 20ms
    2 × waiting for element to be visible, enabled and stable
      - element is visible, enabled and stable
      - scrolling into view if needed
      - done scrolling
      - <div tabindex="0" role="dialog" aria-modal="true" class=" css-1ocmba1" data-uie-name="modal-template-acknowledge">…</div> from <div class="css-18o04gs">…</div> subtree intercepts pointer events
    - retrying click action
      - waiting 100ms
    39 × waiting for element to be visible, enabled and stable
       - element is visible, enabled and stable
       - scrolling into view if needed
       - done scrolling
       - <div tabindex="0" role="dialog" aria-modal="true" class=" css-1ocmba1" data-uie-name="modal-template-acknowledge">…</div> from <div class="css-18o04gs">…</div> subtree intercepts pointer events
     - retrying click action
       - waiting 500ms


   at pageManager/webapp/components/conversationSidebar.component.ts:56

  54 |
  55 |   async clickAllConversationsButton() {
> 56 |     await this.allConverationsButton.click();
     |                                      ^
  57 |   }
  58 |
  59 |   async clickConnectButton() {
    at ConversationSidebar.clickAllConversationsButton (/home/runner/work/wire-webapp/wire-webapp/test/e2e_tests/pageManager/webapp/components/conversationSidebar.component.ts:56:38)
    at /home/runner/work/wire-webapp/wire-webapp/test/e2e_tests/specs/ArchiveSpecs/archive.spec.ts:92:47

Attempt 2
Result: ✅ Passed
Duration: 36294ms

.github/workflows/precommit-crit-flows.yml

markbrockhoff · 2025-11-26T11:54:08Z

.github/workflows/precommit-crit-flows.yml

+          distribution: 'temurin'
+
+      - name: Gradle Cache
+        uses: gradle/actions/setup-gradle@4d9f0ba0025fe599b4ebab900eb7f3a1d93ef4c2 # SHA of tag v5.0.0


Fyi: the only reason I pinned the actions version to a specific commit is because sonar was worried of a supply chain attack. Not sure what's better, having to manually update every time to get e.g. security patches or the risk of malicious code being merged into the action unnoticed.

… & job to collect reports

…eport for PR comment

…st suites

(So Sonar shuts up)

The report annotations are too annoying

sonarqubecloud · 2025-11-26T14:55:45Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

otto-the-bot added comp: infrastructure 👕 size: M labels Nov 25, 2025

markbrockhoff force-pushed the ci/poc-shard-playwright-tests branch from 7345963 to 3e192de Compare November 25, 2025 11:00

markbrockhoff force-pushed the ci/poc-shard-playwright-tests branch 5 times, most recently from 0c402c9 to 9767bf3 Compare November 25, 2025 16:05

markbrockhoff marked this pull request as ready for review November 25, 2025 16:05

markbrockhoff requested review from a team and otto-the-bot as code owners November 25, 2025 16:05

markbrockhoff changed the title ~~ci: shard playwright tests across multiple runners~~ ci(WPB-21987): shard playwright tests across multiple runners Nov 25, 2025

arjita-mitra reviewed Nov 26, 2025

View reviewed changes

.github/workflows/precommit-crit-flows.yml Show resolved Hide resolved

arjita-mitra reviewed Nov 26, 2025

View reviewed changes

.github/workflows/precommit-crit-flows.yml Outdated Show resolved Hide resolved

markbrockhoff requested a review from arjita-mitra November 26, 2025 09:35

markbrockhoff commented Nov 26, 2025

View reviewed changes

markbrockhoff added 12 commits November 26, 2025 15:54

refactor: use nvmrc to select node version

a131263

feat: add job to build test service

a31a393

feat: shard tests across multiple runners with integrated testservice…

0a0bdd6

… & job to collect reports

fix: build shadowJar of testservice

b09b647

fix: generate json report when merging blobs

f44fd6e

refactor: only download chrome browser for tests

aca079c

ci: use github reporter in CI

a5226fc

fix: re-use playwright config when merging reports to generate json r…

6155afc

…eport for PR comment

refactor: don't pass github user and token when building testservice

59f632a

feat: use 16 runners for sharding

bbf4713

feat: execute regression tests in addition to critical flow

75fad0a

fix: make script for reporting playwright summary work with nested te…

f5e3c06

…st suites

markbrockhoff added 3 commits November 26, 2025 15:54

chore: use SHA for third party actions

a0b1583

(So Sonar shuts up)

refactor: use line reporter instead of github

62bfa16

The report annotations are too annoying

refactor: stop background process of testservice after test

6343db7

markbrockhoff force-pushed the ci/poc-shard-playwright-tests branch from 59a83fa to 6343db7 Compare November 26, 2025 14:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

ci(WPB-21987): shard playwright tests across multiple runners #19807

ci(WPB-21987): shard playwright tests across multiple runners #19807

Uh oh!

markbrockhoff commented Nov 25, 2025 •

edited by github-actions bot

Loading

Uh oh!

codecov bot commented Nov 25, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Nov 25, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

markbrockhoff Nov 26, 2025

Uh oh!

sonarqubecloud bot commented Nov 26, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ci(WPB-21987): shard playwright tests across multiple runners #19807

Are you sure you want to change the base?

ci(WPB-21987): shard playwright tests across multiple runners #19807

Uh oh!

Conversation

markbrockhoff commented Nov 25, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request

Summary

Security Checklist (required)

Standards Acknowledgement (required)

Notes for reviewers

Uh oh!

codecov bot commented Nov 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions bot commented Nov 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🧪 Playwright Test Summary

Failed Tests:

Flaky Tests:

Uh oh!

Uh oh!

Uh oh!

markbrockhoff Nov 26, 2025

Choose a reason for hiding this comment

Uh oh!

sonarqubecloud bot commented Nov 26, 2025

Quality Gate passed

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

markbrockhoff commented Nov 25, 2025 •

edited by github-actions bot

Loading

codecov bot commented Nov 25, 2025 •

edited

Loading

github-actions bot commented Nov 25, 2025 •

edited

Loading