feat(amazonq): package.json npm scripts for build issues with vscode-extension-tester (VET) #7666

surajrdy-aws · 2025-07-14T21:08:33Z

Problem

Users running our new E2E UI testing suite have to run a plethora of commands in order to setup and run our test suite each time their testing code may change or a new workspace in VSCode is instantiated. This document contains rationale. These commands are exact and can be easy to forget and small errors in the CLI commands lead to headaches with the setup. Additionally, we run into a problem when running npm run package where each VSIX installation will have a unique hash and VSCode version attached to the name, and due to the lack of CLI args for renaming our VSIX package, we must be able to reference the path to this VSIX installation on anyones local devices. Notably, many Vscode-extension-tester (VET) projects use "setup-tests" to attach a VSCode instance and a Chromedriver instance which is crucial for the functionality of VET; however, this leads to build issues due to the inherent nature of VET's "setup-tests" packaging its own VSIX automatically. In normal circumstances, this would be fine; however, due to our unique packaging structure of the amazonq directory, we require a package.ts script to package our directory properly.

Solution

My solution encompasses adding additional scripts to the package.json file in the root directory of the aws-toolkit-vscode repository. These scripts must be run in a specific order in order for them to work correctly and must all by run in the root directory. Namely these npm scripts are:

"test:ui:prepare": "./node_modules/.bin/extest get-vscode -s ~/.vscode-test-resources -n && extest get-chromedriver -s ~/.vscode-test-resources -n",

This script essentially does the bulk of the previous "setup-tests", it installs a local VSCode instance and clears the previous cache and installs the chromdriver into the VSCode instance. This is stored in a vscode-test-resources directory in ones local User directory.

"test:ui:install": "cd packages/amazonq && npm run package 2>&1 | grep -o 'VSIX Version: [^ ]*' | cut -d' ' -f3 | xargs -I{} bash -c 'cd ../../ && ./node_modules/.bin/extest install-vsix -f amazon-q-vscode-{}.vsix -e packages/amazonq/test/e2e/amazonq/resources -s ~/.vscode-test-resources'",

This script installs our VSIX by going into the amazonq package and running our npm run package. This eliminates the need for any additional scripts by allowing for a CLI/RegEx command to be utilized to find the specific hash of the VSIX in our terminal output and input it within the install-vsix command for extest.

"test:ui:run": "npm run testCompile && ./node_modules/.bin/extest run-tests -s ~/.vscode-test-resources -e packages/amazonq/test/e2e/amazonq/resources packages/amazonq/dist/test/e2e/amazonq/VET.test.js",

This script runs our VET.test.js script and does the testCompile npm script that has already been built into the repo. This is because when we run "npm run package", our compiled tests within the dist directory are deleted. We must run testCompile after this command in order to properly compile our tests.

"test:ui": "npm run test:ui:prepare && npm run test:ui:install && npm run test:ui:run",

This script combines all of our scripts into a chronological order.

Treat all work as PUBLIC. Private feature/x branches will not be squash-merged at release time.
Your code changes must meet the guidelines in CONTRIBUTING.md.
License: I confirm that my contribution is made under the terms of the Apache 2.0 license.

… that can automatically pull the correct vsix version name

github-actions · 2025-07-14T21:08:46Z

This pull request implements a feat or fix, so it must include a changelog entry (unless the fix is for an unreleased feature). Review the changelog guidelines.
- Note: beta or "experiment" features that have active users should announce fixes in the changelog.
- If this is not a feature or fix, use an appropriate type from the title guidelines. For example, telemetry-only changes should use the telemetry type.

Hweinstock

Nice, great start!

Hweinstock · 2025-07-15T16:34:19Z

package.json

@@ -28,6 +28,8 @@
        "test": "npm run test -w packages/ --if-present",
        "testWeb": "npm run testWeb -w packages/ --if-present",
        "testE2E": "npm run testE2E -w packages/ --if-present",
+        "setupE2E": "cd packages/amazonq && npm run package > ../../vsix.log 2>&1 && cd ../.. && node_modules/.bin/extest install-vsix --vsix_file amazon-q-vscode-$(node scripts/get-vsix-version.js).vsix",


Is it worth making this setupUI to match the testUI below?

Hweinstock · 2025-07-15T16:34:46Z

packages/amazonq/.vscodeignore

@@ -0,0 +1,32 @@
+# Source files


I don't think you meant to commit this file.

Hweinstock · 2025-07-15T16:35:35Z

packages/amazonq/package.json

@@ -2,7 +2,7 @@
    "name": "amazon-q-vscode",
    "displayName": "Amazon Q",
    "description": "The most capable generative AI-powered assistant for building, operating, and transforming software, with advanced capabilities for managing data and AI",
-    "version": "1.83.0-SNAPSHOT",


I believe this must stay as SNAPSHOT. Is there we reason we want to change it here?

Hweinstock · 2025-07-15T16:36:34Z

packages/amazonq/package.json

@@ -13,6 +13,7 @@
        "type": "git",
        "url": "https://github.com/aws/aws-toolkit-vscode"
    },
+    "homepage": "https://github.com/aws/aws-toolkit-vscode",


Is this change still necessary since we are no longer running the setup-test command?

Hweinstock · 2025-07-15T16:37:22Z

packages/amazonq/package.json

@@ -1322,26 +1323,40 @@
                    "fontCharacter": "\\f1de"
                }
            },
-            "aws-schemas-registry": {
+            "aws-sagemaker-code-editor": {


I think you'll have to rebase to remove these changes. Solved this issue with Laura, so she is an expert now :)

Hweinstock · 2025-07-15T16:38:03Z

packages/amazonq/test/e2e/amazonq/VET.test.ts

    // need this timeout because Amazon Q takes awhile to load
-    this.timeout(30000)
+    this.timeout(150000)


I think these changes are not meant to be part of this CR? They look like the ones Laura made in the other open one.

Hweinstock · 2025-07-15T16:39:30Z

packages/amazonq/vsix.log

I think this was accidentally committed.

Hweinstock · 2025-07-15T16:39:46Z

packages/toolkit/package.json

@@ -2,7 +2,7 @@
    "name": "aws-toolkit-vscode",
    "displayName": "AWS Toolkit",
    "description": "Including CodeCatalyst, Infrastructure Composer, and support for Lambda, S3, CloudWatch Logs, CloudFormation, and many other services.",
-    "version": "3.69.0-SNAPSHOT",
+    "version": "3.69.0-g8500b1e",


same as previous comment about keeping this as snapshot.

Hweinstock · 2025-07-15T16:41:18Z

scripts/get-vsix-version.js

out of curiosity, could we do this via CLI? I imagine we could use grep somehow. It might be somewhat complex, but I bet AI could write it.

The advantage of using a CLI command is that we avoid adding more files, and use already tested/maintained tools instead of creating our own.

Hweinstock · 2025-07-15T16:43:16Z

package.json

@@ -28,6 +28,8 @@
        "test": "npm run test -w packages/ --if-present",
        "testWeb": "npm run testWeb -w packages/ --if-present",
        "testE2E": "npm run testE2E -w packages/ --if-present",
+        "setupE2E": "cd packages/amazonq && npm run package > ../../vsix.log 2>&1 && cd ../.. && node_modules/.bin/extest install-vsix --vsix_file amazon-q-vscode-$(node scripts/get-vsix-version.js).vsix",


Is there a way we can do this without creating a temporary file (i.e. keeping the contents in memory).

If we do need the file, then we have to worry about conflicts with existing files as well as cleaning it up properly.

Good idea, I just implemented that in the new PR

Hweinstock · 2025-07-15T16:43:59Z

Looks like you may have to rebase anyway as I see some merge conflicts in the UI.

…guements, fixed PR conflicts

## Changes This is the initial commit for UI Tests using the framework vscode-extension-tester. It instantiates the VSCode instance which opens Amazon Q and goes through some basic flows to login. This is done using the [VSCode-Extension-Tester Framework](https://github.com/redhat-developer/vscode-extension-tester/wiki/). (Note: It is missing full authentication and stops working once Amazon Q needs to open the browser). **To run this test** 1. First compile the test files `npm run testCompile` 2. Then run the test setup `node_modules/.bin/extest setup-tests` 2. Then run the script (NOTE: this js file should be generated when you compile the test files) `node_modules/.bin/extest run-tests aws-toolkit-vscode/packages/amazonq/dist/test/e2e/amazonq/VET.test.js` For information why each dependency / webpack change is needed, please see [here](https://quip-amazon.com/lCuBAOGibHzm/UI-Tests-PR-Explanation). Demo Video: https://github.com/user-attachments/assets/f1c6e59f-d4e3-4ae0-a164-1da389ec1339 --- - Treat all work as PUBLIC. Private `feature/x` branches will not be squash-merged at release time. - Your code changes must meet the guidelines in [CONTRIBUTING.md](https://github.com/aws/aws-toolkit-vscode/blob/master/CONTRIBUTING.md#guidelines). - License: I confirm that my contribution is made under the terms of the Apache 2.0 license.

… that can automatically pull the correct vsix version name

…guements, fixed PR conflicts

Hweinstock

wooo, improvements!

Hweinstock · 2025-07-16T15:32:45Z

scripts/package.ts

looks like its trying to format this file. I think you'll have to checkout the version from master and commit without the preommit hook. Something like:

git checkout origin/master scripts/package.ts git commit -m 'fix: ....' -n

I believe the -n flag forces it to not run the linting on that commit.

Hweinstock · 2025-07-16T15:34:39Z

packages/amazonq/package.json

@@ -847,6 +865,24 @@
            }
        ],
        "keybindings": [
+            {


I think you can do the same thing as below to not include this file since it doesn't look like these are necessary changes.

package.json

Hweinstock · 2025-07-21T20:42:48Z

package.json

@@ -28,6 +28,10 @@
        "test": "npm run test -w packages/ --if-present",
        "testWeb": "npm run testWeb -w packages/ --if-present",
        "testE2E": "npm run testE2E -w packages/ --if-present",
+        "test:ui:prepare": "./node_modules/.bin/extest get-vscode -s ~/.vscode-test-resources -n && extest get-chromedriver -s ~/.vscode-test-resources -n",


its not entirely clear to me when each of these commands should be run and how they are distinguished. Do you mind updating the PR description to give some more context here?

Ex. What order are they run? Why are they split up? etc.

I suspect they are very good reasons for this, but looking at the commands its not obvious.

Will do, not sure how I missed that, thanks!

Hweinstock

woohoo!

Hweinstock · 2025-07-22T16:22:02Z

Looks like there might be a build issue causing the package to stall? Once thats resolved this LGTM.

…extension-tester (VET) (aws#7666) Users running our new E2E UI testing suite have to run a plethora of commands in order to setup and run our test suite each time their testing code may change or a new workspace in VSCode is instantiated. This [document](https://quip-amazon.com/KfISA389hGty/Setup-Instructions-for-Amazon-Q-VET-UI-Tests) contains rationale. These commands are exact and can be easy to forget and small errors in the CLI commands lead to headaches with the setup. Additionally, we run into a problem when running `npm run package` where each VSIX installation will have a unique hash and VSCode version attached to the name, and due to the lack of CLI args for renaming our VSIX package, we must be able to reference the path to this VSIX installation on anyones local devices. Notably, many Vscode-extension-tester (VET) projects use "setup-tests" to attach a VSCode instance and a Chromedriver instance which is crucial for the functionality of VET; however, this leads to build issues due to the inherent nature of VET's "setup-tests" packaging its own VSIX automatically. In normal circumstances, this would be fine; however, due to our unique packaging structure of the amazonq directory, we require a package.ts script to package our directory properly. My solution encompasses adding additional scripts to the package.json file in the root directory of the aws-toolkit-vscode repository. These scripts must be run in a specific order in order for them to work correctly and must **all by run in the root directory.** Namely these npm scripts are: ``` "test:ui:prepare": "./node_modules/.bin/extest get-vscode -s ~/.vscode-test-resources -n && extest get-chromedriver -s ~/.vscode-test-resources -n", ``` This script essentially does the bulk of the previous "setup-tests", it installs a local VSCode instance and clears the previous cache and installs the chromdriver into the VSCode instance. This is stored in a vscode-test-resources directory in ones local User directory. ``` "test:ui:install": "cd packages/amazonq && npm run package 2>&1 | grep -o 'VSIX Version: [^ ]*' | cut -d' ' -f3 | xargs -I{} bash -c 'cd ../../ && ./node_modules/.bin/extest install-vsix -f amazon-q-vscode-{}.vsix -e packages/amazonq/test/e2e/amazonq/resources -s ~/.vscode-test-resources'", ``` This script installs our VSIX by going into the amazonq package and running our npm run package. This eliminates the need for any additional scripts by allowing for a CLI/RegEx command to be utilized to find the specific hash of the VSIX in our terminal output and input it within the install-vsix command for extest. ``` "test:ui:run": "npm run testCompile && ./node_modules/.bin/extest run-tests -s ~/.vscode-test-resources -e packages/amazonq/test/e2e/amazonq/resources packages/amazonq/dist/test/e2e/amazonq/VET.test.js", ``` This script runs our VET.test.js script and does the testCompile npm script that has already been built into the repo. This is because when we run "npm run package", our compiled tests within the dist directory are deleted. We must run testCompile after this command in order to properly compile our tests. ``` "test:ui": "npm run test:ui:prepare && npm run test:ui:install && npm run test:ui:run", ``` This script combines all of our scripts into a chronological order. --- - Treat all work as PUBLIC. Private `feature/x` branches will not be squash-merged at release time. - Your code changes must meet the guidelines in [CONTRIBUTING.md](https://github.com/aws/aws-toolkit-vscode/blob/master/CONTRIBUTING.md#guidelines). - License: I confirm that my contribution is made under the terms of the Apache 2.0 license. --------- Co-authored-by: laura-codess <[email protected]>

Added functionality of node scripts with usage of vsix file scripting…

61bf08e

… that can automatically pull the correct vsix version name

surajrdy-aws requested a review from a team as a code owner July 14, 2025 21:08

Hweinstock reviewed Jul 15, 2025

View reviewed changes

surajrdy-aws and others added 9 commits July 15, 2025 15:40

Removed Javascript file for npm script and made it all through CLI ar…

1c7b855

…guements, fixed PR conflicts

Resolve .gitignore

1127842

Delete vsix.log

d14bcd5

Added functionality of node scripts with usage of vsix file scripting…

74c5770

… that can automatically pull the correct vsix version name

Removed Javascript file for npm script and made it all through CLI ar…

b4af3b4

…guements, fixed PR conflicts

Merge remote-tracking branch 'fork/featureUI' into featureUI

e63ac47

Fix git ignore

4b464ca

chore: rebase onto mainline

9f01f9e

surajrdy-aws force-pushed the featureUI branch from 4fc2495 to 9f01f9e Compare July 16, 2025 15:20

surajrdy-aws added 5 commits July 16, 2025 11:21

fix: undo changes to unrelated file

e902fc0

fix: removing unrelated files

1940c99

fix: removing unrelated files

0a24c10

fix: removing files that shouldnt be changed

7ac44f5

fix: accidently removed package.ts file

9d53f2a

Hweinstock reviewed Jul 16, 2025

View reviewed changes

surajrdy-aws added 4 commits July 16, 2025 11:44

fix: package.ts from master branch to fix linter issues

7865392

fix: rebase from master branch to fix package.json changes

85aad28

create new npm scripts which avoids the setup-tests process

0c0b550

fix: outdated npm scripts

5277991

Hweinstock reviewed Jul 21, 2025

View reviewed changes

Hweinstock approved these changes Jul 22, 2025

View reviewed changes

fix: packaging issues

9fa3eb7

surajrdy-aws changed the title ~~feat(amazonq): package.json npm scripts with js parsing script for vsix extension path~~ feat(amazonq): package.json npm scripts for build issues with vscode-extension-tester (VET) Jul 22, 2025

surajrdy-aws merged commit daf0938 into aws:feature/ui-e2e-tests Jul 22, 2025
22 checks passed

feat(amazonq): package.json npm scripts for build issues with vscode-extension-tester (VET) #7666

feat(amazonq): package.json npm scripts for build issues with vscode-extension-tester (VET) #7666

Uh oh!

Conversation

surajrdy-aws commented Jul 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Solution

This script combines all of our scripts into a chronological order.

Uh oh!

github-actions bot commented Jul 14, 2025

Uh oh!

Hweinstock left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Hweinstock commented Jul 15, 2025

Uh oh!

Hweinstock left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Hweinstock left a comment

Choose a reason for hiding this comment

Uh oh!

Hweinstock commented Jul 22, 2025

Uh oh!

Uh oh!

Uh oh!

surajrdy-aws commented Jul 14, 2025 •

edited

Loading