A few questions on the implementation (comparison with Lullabot/ddev-playwright)

[deviantintegral]

Hello, great to see this!

I've also been working on this on and off over the past month. So far it's all custom code in a single site. Here's a few notes and challenges I ran into, and I'm curious if you've hit them yet. I don't think you have because I don't see workarounds in the code.

1. For writing reports, how are permissions working? While I posted How to get the $uid / $gid variables in custom docker compose services? ddev/ddev#4733 about ZAP, we had exactly the same issue with Playwright.
2. Have you run into this bug yet with Firefox? This plus the above led me to stacking Playwright into the web container, instead of a separate service: [BUG] Running Playwright on Firefox inside a Docker container always hangs forever microsoft/playwright#16491
3. Some tools like https://github.com/chanzuckerberg/axe-storybook-testing also have a Playwright dependency. Any thoughts on how to run them inside the playwright container without having to double-download browsers?

At a high level, what we've ended up with is:

1. A separate yarn project for playwright in the project repo in test/playwright.
2. An additional dockerfile for the web service that is enabled via a ddev playwright command that copies it in. As well, a pre-start host hook that updates it if it already exists. Unfortunately, symlinks and hardlinks aren't working.
3. Adding a VNC server and xvfb for headed operations.
4. Lots of use of of build caches to speed up development and local rebuilds:

# Debian images by default disable apt caching, so turn it on until we finish
# the build.
RUN mv /etc/apt/apt.conf.d/docker-clean /etc/apt/docker-clean-disabled

# Install Playwright browsers and system dependencies.
#
# We want to install whatever browsers are used by the currently locked version
# of playwright in test/playwright. What this does is:
#
# 1. Creates a copy of the playwright directory in the image. We need this
#    because Playwright keeps a hidden .links/<hash> file with the path of
#    what browsers are installed by what apps. Without it, Playwright will re-
#    download browsers even though they are in the directory.
# 2. $username doesn't work in --mount... lines as it's a shell variable set by
#    ddev. Instead, we mount the .cache cache into /dot-cache, and symlink it
#    in.
# 3. Then we remove the symlink, and copy the cache directory into ~/.cache so
#    it is persisted in the image.
RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
  --mount=type=cache,target=/var/lib/apt,sharing=locked \
  --mount=type=cache,target=/dot-cache,sharing=locked \
  --mount=type=bind,source=./test/playwright,target=/playwright \
  sudo -u $username mkdir -p /var/www/html/test \
  && cp -a /playwright /var/www/html/test \
  && chown -R $username:$username /var/www/html/test/playwright \
  && chown -R $username:$username /dot-cache \
  && sudo -u $username ln -s /dot-cache /home/$username/.cache \
  && cd /var/www/html/test/playwright \
  && sudo -u $username yarn playwright install-deps \
  && sudo -u $username yarn playwright install \
  && rm -rf /var/www/html/test/playwright \
  && rm /home/$username/.cache \
  && sudo -u $username mkdir -p /home/$username/.cache \
  && sudo -u $username cp -a /dot-cache/* /home/$username/.cache/

# Install a window manager and VNC server.
RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
  --mount=type=cache,target=/var/lib/apt,sharing=locked \
  sudo apt-get install -y fluxbox \
    xauth \
    x11vnc

# We're done with apt so disable caching again for the final image.
RUN mv /etc/apt/docker-clean-disabled /etc/apt/apt.conf.d/docker-clean

# Install novnc to access the GUI over a browser.
ENV NOVNC_VERSION="1.4.0" \
    WEBSOCKIFY_VERSION="0.11.0"
RUN  wget -nv -O noVNC.zip \
       "https://github.com/novnc/noVNC/archive/refs/tags/v${NOVNC_VERSION}.zip" \
  && unzip -x noVNC.zip \
  && mkdir -p /opt/bin \
  && mv noVNC-${NOVNC_VERSION} /opt/bin/noVNC \
  && cp /opt/bin/noVNC/vnc.html /opt/bin/noVNC/index.html \
  && rm noVNC.zip \
  && wget -nv -O websockify.zip \
      "https://github.com/novnc/websockify/archive/refs/tags/v${WEBSOCKIFY_VERSION}.zip" \
  && unzip -x websockify.zip \
  && rm websockify.zip \
  && rm -rf websockify-${WEBSOCKIFY_VERSION}/tests \
  && mv websockify-${WEBSOCKIFY_VERSION} /opt/bin/noVNC/utils/websockify

# Copy start scripts for the x server, x11vnc, and novnc.
COPY .ddev/web-build/start-*.sh /usr/local/bin
RUN chmod +rx /usr/local/bin/start-*.sh

# novnc requires a password, so we set it to secret matching Selenium's images.
RUN sudo -u $username mkdir -p /home/$username/.vnc
RUN sudo -u $username x11vnc -storepasswd secret /home/$username/.vnc/passwd

[julienloizelet]

Hi @deviantintegral,
Thanks for your message.

I'm not sure if I can help you much, but I'll try to tell you how I use ddev with playwright. Maybe it will give you some clues.

First, to answer your questions, here is what I can say:

1. As you can see in the test.bats file of this repo, I use a command like ddev exec -s playwright yarn --cwd /var/www/html/yarn test --json --outputFile=./.test-results.json "__tests__/1-simple-test.js" to run test. As a result, the .test-results.json has the following permissions: -rw-r--r-- 1 root root. Since there are read rights, this has never been a problem for me. But I am not sure if you were talking about those kinds of reports.

2. Until now, I only test with Chromium. I just tried Firefox by adding playwright-firefox in the package.json file and modifying jest-playwright.config.js too by adding browsers: ["firefox"],devices: ["Desktop Firefox"]. I also had to run some ddev exec -s playwright apt-get install libxtst6 to make it work. Tests ran successfully, but this maybe too simple.

3. Sorry, but I have no idea for this :)

Then, if you are curious, and I have time, here is how I use playwright :

I only need javascript tools like playwright, jest, etc. to make end-to-end tests for some PHP development (mainly Magento 2 modules and Wordpress plugin but others PHP project too). And I wanted to be able to run test from my host (without headless) and from GitHub actions (headless)

I ended up with the following set up :

• All yarn and js files are in a tests/end-to-end folder in the source files of the tested plugin/module (maybe you can look at some WordPress plugin example)
• In the docker-compose.playwright.yaml file, you can see that the playwright service depends on the web service and that there is a volume : ../:/var/www/html
• To initialize tests (mainly yarn install), I run a script like this one
• And to run tests, I run a script like this one. This is a quite complicated script file and I don't even remember why I ended up with this...but it works for me :) . The interesting part is probably  ddev exec -s playwright xvfb-run --auto-servernum -- yarn --cwd ${YARN_PATH} cross-env for a "GitHub action" case.

If you want to see all the command for a GitHub action, you can take a look here.

Hope it could help.

Thanks

[deviantintegral]

I was curious why you needed playwright-firefox. I think that's because you're using jest for tests instead of playwright/test. Makes sense, and matches the note at https://www.npmjs.com/package/playwright-firefox. I'm really surprised you didn't run into Firefox hangs - but perhaps it's now been fixed upstream.

I'm going to post a repo with the setup we have, and then come back and do a comparison.

[deviantintegral]

Here's the repo: https://github.com/deviantintegral/drupal-playwright

[julienloizelet]

Hi @deviantintegral, thanks for the repo. Very complete !

For information, I just tested a full test suite with Firefox and some WordPress plugin I maintain.

Despite errors on some tests (which could be corrected, I guess, by adding some of the usual workarounds), some tests passed at 100% like this one : https://github.com/julienloizelet/cs-wordpress-bouncer/actions/runs/4508204588/jobs/7936683789

As you said, maybe the bug is fixed upstream.

[deviantintegral]

To follow up, I've been working on converting the example repo to a ddev addon, which I've done here: https://github.com/deviantintegral/ddev-playwright. We've been happy with the "all in one container" approach for a few months now, so I think we'll be sticking with it. Thanks for your feedback!

[jcandan]

Really cool conversation.

The projects seem to both have come a long way since. Any chance for a round 2 discussion on the different approaches?

For instance, Lullabot's Similar Tools notes the following:

julienloizelet/ddev-playwright [. . .] uses Playwright containers built by Microsoft [ . . . which] do not ship with any sort of remote access to the Playwright UI. This repository includes KasmVNC to run tests in headed mode or to generate code.

However, it seems like julienloizelet/ddev-playwright now also includes KasmVNC and supports headed mode.

What other differences now exist? What might be the current pros/cons of each?

[deviantintegral]

I'll get the README updated next week, thanks for pointing that out!

[julienloizelet]

Hi @jcandan ,
thanks for your message.

I won't be able to answer because I'm only using the https://github.com/julienloizelet/ddev-playwright one :)

But I can at least invite you to read this closed issue where it appears that it's obviously simpler to interact between the web container and playwright when playwright is inside the web container.

Basically, I suppose the pros/cons are similar to the pros/cons of the monolithic approach versus the micro-services approach:

• With the monolithic approach, it's easier to get services to interact with each other, but you can end up with a growing mess.
• With the microservices approach, everything is well separated and easy to clean up, but it can be a nightmare to get services to interact with each other.

DDEV itself does not choose between the two approaches:

• add-ons are clearly a good way of adding/removing distinct services
• but the web container contains a mixture of services: PHP, web server, etc...

Playwright may be designed to test code running on the web server, so it makes sense to have it in the web container.
I use it in a separate container because it suits me and I started with this approach, and also because I don't want to change this approach on the many projects I maintain...

I suppose the choice is yours and you can test both to see which one best suits your needs.

Thanks!

[deviantintegral]

@julienloizelet it sounds like core is leaning towards switching tests to Playwright, and there's also discussion about optimizing setup steps. Given Drupal core also uses ddev for development, I wonder if we could get feedback on which method core maintainers would prefer. While the monolithic approach has served us well, if core would rather the separate container approach and there's a good solution for "run this drush or php script" that would be fine by me.

https://www.drupal.org/project/drupal/issues/3467492

[jcandan]

Wow. Thank you for that unbiased answer. That’s cool.

I’ll give them both a shot. I’d imagine that, being a DDEV addon, most of the complexities you mention are handled for simple cases. Any insight or example solutions into complex integrations you warn of?