Transformer: Implement self-hosting of AMP scripts

[westonruter]

Feature description

Depends on ampproject/amphtml#25873.

Self-hosting will need to rely on fetching the required scripts from the AMP CDN to then serve locally via new rewrite rules that render out the cached responses. The AMP plugin would serve essentially as a proxy.

Some key benefits to doing this include:

• Fewer HTTP connections to obtain the JS critical for the page to load.
• Elimination of dependency on CDN which can be a seen as a privacy concern (nevertheless, see Add support for release channel(s) and crossorigin=anonymous scripts and links #4403).

---

Do not alter or remove anything below. The following sections will be managed by moderators only.

Acceptance criteria

Implementation brief

QA testing instructions

Demo

Changelog entry

[westonruter]

The AMP core proposal for implementing this is to have a document add a runtime-host meta tag like this:

<meta name="runtime-host" content="https://example.com/optional/path/prefix">

And then it would be allowed to reference AMP component scripts like:

<script async src="https://example.com/optional/path/prefix/amp/v0.js"></script>
<script async custom-element="amp-bind" src="https://example.com/optional/path/prefix/amp/v0/amp-bind-0.1.js"></script>  

So, in a WordPress context how would we do this?

One idea I had was to pull down the files and store the contents in the DB, and then serve them dynamically from WordPress. Nevertheless, this would face a key challenge in that many web servers that Wordpress runs on are configured to route .js requests straight to the filesystem and bypass PHP entirely routing. To work around this issue for PWA, we ended up using a different file extension for the PHP-served service worker endpoint: /wp.serviceworker. See GoogleChromeLabs/pwa-wp#287 (comment) and GoogleChromeLabs/pwa-wp#289 (review).

However, the AMP core proposal doesn't allow for varying file extensions like this. So this doesn't seem to be feasible, as it stands.

The other solution to implement this would be to actually write the JS files out to the filesystem inside of wp-content. Could we create a wp-content/uploads/amp-assets which is configured to be the base URL for all self-hosted AMP assets? Would this be a best practice? I suppose optimization plugins often concatenate and minify scripts and dump them inside of wp-content/uploads as well. Any drawbacks to this approach?

One thing that comes to mind is that if we serve from the filesystem, we'll need to make sure that we put each day's AMP CDN assets into a separate directory, if a new version of AMP has been released. This will ensure that the web server can serve the scripts with far-future expires for optimal performance.

The process I think we'd follow is this:

1. AMP plugin is initially activated and no AMP assets have been downloaded, so the runtime-host meta tag is not added to the page and no self-hosting is done.
2. The plugin registers a daily WP Cron task to fetch the scripts (and styles, etc) from the AMP CDN.
3. When the WP Cron task runs:
   1. It obtains the most recent stable version and stores it in $AMP_VERSION.
   2. It checks to see $AMP_VERSION has been fetched yet. If it has, it quits. Otherwise:
   3. It downloads all the assets and puts them into wp-content/uploads/amp-assets/$AMP_VERSION.
   4. The $AMP_VERSION then gets stored in a DB option to indicate the version that is ready for self-hosting.
   5. It checks for any wp-content/uploads/amp-assets/* directories other than the current $AMP_VERSION and previous X versions and deletes them. (Keeping old versions around for a bit will help guard against the assets from 404ing while waiting for clients to update.)
4. Thereafter, whenever an AMP page is being served, the $AMP_VERSION should be examined from the DB and if present, rewrite all registered AMP CDN assets to point to wp-content/uploads/amp-assets/$AMP_VERSION as well as serve the required meta tag.

Any concerns with this approach?

cc @schlessera @pierlon @johnwatkins0

[westonruter]

We'll make this an opt-in transformer. The concern is that a majority of users will be negatively impacted by serving assets from the origin rather than the AMP CDN. So it would be off by default.

[jonoalderson]

"The other solution to implement this would be to actually write the JS files out to the filesystem inside of wp-content. Could we create a wp-content/uploads/amp-assets which is configured to be the base URL for all self-hosted AMP assets? Would this be a best practice? I suppose optimization plugins often concatenate and minify scripts and dump them inside of wp-content/uploads as well. Any drawbacks to this approach?"

Sounds good. Only risks I've come across are folder creation permissions, and, collision with existing folder names.

"One thing that comes to mind is that if we serve from the filesystem, we'll need to make sure that we put each day's AMP CDN assets into a separate directory, if a new version of AMP has been released. This will ensure that the web server can serve the scripts with far-future expires for optimal performance."

Can't we just serve the current/latest file with a query-busting param, based on the lastmod timestamp? E.g., v0.js?timestamp=1234567

[westonruter]

Can't we just serve the current/latest file with a query-busting param, based on the lastmod timestamp? E.g., v0.js?timestamp=1234567

Good question. I've requested that: ampproject/amphtml#25873 (comment).