Coherent story for HTML-setting methods

[noamr]

[Note: OP edited after some discussion]

What is the issue with the HTML Standard?

With the introduction of setHTML{Unsafe}, we have multiple ways of applying an HTML string into an existing document:

• setHTML
• setHTMLUnsafe
• innerHTML and outerHTML setters
• createContextualFragment
• insertAdjacentHTML
• Detached document streaming (document.write() into an inactive document and inserting the root node the active one)

With #2142 and #11542, we are about to introduce even more methods that insert HTML asynchronously using a stream.
Some of these techniques support different levels of sanitation and slightly different behaviors.

So this is a good time to have a wider overview of these methods, and come up with a consistent structure for their API.

Specifically, the following variants apply when inserting HTML:

• Does the HTML have to go through the sanitizer ("safe")
• Does the HTML replace the whole contents of the element or just parts
• Is the HTML passed as a string or as a stream (as per A way to stream content into an element #2142)
• Is the HTML parsed as a whole, or containing interleaved patches (as per Out of order HTML streaming ("patching") #11542)
• How does script execution work
• How this works with TrustedTypes.

API shape

-Treat unsafe/safe stream/set and replaceChildren/replaceWith/append/prepend/before/after as permutations that are best exposed as different methods (226 = 20 methods). We can tweak this of course

dictionary UnsafeHTMLSetterOptions  {
  (Sanitizer or SanitizerConfig)? sanitizer = null;
  boolean? runScripts = false;
};

dictionary SafeHTMLSetterOptions {
  // The user of this dictionary must ensure a sanitizer is provided.
  (Sanitizer or SanitizerConfig) sanitizer;
};

[Exposed=Window]
interface ParentNode {
  void setHTML((DOMString or TrustedHTML) html, SafeHTMLSetterOptions options);
  void setHTMLUnsafe((DOMString or TrustedHTML) html, optional UnsafeHTMLSetterOptions options = {});
  void beforeHTML((DOMString or TrustedHTML) html, SafeHTMLSetterOptions options);
  void beforeHTMLUnsafe((DOMString or TrustedHTML) html, optional UnsafeHTMLSetterOptions options = {});
  void afterHTML((DOMString or TrustedHTML) html, SafeHTMLSetterOptions options);
  void afterHTMLUnsafe((DOMString or TrustedHTML) html, optional UnsafeHTMLSetterOptions options = {});
  void appendHTML((DOMString or TrustedHTML) html, SafeHTMLSetterOptions options);
  void appendHTMLUnsafe((DOMString or TrustedHTML) html, optional UnsafeHTMLSetterOptions options = {});
  void prependHTML((DOMString or TrustedHTML) html, SafeHTMLSetterOptions options);
  void prependHTMLUnsafe((DOMString or TrustedHTML) html, optional UnsafeHTMLSetterOptions options = {});
  void replaceWithHTML((DOMString or TrustedHTML) html, SafeHTMLSetterOptions options);
  void replaceWithHTMLUnsafe((DOMString or TrustedHTML) html, optional UnsafeHTMLSetterOptions options = {});
  WritableStream streamHTML(SafeHTMLSetterOptions options);
  WritableStream streamHTMLUnsafe(optional UnsafeHTMLSetterOptions options = {});
  WritableStream streamBeforeHTML(SafeHTMLSetterOptions options);
  WritableStream streamBeforeHTMLUnsafe(optional UnsafeHTMLSetterOptions options = {});
  WritableStream streamAfterHTML(SafeHTMLSetterOptions options);
  WritableStream streamAfterHTMLUnsafe(optional UnsafeHTMLSetterOptions options = {});
  WritableStream streamAppendHTML(SafeHTMLSetterOptions options);
  WritableStream streamAppendHTMLUnsafe(optional UnsafeHTMLSetterOptions options = {});
  WritableStream streamPrependHTML(SafeHTMLSetterOptions options);
  WritableStream streamPrependHTMLUnsafe(optional UnsafeHTMLSetterOptions options = {});
  WritableStream streamReplaceWithHTML(SafeHTMLSetterOptions options);
  WritableStream streamReplaceWithHTMLUnsafe(optional UnsafeHTMLSetterOptions options = {});
};

Alternative: 12 methods, with insertion mode (taken from #10122)

enum HTMLInsertionPoint { "before", "after", "start", "end" };

// Hypothetical interface where these methods would live,
// for example, on Element or ShadowRoot.
[Exposed=Window]
interface ParentNode {

  void setHTML((DOMString or TrustedHTML) html, SafeHTMLSetterOptions options);
  void setHTMLUnsafe((DOMString or TrustedHTML) html, optional UnsafeHTMLSetterOptions options = {});
  void insertHTML((DOMString or TrustedHTML) html, HTMLInsertionPoint insertionPoint, SafeHTMLSetterOptions options);
  void insertHTMLUnsafe((DOMString or TrustedHTML) html, HTMLInsertionPoint insertionPoint, optional UnsafeHTMLSetterOptions options = {});
  void replaceWithHTML((DOMString or TrustedHTML) html, SafeHTMLSetterOptions options);
  void replaceWithHTMLUnsafe((DOMString or TrustedHTML) html, optional UnsafeHTMLSetterOptions options = {});
  WritableStream streamHTML(SafeHTMLSetterOptions options);
  WritableStream streamHTMLUnsafe(optional UnsafeHTMLSetterOptions options = {});
  WritableStream streamInsertHTML(HTMLInsertionPoint insertionPoint, SafeHTMLSetterOptions options);
  WritableStream streamInsertHTMLUnsafe(HTMLInsertionPoint insertionPoint, optional UnsafeHTMLSetterOptions options = {});
  WritableStream streamReplaceWithHTML(SafeHTMLSetterOptions options);
  WritableStream streamReplaceWithHTMLUnsafe(optional UnsafeHTMLSetterOptions options = {});
}};

Alternative: pass ReadableStream instead of returning a WritableStream:

[Exposed=Window]
interface ParentNode {
  Promise<void> streamHTML((ReadableStream or TrustedReadableStream), SafeHTMLSetterOptions options);
  // ...
};

This is simpler from a trusted types perspective, as the stream has to be trusted as a whole rather than the chunks.
OTOH the API feels a bit less "streamy" than returning a Writable.

Other notes

For the interleaved case, as well as for trusted types, suggesting to do that via transform streams:

• A TransformStream that resolves a raw HTML streams to patches as per Out of order HTML streaming ("patching") #11542
• A TransformStream that allows a trusted types to add a control point in the middle of a stream, as per Streaming support w3c/trusted-types#594

The aforementioned issues include more details about these, but they are designed in a way that shouldn't add more complexity to the normal case.

[zcorpan]

I like this!

There's another axis, which is whether script elements run when inserted. See #10090 and demo https://software.hixie.ch/utilities/js/live-dom-viewer/saved/14067

Maybe the UnsafeHTMLSetterOptions can have boolean runScripts = false; -- or we could decide that this shouldn't be possible (with new methods at least).

[noamr]

I like this!

There's another axis, which is whether script elements run when inserted. See #10090 and demo https://software.hixie.ch/utilities/js/live-dom-viewer/saved/14067

Maybe the UnsafeHTMLSetterOptions can have boolean runScripts = false; -- or we could decide that this shouldn't be possible (with new methods at least).

Good call! added to the OP. I think we can have a design that handles all of these options and then reason about each of these things individually.

[annevk]

insertAdjacentHTML is only broken in the sense of its specification, which on its own is not a good reason of course. #10122 discusses some reasons for why we might want a replacement. Using positional arguments could be reasonable, but is quite a bit more verbose. We also don't really have any mutation operations that cleanly map to that so you'd end up with multiple mutation records as a result. That might be okay.

[noamr]

insertAdjacentHTML is only broken in the sense of its specification, which on its own is not a good reason of course. #10122 discusses some reasons for why we might want a replacement. Using positional arguments could be reasonable, but is quite a bit more verbose.

Yea it's "splice" rather than "insert", which is perhaps more verbose but is a common lower level primitive for mutations?

It's quite easy to implement the #10122 semantics on top of the proposal here, while keeping the consistency with passing the rest of the options (sanitizer, streaming, allow-scripts), or add them as sugar into the platform (e.g. position instead of before and after).

[noamr]

... the other option for insertAdjacentHTML is to have insertHTML and insertHTMLUnsafe methods (so 6 instead of 4) that have the position argument instead of before + after and doesn't remove any element... but then if we want to add streaming to that we end up with 8 methods altogether which starts to feel like a lot?

4 HTML-setting methods with splicing feels like a good balance to me with leaving it open to adding insert* variants in the future.

[evilpie]

Do we really need the explicit runScripts? You can use a lenient Sanitizer instance together with removeUnsafe.

[annevk]

@evilpie runScripts is about whether or not scripts get executed post-insertion. They currently do not.

[dmsnell]

Over in WordPress we’ve been stewing on this issue for some time now, ever since we built a pure-PHP HTML5 parser.

We are wanting to provide a convenient API for server-side code to understand and manipulate HTML, ideally to do things like stitch in HTML fragments from one document into another.

For us, this means we are fundamentally less concerned with script execution and more concerned with escaping outside of the context in which an edit is made. For instance, when inserting into the DOM it’s impossible to escape from the parent. It’s possible to construct invalid DOM trees which cannot be re-serialized into HTML. One example of this is creating a P element as a child inside of another P element.

But in the HTML world we have this conundrum where if we allow the insertion of such a <p> inside another open P element, we have allowed server-side code to create HTML which will parse differently once it reaches the browsers. This is why this issue has been delayed and held up for so long for us, because the fragment parsing algorithm isn’t relevant for this case, but it seems like the issue at hand with streaming HTML is identical or close enough to be relevant.

The fundamental question I would love to see specified and standardized is what happens when something appears which should breach a context. Maybe here it’s still left to the DOM and unrepresentable DOM trees are the answer, but I see three possible ways to handle this on the server:

• Outright reject updates which would cause any open element to close, if that open element is a parent or ancestor of the node in which inner HTML was set or changed.
• Remove elements which would trigger modifications to the open elements. This could lead to extreme behavior in edge cases and distorts updates.
• Wrap the offending elements and reconstruct them via DOM operations once the page loads. E.g. if setting inner HTML to <h2> when already inside an H1 element, wrap it as <invalid-element data-tag-name=h2>.

This is pretty relevant for any system that brings together separate HTML snippets or templates, and I would imagine even those which heavily modify the page via JS. It’s nice that the DOM allows for unrepresentable constructions, but it would be ideal to be able to agree on the server and the client on what the rendered HTML will do.

Some API which provides for a kind of isolation of a parent element for inner markup changes would be extremely helpful. “No matter what HTML is set to the inside of this element, it will not bleed into the surrounding page.” This is helpful for proper understanding, it’s helpful for security issues, and it’s helpful for the resilience of sites which deal with user-supplied content.

[noamr]

Over in WordPress we’ve been stewing on this issue for some time now, ever since we built a pure-PHP HTML5 parser.

We are wanting to provide a convenient API for server-side code to understand and manipulate HTML, ideally to do things like stitch in HTML fragments from one document into another.

For us, this means we are fundamentally less concerned with script execution and more concerned with escaping outside of the context in which an edit is made. For instance, when inserting into the DOM it’s impossible to escape from the parent. It’s possible to construct invalid DOM trees which cannot be re-serialized into HTML. One example of this is creating a P element as a child inside of another P element.

But in the HTML world we have this conundrum where if we allow the insertion of such a <p> inside another open P element, we have allowed server-side code to create HTML which will parse differently once it reaches the browsers. This is why this issue has been delayed and held up for so long for us, because the fragment parsing algorithm isn’t relevant for this case, but it seems like the issue at hand with streaming HTML is identical or close enough to be relevant.

The fundamental question I would love to see specified and standardized is what happens when something appears which should breach a context. Maybe here it’s still left to the DOM and unrepresentable DOM trees are the answer, but I see three possible ways to handle this on the server:

• Outright reject updates which would cause any open element to close, if that open element is a parent or ancestor of the node in which inner HTML was set or changed.
• Remove elements which would trigger modifications to the open elements. This could lead to extreme behavior in edge cases and distorts updates.
• Wrap the offending elements and reconstruct them via DOM operations once the page loads. E.g. if setting inner HTML to <h2> when already inside an H1 element, wrap it as <invalid-element data-tag-name=h2>.

This is pretty relevant for any system that brings together separate HTML snippets or templates, and I would imagine even those which heavily modify the page via JS. It’s nice that the DOM allows for unrepresentable constructions, but it would be ideal to be able to agree on the server and the client on what the rendered HTML will do.

Some API which provides for a kind of isolation of a parent element for inner markup changes would be extremely helpful. “No matter what HTML is set to the inside of this element, it will not bleed into the surrounding page.” This is helpful for proper understanding, it’s helpful for security issues, and it’s helpful for the resilience of sites which deal with user-supplied content.

Over in WordPress we’ve been stewing on this issue for some time now, ever since we built a pure-PHP HTML5 parser.

Thanks for sharing this in detail!

But in the HTML world we have this conundrum where if we allow the insertion of such a <p> inside another open P element, we have allowed server-side code to create HTML which will parse differently once it reaches the browsers. This is why this issue has been delayed and held up for so long for us, because the fragment parsing algorithm isn’t relevant for this case, but it seems like the issue at hand with streaming HTML is identical or close enough to be relevant.

Perhaps this is what I don't understand. Why is the fragment parsing algorithm isn't relevant here? Sounds like it's exactly tackling the use case you've described, of parsing a piece of HTML in the context of where it is applied to?

[dmsnell]

Perhaps this is what I don't understand. Why is the fragment parsing algorithm isn't relevant here? Sounds like it's exactly tackling the use case you've described, of parsing a piece of HTML in the context of where it is applied to?

@noamr I think the following should illustrate the problem. Live DOM

<div>
   <h1>This is a page</h1>
   <p id=p1>And this is a paragraph.</p>
</div>
<script>
   document.getElementById( 'p1' ).innerHTML = 'And <p>another paragraph</p> is here now.';
</script>

In this setup we’re calling the fragment parsing algorithm, if I understand it properly, which runs the steps on the inner HTML and creates the DOM nodes accordingly. It creates an invalid DOM with nested paragraphs, which is fine. That works in the browser, but it also means that this edit operation is not possible via HTML. There is no way to represent it.

Making this same edit within the HTML domain would lead to the premature closing of the containing P element, were it allowed.

---

The problem extends beyond basic nesting rules, however. Because the fragment parsing algorithms creates a new Document and stack of open elements, there are parsing rules which depend on the stack of open elements and will behave differently in the fragment parsing context than in the normal context. Here is one where the containing H1 element is not present in the fragment context. Live DOM

<div>
   <h1>This is a <span id=p1>page</span></h1>
   <p>And this is a paragraph.</p>
</div>
<script>
   document.getElementById( 'p1' ).innerHTML = 'And <h2>another paragraph</h2> is here now.';
</script>

And in this case the DOM allows nesting an H2 inside an H1 which would be interpreted differently were the browser to see <h1>This is a <span id=p1>And <h2>another paragraph</h2> is here now.</h1>

In our experimentation we have found cases where the missing elements on the stack of open elements is relevant for tokens which impact the insertion mode, affect integration points and name spacing, and a number of other state values.

In WordPress/wordpress-develop#7777 we started looking at ways to extend the fragment parser to clone the stack of open elements and provide guards which would detect if the parsing algorithm attempted to close any open element. I have explored a weaker approach in a builder class which simply examines whether there are any remaining tokens in the new inner HTML after the outer-most stack is back to the depth it had when the parsing started.

In all of the work I’ve explored, the one simple rule that I think is relevant to isolate changes is to somehow freeze the stack of open elements, run the parse algorithm in place, and if there are changes or pops to the stack of open elements above the point of setting inner HTML, it’s a violation. Probably other things need to be examined as well, such as whether the parse changed the insertion mode, updated the form pointer, changed the document encoding, etc…

---

Thanks for asking. I hope this clarifies things instead of making them less clear. The entire problem remains that the DOM operations allow creating DOM trees which cannot be represented in HTML. Even the latest work to bring in DOM\HtmlDocument in PHP leave us without reasonable tools to mix and match HTML from different sources because updates through its internal DOM don’t round-trip (we can’t send it’s internal DOM to a browser to we have to serialize, which creates these situations where the browser reads a different DOM than we had on the server).

It seems like streaming into the DOM via HTML is going to present similar quandaries. Suppose we load in the HTML for a comment on a blog. Should that comment be allowed to interrupt the flow of the page? How can we ensure that it remains isolated and inert?

[annevk]

I think for streaming we should guarantee you stay within a parent, although runScripts might run afoul of that. But generally enforcing idempotency would require much more invasive changes and I don't think is easily doable.

[mozfreddyb]

If you fully want to match what insertAdjacentHTMLdoes, why not use the same terms? Is there a universal understanding that it's a bad pattern that shouldn't be continued that I am not aware if?

For consistency, I would have considered a position key with the afterend, beforeend, afterbegin, beforebegin terms.

Nevermind. I found #10122.