HTML codes in catalogues are not recognised

[Zack-83]

Description / Beschreibung

HTML entities within a catalogue need to be escaped. In this context, some special characters are replaced by their HTML codes, e.g. >, <, ".
Unfortunately, some other HTML codes are not recognised by the CI pipeline, among which   (insecable space) as well « and » (guillemets), which are all used for the French localisation.

Expected behaviour / Erwartetes Verhalten

I expect all HTML codes to be resolved when a job runs after any commit.

References / Verweise

See failed job: https://github.com/rdmorganiser/rdmo-catalog/actions/runs/18401171719/job/52430559969

[jochenklar]

It seems that those codes are not allowed in XML. Maybe the best way would be to use CDATA, but we need to implement that in RDMO. @MyPyDavid ideas?

[MyPyDavid]

I've asked around (GPT) and apparently there seems to be another notation for these HTML characters as well that is valid for the XML linting. So these decimal references (https://userpage.fu-berlin.de/mirjamk/htmlkurs/zeichen.html) can be written in the XML (and maybe we would need to slightly adapt the import in RDMO).
How does this sound?
I don't think that we would need CDATA for this.

This issue seems to be related to #393 , and this   is a little hack right?!

Infos:

Entity Name	Character	Decimal Reference
 	(non-breaking space)	 
«	« (left guillemet)	«
»	» (right guillemet)	»

In XML there are only five predefined named entities (& &, <<, >>, "", '')

[jochenklar]

yes, I found the same, but I guess it is much easier to just use CDATA since we just want to store literal text.

[MyPyDavid]

And what about the export??

Aspect	Decimal references	CDATA
rdmo-catalog XML validity	✅ strict valid XML	✅ valid XML
CI (xmllint, ET.parse)	✅	✅
Import complexity	✅ none (parser resolves)	⚠️ policy choice: literal vs decode
DB representation	✅ canonical Unicode	⚠️ either literal entities or Unicode
Export to XML	✅ normal XML	⚠️ likely needs CDATA again if literal
Export to JSON	✅	✅
HTML rendering	✅ straightforward	⚠️ literal shows   unless decoded/unsafe output
Works in attributes	✅ yes	❌ no

[MyPyDavid]

this would be something to add to the xml import

...
from html.entities import name2codepoint
...

HTML_ENTITY_PATTERN = re.compile(r'&([A-Za-z][A-Za-z0-9]+);')


def replace_html_named_entities(xml_content: str) -> str:
    """Replace named HTML entities with their decimal representations.

    This keeps XML parsing happy even if the content uses HTML entities
    that are not part of the five predefined XML entities.
    """

    def replace(match: re.Match[str]) -> str:
        name = match.group(1)
        codepoint = name2codepoint.get(name)
        if codepoint is not None:
            return f'&#{codepoint};'
        return match.group(0)

    return HTML_ENTITY_PATTERN.sub(replace, xml_content)

and

xml_content = file.read_text(encoding='utf-8')
xml_content = replace_html_named_entities(xml_content)
root = ET.fromstring(xml_content)

[jochenklar]

Again, I dont't see the benefit in respcet to use CDATA. We don't need to the texts as valid XML. People will edit the XML files, and add HTML codes again.

[MyPyDavid]

CDATA could contain anything '<![CDATA[<script>alert(\'x\')</script>]]>' ?!

How should it look like in the end then?

<help lang="en"><![CDATA[&lt;h3&gt;Organisation...&lt;/h3&gt;]]></help>

or this

<help lang="en"><![CDATA[<h3>Organisation of data into data sets</h3> ...]]></help>

or just partially like this?

<help lang="en">
  Some normal XML text (entities work here: &lt; &amp; etc.)
  <![CDATA[
    Raw text here (no entity expansion): &nbsp; &laquo; <b>tags</b>
  ]]>
  More normal text again.
</help>

[jochenklar]

I don't see the problem, the second option. Then we don't need to do anything to the database content and just put it into the XML is it is. Script tags are a thing for validation anyway.

[Zack-83]

Thanks for your suggestion @jochenklar and @MyPyDavid ! I will try it at the next possibility.

[jochenklar]

Hi @Zack-83 lets keep this open since it is something we need to fix in the RDMO export. You can, however, test if either the CDATA or the XML Entity aproach works already.

[MyPyDavid]

to be continued then..
Do you not mean the "HTML Entity approach" ?? Like so:

<help lang="en">
  Some normal XML text (entities work here: &lt; &amp; etc.)
  Raw text here: &#160; &#171; &lt;b&gt;tags&lt;/b&gt; &#187;
  More normal text again.
</help>