Align DHd XML with dh-abstracts XML submission requirements

[scottbot]

@PatrickHelling / @nubuker has uploaded DHd abstracts to github (https://github.com/DHd-Verband), using dhconvalidator, but they don't quite align with our XML schema for upload.

Since I moved jobs I'm no longer @mdlincoln's supervisor and can't put him to the task, and I certainly have no control over work Patrick or Harald does, but perhaps we can work together to align the formats? Feels like this could be an easy win, but I don't know enough XML/XSLT to get the job done. If nobody here has time, perhaps we can loop in another volunteer? Once the formats are aligned, I can do all the import, unique identity reconciliation for authors/institutions already existing in our database, etc.

cc @quinnanya, @nickoal, @ariddell, & @christofs just to keep everyone in the loop

Linking to other GH issue for additional context: #540 (XML format discussion) & #549 (XML samples documenting data submission requirements)

[mdlincoln]

I'd be eager to help get things aligned (albeit with the very limited time I have for non-DAMS migration work right now at CMU 😱)

@PatrickHelling @nubuker take a look at the draft XSD files in #549 - the most important addition compared to the dhconvalidator schema is the more strict structure required for expressing author institutions/affiliations.

[PatrickHelling]

@scottbot / @mdlincoln thank you very much for mentioning this. I wanted to get in touch with you myself after our workshop at the vDHd conference in March (https://vdhd2021.hypotheses.org/): indeed, shortly after we published 917 conference papers of all DHd conferences (2014-2020) on zenodo as individual publications (https://zenodo.org/communities/dhd/?page=1&size=20). In addition, we have now published the XML-files, PDF-files, metadata-files and publication lists as you have seen on GH.

It would be really great if we could add the abstracts to your Index of Digital Humanities Conferences. I’ve had a look at #549 and on that basis we are now working on the harmonization of the XML-files. However, for two conferences (2014 and 2015) there are no XML-files. Maybe a revisioned version of the metadata-files (as seen in the GH-repositories) could help one index these abstracts as well?

[PatrickHelling]

@scottbot / @mdlincoln it took a little while because of other tasks and issues, but we worked on some scripts to transform the abstracts of the DHd-conferences into the required XML-structures. I added an example file (Beispiel_Steyer_AbstractEnhancement.xml) to the following repository: transform-DHd-IoDHC maybe you can have a look at it? I am no expert in XML/XSLT but I guess it shoud meet your requirements regarding the Index of Digital Humanities Conferences. If so, please let me know so that we can transform all data we have collected and make it available to you.

[scottbot]

Thanks @PatrickHelling! @mdlincoln at your leisure would you mind checking the structure against our requirements, and if it fits, give the green light? From there, I can do the mass import + cleaning.

[mdlincoln]

Thanks for the example files @PatrickHelling! I was on vacation last week, so once I catch up with the backlog at work I wil give them a look (probably late this week or early next)

[mdlincoln]

The example file is very close to correct - and in fact, I had to adjust our validation schema to allow elements like xml:id, and fixed a broken link to our example TEI-XML file.

The pull request I made on your repository shows the 3 changes that are needed:

1. You must add xmlns="http://www.tei-c.org/ns/1.0" to the TEI element
2. xml:id cannot start with a number (I found this surprising, but it's in the XMLSchema definition apparently - so it was underlying XMLSchema rules, not our own custom DH Abstracts schema, that was throwing errors here)
3. the title and text elements need to have xml:lang attributes. In this case, xml:lang="de"

Let me know how that sounds.

[mdlincoln]

example valid TEI file: https://dh-abstracts.library.cmu.edu/static/tei/valid_tei/abstract_tei.xml

[christofs]

Awesome! Patrick will know more but we discussed plans to fix your (2) already; the other points should not be a big deal either, except that we may have a few abstracts in English in between and it would make sense to identify them.

[scottbot]

Glad this is doable @christofs! Let me know when you're able to get to it, and I can start importing.

[PatrickHelling]

@mdlincoln Thank you for your feedback. I added an new example file (Beispiel_Steyer_AbstractEnhancement_092021.xml) to the repository: transform-DHd-IoDHC maybe you can have a look at it again? Hopefully it meets your requirements now. If so, please let me know so that we can transform all data we have collected and make it available to you.

[mdlincoln]

FYI @PatrickHelling your new xml seems to work just fine 👍

[PatrickHelling]

@mdlincoln we finally added all data from 2016-2020 to the repository transform-DHd-IoDHC. Hopefully the data is now ready for your indexing process. If there are any further problems, please let me know.