TEI-XML import format for conference abstracts

[scottbot]

Per request (https://twitter.com/s_papastamkou/status/1310902488145506304 & https://twitter.com/marijnkoolen/status/1310590002817126401), we should probably figure out a format for data input. Perhaps it's standard DH conference XML, or perhaps it's long data form, such as:

WorkID | VariableType | Value1 | Value2 | Value3 | Value4
1 | Title | "This is a work"
1 | Author | 1 | "Weingart, Scott" | Libraries | Carnegie Mellon University
1 | Author | 2 | "Lincoln, Matt" | Libraries | Carnegie Mellon University
1 | Keyword | "Networks"
...

Then, theoretically, we could have an easy import, that would necessarily be followed by Nickoal or I cleaning up all the import data.

For new conferences (not yet in the database), we'd also need to specify how to give us data on new conferences.

Perhaps we could have a little public upload spot in github, that will act as a queue of work for us, and make it so that even if we don't get to conferences, the data will still be available in the public should people want it, or should we get to it much later.

[lb42]

Is there a standard dB conference xml format? Really? With a schéma and e erything ?

[scottbot]

@lb42 Not formally standardized as far as I know, but ADHO's been relatively consistent in recent years (https://github.com/ADHO/) with xml outputs, so we might want to stick with what works and is becoming normal.

[lb42]

Sure. I can send you entries for drha and for early tei conferences if u need

[mdlincoln]

Picking this up re: dhd-boas-app convo with @scottbot today. The three issues are:

1. Importing conference / conference series data
2. Importing abstracts for a conference
3. Make import code usable from front-end/admin interface

Importing conference / conference series data

I wrote the existing management command import_conferences.py to load the original spreadsheet @scottbot and @nickoal put together (re: #344). Using a spreadsheet with the format specified in that issue should work to bulk load new conference / series data.

Importing abstracts for a conference

I wrote import_dh_xml.py to import folders of TEI following the format in https://github.com/ADHO/. This imports one folder full of TEI for a single conference at a time, taking the database ID of an already-entered conference and then the path to the directory of TEI files.

The ADHO TEI format will correctly separate first and last names, capture work_type from the category element and keywords from the keywords element in that TEI. It gets sticky for matching up affiliations. The affiliation element in ADHO TEI is just a string. The parsing strategy I wrote was:

<affiliation>token1..., token1.3, token1.2, token1.1; token2..., token2.3, token 2.2, token2.1</affiliation>

1. Split affiliation string on ; to process multiple affiliations
2. Tokenize affiliation string by ,. Starting from the right-most token, try to match
   1. Country
   2. City
   3. Institution
   4. Any remaining tokens are kept together and treated as the "Department" field of an Affiliation linked to any matched Institution

This strategy kept the parsing flexible enough that both "Carnegie Mellon University" as well as "Carnegie Mellon University, Pittsburgh, USA" would get successfully matched.

Make import code usable from the front-end/admin interface

Assuming no changes to the underlying parsing code, this would take 2-4 days of work. The extra uncertainty is depending on how troublesome file upload management proves to be.

[scottbot]

We should provide an exemplar TEI file that someone can match. Additionally, perhaps we can include what the file ought to include to deal with the affiliation element, to separate keywords from topics, etc.

[mdlincoln]

Makes sense. I can work off of one of the existing openly-licensed TEI docs and add in some comments to it.

[mdlincoln]

First draft of a sample TEI-XML file up here: https://github.com/cmu-lib/dhweb_app/blob/master/dh_abstracts/app/abstracts/static/files/abstract_tei.xml

@scottbot lmk if the comments make sense.

[lb42]

If this is a sample of what you might hope people will produce for you, maybe some comments on its use of TEI would be helpful. So here are a few
-- the sample has some text in Spanish and some in English. It would be nice to distinguish these (use the xml:lang attribute)
-- the text has lots of elements with @rend="normal", presumably artefacts of some conversion from Word vel sim. Are these much use for anything? Just use <hi> if you want to show italicized text
-- what is the sample <publicationStmt> for? It should contain publication details of the TEI file itself if any. A simple <p> with some content like 'Sample for Name of Project' would do
-- is this meant to conform to a specific TEI Schema or not? "based on the format accepted by ..." is a bit vague!
-- TEI says, under spec for <abstract> "Any abstract already present in the source document should be encoded as a div within the front, as it should for a born-digital document." Not in the <body>. Just sayin.

[mdlincoln]

@lb42 all great points for ADHO that produces this TEI. Scott this should definitely be a convo with... Christof Schöch?

[scottbot]

I'm not entirely certain. @christofs @djakacki we've got some dhconvalidator questions - to whom do we direct them?