USE 259 - parse HTML for mitlibwebsite source

[ghukill]

Purpose and background context

This PR updates the transformer for mitlibwebsite to utilize the full, rendered HTML that is now coming out of browsertrix-harvester for the source records.

Most impactfully, we can now set the new fulltext field with full-text extracted from the HTML in a way that makes sense for this source specifically (not just using the full-text extraction from browsertrix which is good, but not great).

More mechanically, this recreates the minimal amount of metadata parsing from browsertrix-harvester that was performed into Transmogrifier (at this time, really only an OpenGraph og:description element that we map to summary). However, we're setup nicely in the future if we want to extract more metadata from the original page HTML.

How can a reviewer manually see the effects of these changes?

1- Set AWS Dev1 credentials

2- Start ipython shell

pipenv run ipython

3- Load the MITLibWebsite transformer with example records from S3:

from transmogrifier.sources.transformer import Transformer


transformer = Transformer.load(
    "mitlibwebsite",
    "s3://timdex-extract-dev-222053980223/scratch/mitlibwebsite-2025-12-10-full-extracted-records-to-index.jsonl",
)

• These records were prepared with the updated browsertrix-harvester that includes the full HTML in the records

4- Transform a single record to inspect:

import json

record = next(transformer)
transformed_record = json.loads(record.transformed_record.decode())

# note that summary is still set, parsed from HTML in Transmog
print(transformed_record['summary'])

# note that we are now setting the fulltext field as well
print(transformed_record['fulltext'])

Includes new or updated dependencies?

NO

Changes expectations for external applications?

YES:

• Full-text is now available in the TIMDEX record for the mitlibwebsite source.
• If needed, this HTML parsing could be utilized to extract more granular, source specific metadata in the future.

What are the relevant tickets?

• https://mitlibraries.atlassian.net/browse/USE-259

Code review

• Code review best practices are documented here and you are encouraged to have a constructive dialogue with your reviewers about their preferences and expectations.

[jonavellecuerdo]

Exciting! 🤓