De‐duplicate and Cluster Existing FOLIO Inventory Records

For this use case, use LLMs to cluster and de-duplicate FOLIO Inventory records (Instances, Holdings, and Items) with both a single institution's collections as well as combined union catalog for a consortium of libraries.

• Could a hybrid approach using LLMs and other technologies reduce the time and complexity of current approaches? How?

Yes, an LLM could perform more fuzzy de-dupe analysis on fields like title or author.

• Is there an opportunity to coordinate with other libraries (both FOLIO and other ILSs) to share the models and approaches?
• What opportunities exist for the larger organizations (OCLC, POD, Blue Core, etc) for collaboration?

Stakeholders

• Catalog Librarians
• Discovery users
• AI Implementers
• Creator

Ethical Issues

• The transparency of the technique
• The loss of gainful employment (loss of institutional knowledge)

Requirements

• Implicated Modules:
  • Mod Source Record Storage
  • Mod Inventory
  • Mod Inventory Storage

Possible Workflows

possible github issue for this:

Here’s a GitHub issue template that describes the requirements for the creation of this DAG in detail:

---

Title: Create Airflow DAG for MARC Record Comparison using FOLIO Source Record Storage

---

Description

We need to create an Apache Airflow DAG that retrieves two MARC records from FOLIO’s Source Record Storage by HRID and compares the records by specific fields to identify duplicates. The output should be a JSON object with key-value pairs for each field indicating whether a match was found (true/false).

Acceptance Criteria

• The DAG should retrieve two MARC records from FOLIO by their HRID.
• The retrieved records will be compared on the following fields: OCLC, ISBN, ISSN, LCCN, and Call Number.
• The comparisons should follow specific normalization rules as described below.
• The DAG should output a JSON object containing the comparison results for each field.

Normalization and Comparison Rules

OCLC Match

• Only consider values with the (OCoLC) prefix as valid OCLC numbers.
• Normalize OCLC numbers by:
  • Removing all non-numerical characters.
  • Trimming all leading zeros.
• An OCLC match is considered valid if:
  1. The normalized OCLC number from 035 ord:1 sf:a of the source record matches the normalized OCLC number from 035 ord:1 sf:a of the target record.
  2. The normalized OCLC number from 035 ord:1 sf:a of the source record matches any of the normalized OCLC numbers within the 035 sf:z fields of the target record.
• The srs_marctab table in FOLIO should be used for this comparison.

ISBN Match

• Normalize the ISBN by:
  • Trimming everything after the first white space.
  • Taking the last 10 characters and removing the last character.
• The normalized ISBN of the source instance is considered a match if it is identical to the normalized ISBN of the target instance.
• The instance_identifiers table should be used for this comparison.

ISSN Match

• No normalization is required for ISSNs.
• A match is considered valid if the 022 ord:0 sf:a of the source instance is identical to the 022 ord:0 sf:a of the target record.
• The srs_marctab table should be used for this comparison.

LCCN Match

• No normalization is required for LCCNs.
• A match is considered valid if the source LCCN is identical to the target LCCN.
• The instance_identifiers table should be used for this comparison.

Call Number Match

• No normalization is required for Call Numbers.
• A match is considered valid if any of the source instance’s holdings’ call numbers match any of the target instance’s holdings’ call numbers.
• Use the call_number, call_number_prefix, and call_number_suffix columns from the holdings_ext table in FOLIO Reporting for this comparison.

Technical Requirements

• The DAG should use Python and appropriate Airflow operators (e.g., PythonOperator).
• XComs should be used to pass MARC records between tasks.
• The comparison results should be output as a JSON object with fields corresponding to oclc, isbn, issn, lccn, and call_number, each returning true or false.

Example Output

{
    "oclc": true,
    "isbn": false,
    "issn": true,
    "lccn": false,
    "call_number": true
}

References

• FOLIO Source Record Storage API documentation
• FOLIO Reporting tables (srs_marctab, instance_identifiers, holdings_ext)

---

Additional Context

• Authentication to FOLIO API should use an existing access token mechanism.
• Ensure the correct FOLIO environment URL and authentication headers are used in the DAG.
• The task should be able to handle errors such as missing HRIDs or failed API requests gracefully.

ChatGPT Applying this criteria

You said: Using these guidelines please normalize this: (OCoLC)04114770 ChatGPT said: ChatGPT The normalized OCLC number for (OCoLC)04114770 is 4114770. This result was achieved by removing non-numeric characters and leading zeros, as per the normalization guidelines. ​​

You said: Using these guidelines please normalize this: 0084-1382 ChatGPT said: ChatGPT The normalized ISSN for 0084-1382 is 0084-1382. According to the guidelines, no normalization is required for ISSNs, so it remains unchanged. ​​

You said: Using these guidelines please normalize this: 9783828840355 ChatGPT said: ChatGPT The normalized ISBN for 9783828840355 is 382884035. This was done by taking the last 10 characters and removing the last one, according to the normalization guidelines. ​