ETLLib

ETLLib is a command-line toolkit and Python library for munging JSON, TSV, and related data — using Apache Tika where field cleanup helps — and posting the result to Apache Solr.

It is not a workflow engine. You run the commands from a shell, import etl from Python, or call them from a Mnemosyne pipeline. BigTranslate does the last of those: TSV → JSON → translate → Solr.

tsvtojson → repackage → poster
     TSV        split      Solr

It started as Python scripts on DARPA XDATA corpora (Kiva JSON dumps, Computrabajo employment TSVs) wrapped by Apache OODT workflows. Those workflows now run on Mnemosyne.

Commands

Six console scripts install on your PATH. Each is a thin wrapper around etl.etllib. Run command -h for the live usage string.

Command	What it does
tsvtojson	TSV + column headers → one aggregate JSON file
repackage	Split an aggregate JSON into one {id}.json per record
poster	POST JSON documents to Solr
repackageandpost	Split and POST without writing the intermediate files
translatejson	Translate named JSON fields (Tika). Optional extra; see below
similarity	Jaccard similarity / clusters over a directory (Tika metadata)

Install

Python 3.10–3.13. CI covers that range. Packaging is pyproject.toml (setuptools), the same layout as tika-python. You also need the libmagic shared library (python-magic is only the ctypes binding).

man libmagic    # should exist after install

macOS: brew install libmagic. Debian/Ubuntu: sudo apt-get install libmagic1.

git clone https://github.com/chrismattmann/etllib.git
cd etllib
python3 -m pip install -e .
tsvtojson -h

translatejson needs hirlite, which often fails to build on current Python. It is an extra, not a required dependency:

python3 -m pip install -e ".[translate]"

For many-to-English at scale, use BigTranslate / Pantogloss instead. Tika (tika>=1.13) is a normal dependency.

Tests:

python3 -m pip install -e . --group=tests
python3 -m pytest

Example

A Computrabajo-style TSV, one job per row, to one JSON file per job, then Solr:

# colheaders.txt: one field name per line. "salary:" is optional if the row
# is short. "url*" is also copied to id. -s is required: higher keeps more.
tsvtojson -t data.tsv -j aggregate.json -c colheaders.txt \
          -o employmentjobs -e encoding.txt -s 0.8 -v

mkdir json && cd json
repackage -j ../aggregate.json -o employmentjobs -v

find . -name '*.json' | poster \
    -u 'http://localhost:8983/solr/jobs/update/json?commit=true' -v

tsvtojson refuses to overwrite -j. Encodings are tried in order, then latin-1, with per-field UTF-8 recovery for mixed Computrabajo files. Details and a CSV walkthrough are on the wiki tutorial.

Library

from etl.etllib import prepareDocs, writeDoc, recoverMisdecoded
from etl.tsvtojson import detectEncoding, near_dedup_jaccard

The CLIs are the supported interface. Import the same functions if you would rather not shell out.

Docs

• Website: https://chrismattmann.github.io/etllib/
• Wiki: Home · Getting Started · Tutorial · Commands

Python 2.7 / buildout notes are under Old. Changelog: docs/HISTORY.txt.

License

Apache License 2.0. See docs/LICENSE.txt.