dm-bip v1.0.0

The first stable release of the Data Model-Based Ingestion Pipeline (dm-bip).

What is dm-bip?

dm-bip is a data ingestion pipeline that uses LinkML tools to standardize and transform scientific data into a harmonized data model. It coordinates efforts between BioData Catalyst (BDC) and INCLUDE projects to create a unified approach to data ingestion.

Key Features

Automated pipeline via Make - Schema creation, validation, and data transformation orchestrated through make pipeline
Schema generation - Auto-generate LinkML schemas from input data using schema-automator
Data validation - Validate data against generated or target schemas
Data transformation - Transform data between schemas using LinkML-Map specifications
Multiple output formats - JSON, JSONL, YAML, and TSV output support
Data cleaning utilities - Tools for removing empty columns, replacing values, and reshaping data

Getting Started

# Clone and install
git clone https://github.com/linkml/dm-bip.git
cd dm-bip
uv sync

# See available commands
make help

# Run the pipeline
make pipeline DM_INPUT_DIR=<input> DM_SCHEMA_NAME=<name> DM_OUTPUT_DIR=<output>