Skip to content

README.md

Latest commit

 

History

History
205 lines (141 loc) · 7.83 KB

README.md

File metadata and controls

205 lines (141 loc) · 7.83 KB
Raw

Data ingestion with dlt

Homework: dlt_homework.md

🎥 Watch the workshop video

Watch the workshop video

Welcome to this hands-on workshop, where you'll learn to build efficient and scalable data ingestion pipelines.

What will you learn in this workshop?

In this workshop, you’ll learn the core skills required to build and manage data pipelines:

  • How to build robust, scalable, and self-maintaining pipelines.
  • Best practices, like built-in data governance, for ensuring clean and reliable data flows.
  • Incremental loading techniques to refresh data quickly and cost-effectively.
  • How to build a Data Lake with dlt.

By the end of this workshop, you'll be able to build data pipelines like a senior data engineer — quickly, concisely, and with best practices baked in.

📂 Navigation & Resources

📖 Course overview

This workshop is structured into three key parts:

1️⃣ Extracting Data – Learn scalable data extraction techniques.
2️⃣ Normalizing Data – Clean and structure data before loading.
3️⃣ Loading & Incremental Updates – Efficiently load and update data.

📌 Find the full course file here: Course File

👩‍🏫 Teacher

Welcome to the DataTalks.Club Data Engineering Zoomcamp the data ingestion workshop!

I'm Violetta Mishechkina, Solutions Engineer at dltHub. 👋

  • I’ve been working in the data field since 2018, with a background in machine learning.
  • I started as a Data Scientist, training ML models and neural networks.
  • Over time, I realized that in production, hitting the highest RMSE isn’t as important as model size, infrastructure, and data quality - so I transitioned into MLOps.
  • A year ago, I joined dltHub’s Customer Success team and discovered dlt, a Python library that automates 90% of tedious data engineering tasks.
  • Now, I work closely with customers and partners to help them integrate and optimize dlt in production.
  • I also collaborate with our development team as the voice of the customer, ensuring our product meets real-world data engineering needs.
  • My experience across ML, MLOps, and data engineering gives me a practical, hands-on perspective on solving data challenges.

Homework

Running dlt pipelines with an orchestrator

To run a dlt pipeline on a schedule, embed it into a workflow orchestrator such as Apache Airflow or Kestra.

Apache Airflow

Wrap the pipeline call in a PythonOperator:

from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime, timedelta
import dlt
from my_dlt_pipeline import load_data  # Import your dlt pipeline function

default_args = {
    "owner": "airflow",
    "depends_on_past": False,
    "start_date": datetime(2024, 2, 16),
    "retries": 1,
    "retry_delay": timedelta(minutes=5),
}

def run_dlt_pipeline():
    pipeline = dlt.pipeline(
        pipeline_name="my_pipeline",
        destination="duckdb",  # Change this based on your database
        dataset_name="my_dataset"
    )
    info = pipeline.run(load_data())
    print(info)  # Logs for debugging

with DAG(
    "dlt_airflow_pipeline",
    default_args=default_args,
    schedule_interval="@daily",
    catchup=False,
) as dag:
    run_dlt_task = PythonOperator(
        task_id="run_dlt_pipeline",
        python_callable=run_dlt_pipeline,
    )
    run_dlt_task

Kestra

Run the pipeline from a script task:

id: dlt_ingestion

namespace: my.dlt

description: "Run dlt pipeline with Kestra"

tasks:

- id: run_dlt

  type: io.kestra.plugin.scripts.python.Commands

  commands:

  - |

    import dlt

    from my_dlt_pipeline import load_data  # Import your dlt function

    pipeline = dlt.pipeline(

      pipeline_name="kestra_pipeline",

      destination="duckdb",

      dataset_name="kestra_dataset"

    )

    info = pipeline.run(load_data())

    print(info)

Replace "duckdb" with your actual destination and adjust load_data to match your own pipeline.

Next steps

As you are learning the various concepts of data engineering, consider creating a portfolio project that will further your own knowledge.

By demonstrating the ability to deliver end to end, you will have an easier time finding your first role. This will help regardless of whether your hiring manager reviews your project, largely because you will have a better understanding and will be able to talk the talk.

Here are some example projects that others did with dlt:

If you create a personal project, consider submitting it to our blog - we will be happy to showcase it. Just drop us a line in the dlt Slack.

💛 If you enjoy dlt, support us!

Community notes

Did you take notes? You can share them by creating a PR to this file!