Data Engineering Workflow

This project constitutes my solutions to the Data Engineering ZoomCamp 2022 Course

During this course the following milestones are achieved:

1. Local hosting of a Postgress DataBase using Docker
2. Local hosting of an Airflow using Docker
3. An ETL pipeline to fill the database with data from the NY Taxi Dataset
4. More comming soon

Setup

Postgress DataBase

In order to create the postgress db simply execute the following command:

docker compose -f ./database/docker-compose.yaml up -d --build

After the initial creation, in order to activate the database execute the previous command without the --build

docker compose -f ./database/docker-compose.yaml up -d