Projects done in the Data Engineer Nanodegree Program by Udacity.com
Documentation for Getting Up and Running w/ indexed.xyz Data
A Semantic Data Reservoir for Heterogeneous Datasets
This is a repository to hold the files and notebooks produced throughout my Udacity's Nanodegree Data Engineering program.
Codebase and data for our paper - Pylon: Semantic Table Union Search in Data Lakes.
The Mannheim Data Integration Benchmark (MaDI-Bench) is an end-to-end benchmark for tabular data integration. It provides integration tasks across five domains, covering schema matching, value normalization, entity matching, and data fusion. It supports difficulty variants and measuring step-wise as well as end-to-end performance.
A Search Join is a join operation which extends a user-provided table with additional attributes based on a large corpus of heterogeneous data originating from the Web or corporate intranets.
Discussion of DTF software architecture Repository
Snowflake — independent third-party profile of a public API surface, by API Evangelist. Snowflake is a cloud-based data platform that provides data warehousing, data lakes, data engineering, data science, and data application development capabilities.
The project aims to create a data architecture that connects marketing, logistics, and inventory, automating order processing and report generation across departments for an industrial and medical gas company. The system will streamline data flow and enhance communication across departments.
Data governance implementation for multiple data lakes and warehouses: inventory, policy, accounting, experiment lineage. Web GUI for visualization and management.Remote API access for AAA over multiple data sources.
Follow along with materials in the book "Modern Data Architectures with Python: A practical guide to building and deploying data pipelines, data warehouses and data lakes" (Lipp, 2023)
Data Engineering Nanodegree Program
An opinionated blueprint for showcasing modern data engineering on commodity hardware. The stack mirrors familiar AWS analytics services with fully open-source components so you can demo end-to-end pipelines, ML workflows, and BI storytelling from your homelab.
A plug-and-play Big Data & Data Lake platform. Features containerized infrastructure with dynamic workspaces, Apache Spark, MinIO, and a custom FastAPI control plane.
Azure data fundamentals portfolio with 3 labs on relational databases, non-relational storage, and large-scale analytics. Hands-on experience with SQL, PostgreSQL, MySQL, Azure Storage, Synapse Analytics, Spark, Delta tables, and KQL for batch and streaming data.
Varada — independent third-party profile of a public API surface, by API Evangelist. Varada was a Tel Aviv-based data lake analytics accelerator that used proprietary, patented adaptive indexing to dramatically speed up Trino/Presto queries directly on data lake storage.
To associate your repository with the data-lakes topic, visit your repo's landing page and select "manage topics."