Data Engineer with extensive expertise in building end-to-end batch data pipelines and real-time streaming solutions. Specialized in cloud platforms (GCP, AWS, Azure) and modern data technologies. Passionate about building scalable, reliable, and cost-efficient data solutions that drive business value.

• Google Cloud Platform (GCP) - Databricks, BigQuery, Dataproc, Pub/Sub, Cloud Run, Cloud Composer, and GCS
• Amazon Web Services (AWS) - S3, DynamoDB, Redshift, Kinesis, Lambda, Airflow, EC2, and Glue
• Microsoft Azure - ADLS, Synapse, ADF, CosmosDB, MS-Fabric, Event Hub, Stream Analytics

• Orchestration: Apache Airflow, Cloud Composer, Azure Data Factory, and SSIS
• Streaming: Apache Kafka, Confluent Cloud, Azure Event Hub, Apache Flink, Redpanda
• Data Warehousing: Snowflake, BigQuery, Redshift, Azure Synapse, Databricks
• Data Processing: PySpark, Apache Spark, dbt, Microsoft Fabric, Delta Lake
• Transformation: dbt, Apache Beam, Dataflow, Azure Data Factory
• Storage: GCS, S3, ADLS, Apache Iceberg, Delta Lake

• Languages: Python, SQL (T-SQL), and Pyspark
• Tools: dbt, SSIS, SSRS, PowerBI, ThoughtSpot, Looker, Streamlit, n8n
• Infrastructure: Docker, GitHub Actions, Azure DevOps CI/CD, ARM Templates
• Version Control: Git, GitHub, TFS

• Kafka Stock Market Pipeline - Real-time stock market data processing
• Azure Event Hub BookMyShow Pipeline - Real-time event booking and payment streams with Stream Analytics
• GCP Uber Car Idle Alerts - Pub/Sub to BigQuery real-time alerts using Dataflow
• Databricks UPI Transactions CDC - Real-time transaction change data capture with PySpark Streaming
• Confluent Kafka MongoDB Streaming - Kafka to MongoDB real-time pipelines
• AWS Kinesis Spark Streaming - Real-time food delivery pipeline with Spark Streaming
• Flink Real-Time Processing - Redpanda, Flink, and Postgres integration for real-time analytics

• Flight Booking Pipeline - Airflow → GCS → BigQuery → Looker
• Credit Card Processing - End-to-end credit card data pipeline with Looker dashboards
• YouTube Trends Pipeline - GCS Iceberg tables to BigQuery analytics
• Weather Map API Integration - API data extraction and PySpark transformation
• Bigtable CRUD Operations - NoSQL database operations with Python
• News API GCS to Snowflake - Multi-warehouse pipeline with Airflow
• Airflow Dataproc PySpark - Ephemeral Dataproc cluster orchestration
• Pyspark Backfill Pipeline - Backfill DAGs with PySpark jobs
• Real-time Gaming Leaderboard - Low-latency analytics with Dataflow
• Snowflake Car Rental - GCP Airflow to Snowflake integration
• SCD Type 2 CI/CD - GitHub Actions automation for data processing

• Airline Data Ingestion - S3 to Redshift pipeline with data quality checks
• Movie Quality Pipeline - Data quality framework and monitoring
• S3 to Redshift Pipeline - Airflow orchestration with Redshift
• DBT Snowflake Preset - dbt transformations with Preset dashboarding

• Airbnb CosmosDB Pipeline - Near real-time pipeline with CosmosDB and n8n workflows
• Airport ADF with CI/CD - Data Factory with DevOps automation and ARM templates
• Fintech SQL Pipeline - Multi-warehouse architecture with Synapse
• Analysis Services Model - OLAP cube development from SQL Server
• Olympic Data Engineering - End-to-end analytics project

• Snowflake Movies Streamlit App - Dynamic tables with interactive Streamlit dashboards
• Databricks Travel Booking SCD2 - Slowly Changing Dimensions Type 2 implementation
• Databricks dbt Project - dbt frameworks on Databricks
• dbt Snowflake - dbt with Snowflake integration

• Microsoft Fabric End-to-End - Bronze-Silver-Gold architecture on Fabric
• Microsoft Fabric Uber Analytics - Comprehensive analytics platform
• Microsoft Fabric API PowerBI - API ingestion with PowerBI reporting
• GCP PySpark Streaming Iceberg - Open table format implementation

• Databricks Ecommerce Event-Driven - Event-driven architecture on Databricks
• Databricks Healthcare DLT - Delta Live Tables for healthcare data
• Airflow GCP Complete Stack - Complete GCP visualization stack
• Yahoo Finance API - Financial data extraction with Airflow

• Data Engineer Handbook - Comprehensive data engineering resources and best practices
• Awesome Data Engineering - Curated list of tools, frameworks, and resources
• System Design Academy - System design principles for scalability

• Data Structures & Algorithms (Python) - DSA templates, solutions, and practical projects
• Python Web Scraping Projects - Web data extraction techniques and examples
• PySpark Project Files - Spark learning resources and implementations
• Linux Project Files - Linux system administration knowledge base

• ☁️ Azure Cloud Professional Certifications
• ☁️ GCP Data Engineering Certifications
• ☁️ AWS Solutions Architect Certifications

• 🏆 Data Engineering Specialized Certifications
• 🏆 Technology Stack Specific Training
• 🏆 Advanced Tools & Frameworks Certifications

• 📚 Multiple Udemy courses on data platforms
• 📚 Official cloud provider training programs

• 🎖️ Awards and Recognitions in IT career
• 🎖️ Extensive hands-on project experience
• 🎖️ Community contributions and knowledge sharing

✅ Multi-Cloud Architecture - Design and implement solutions across GCP, AWS, Azure
✅ Real-Time Streaming - Kafka, Kinesis, Event Hub, Flink, Pub/Sub expertise
✅ Data Warehousing - Snowflake, BigQuery, Redshift, Synapse optimization
✅ Data Transformation - dbt, PySpark, Apache Spark, SQL optimization
✅ Orchestration - Apache Airflow, Cloud Composer, Data Factory automation
✅ CI/CD & DevOps - GitHub Actions, Azure DevOps, Infrastructure as Code
✅ Data Modeling - SCD, Dimensional modeling, Star Schema, Lakehouse architecture
✅ Analytics & BI - PowerBI, Looker, Streamlit dashboard development
✅ Full-Stack Pipelines - End-to-end implementations from ingestion to visualization
✅ Documentation & Knowledge Sharing - Best practices, tutorials, mentoring

I believe in building production-grade data solutions that embody:

• Scalability - Handle data at any volume with optimal performance
• Reliability - Robust error handling, monitoring, and alerting
• Maintainability - Clean, well-documented, and testable code
• Cost-Efficiency - Optimized resource utilization and cloud spending
• Future-Proof - Aligned with industry best practices and emerging technologies

• GitHub Profile: github.com/ViinayKumaarMamidi
• All Repositories: 75+ Active Projects

I'm passionate about:

• 📊 Designing scalable data architectures
• 🔄 Building reliable data pipelines both batch and real time
• ☁️ Leveraging cloud technologies effectively
• 📈 Transforming data into actionable insights
• 🤝 Sharing knowledge and mentoring others
• ☁️ Exploring AI tools and planning on leveraging AI to build data pipelines in cloud environments

Interested in collaborating? Please feel free to explore my repositories, fork projects, or reach out with ideas or any suggestions!