List: Building pipelines | Curated by Sairamdgr8 -- An Aspiring Full Stack Data Engineer | Medium

Sairamdgr8 -- An Aspiring Full Stack Data Engineer

Dec 4, 2023

8 stories

1 save

Building pipelines

Dogukan Ulu

Data Engineering End-to-End Project — PostgreSQL, Airflow, Docker, Pandas

Repository

Sep 19, 2023

Data Engineering End-to-End Project — PostgreSQL, Airflow, Docker, Pandas

Sep 19, 2023

Simardeep Singh

Building a Data Streaming Pipeline: Leveraging Kafka, Spark, Airflow, and Docker

In our rapidly evolving digital age, data engineering has emerged as the backbone of the modern data-driven world. We’re surrounded by an…

Nov 6, 2023

Building a Data Streaming Pipeline: Leveraging Kafka, Spark, Airflow, and Docker

Nov 6, 2023

In

Data Engineer Things

by

Stephen David-Williams

Data pipelines with Python and SQL — Part 1

What data pipelines are and how Python & SQL are involved

Sep 1, 2023

Data pipelines with Python and SQL — Part 1

Sep 1, 2023

In

Dev Genius

by

Haq Nawaz

Setup PySpark locally & build your first ETL pipeline with PySpark

Using Python, Jupyter Notebook

Oct 2, 2022

Setup PySpark locally & build your first ETL pipeline with PySpark

Oct 2, 2022

AGK

ETL Pipeline using AWS DataPipeline, EMR, S3 and PySpark

When I was learning AWS DataPipeline, I didn’t find much resources to create AWS Data Pipeline using Pipeline Definition json and…

Sep 30, 2022

ETL Pipeline using AWS DataPipeline, EMR, S3 and PySpark

Sep 30, 2022

In

ITNEXT

by

Ramses Alexander Coraspe Valdez

Building Real-time communication with Apache Spark through Apache Livy

Dockerizing and Consuming an Apache Livy environment

Jun 12, 2022

Building Real-time communication with Apache Spark through Apache Livy

Jun 12, 2022

In

TDS Archive

by

Edwin Tan

How to Test PySpark ETL Data Pipeline

Validate big data pipeline with Great Expectations

Dec 6, 2022

How to Test PySpark ETL Data Pipeline

Dec 6, 2022

In

TDS Archive

by

Nicholas Leong

Data Engineering — How to Build a Gmail Data Pipeline on Apache Airflow

Hack your Gmail Inbox

Jul 8, 2019

Data Engineering — How to Build a Gmail Data Pipeline on Apache Airflow

Jul 8, 2019

Sairamdgr8 -- An Aspiring Full Stack Data Engineer
421 Followers
Data Engineer @ AWS | SPARK | PYSPARK | SPARK SQL | enthusiast about #DataScience #ML Enthusiastic#NLP#DeepLearning #OpenCV-Face Recognition #ML deployment
Following
The Medium Blog
Stephen David-Williams
Tomas Peluritis
Manikandan Muthiah
See all (27)

Help
Status
About
Careers
Press
Blog
Privacy
Terms
Text to speech
Teams