List: Apache Beam | Curated by David Regalado

Dec 25, 2024
30 stories
Apache BeamUnified Apache Beam Model | The easiest way to do batch and streaming data processing. Write once, run anywhere data processing for mission-critical production workloads. Execute pipelines on multiple execution environments (runners), providing flexibility
Santosh Beora
A Comprehensive Guide to Windowing in Google Cloud DataflowWhen processing real-time data streams, handling unbounded data is essential. In Google Cloud Dataflow, windowing plays a vital role in…
Nov 6, 2024
Nov 6, 2024
Alexander Rodriguez
Save on GCP Log Storage Costs with Dataflow CompressionIn today’s world, managing large volumes of logs can get expensive fast. As logs accumulate, especially for large organizations, storage…
Nov 18, 2024
Nov 18, 2024
In
Google Cloud - Community
by
Neil Kolban
Processing arriving GCS files with PubSub triggersThe article describes experiments in processing Google Cloud Storage (GCS) files through BEAM where the content is to be inserted into…
Aug 12, 2024
1
Aug 12, 2024
1
David Regalado
Running Apache Beam outside of GCPWhere is Apache Beam being used for production workloads when running outside of Google Cloud?
Jul 24, 2024
Jul 24, 2024
In
Towards Data Engineering
by
Carmel Wenga
Simple Apache Beam Pipeline to load CSV data into a Postgres DatabaseSimple data pipeline with Apache Beam.
Dec 26, 2023
Dec 26, 2023
In
Python in Plain English
by
Müller Fourie
Apache BEAM + Flink Cluster + Kubernetes + PythonWithout going on about all the benefits of BEAM such as open-source and its APIs that alleviates some pain with an added level of…
Feb 11, 2021
3
Feb 11, 2021
3
In
The Deep Hub
by
Vu Trinh
How does LinkedIn process 4 Trillion Events every day?Key insights on how LinkedIn leverages Apache Beam for real-time processing
Jun 10, 2024
5
Jun 10, 2024
5
In
Google Cloud - Community
by
jana_om
☁️GCP Data Engineering Project: Streaming Data Pipeline with Pub/Sub and Apache Beam/Dataflow📡When it comes to streaming data, Kafka and Flink are popular topics of discussion. However, if you are working with Google Cloud Platform…
Apr 29, 2024
Apr 29, 2024
In
TDS Archive
by
Stefano Romanazzi
Running an Apache Beam Data Pipeline on Azure DatabricksA brief walk-through on how to execute an Apache Beam Pipeline on Databricks
Jun 3, 2020
2
Jun 3, 2020
2
David
Apache Beam on Azure: Part 1 — A Basic ExampleA two part article exploring Apache beam on Azure, and using it to transform and move data into Cosmos DB.
Mar 19, 2021
1
Mar 19, 2021
1
Felipe Fernandez
Project: Processing Streaming Data with Google Cloud Pub/sub and DataflowThis project provides a hands-on exploration of using Pub/Sub and Dataflow for streaming data processing, based on Chapter 6 of the “Data…
Feb 26, 2024
Feb 26, 2024
In
Google Cloud - Community
by
jana_om
☁️GCP Data Engineering Project: Connect Four game with Python and Apache Beam 🔴⚫️Get ready for an exciting adventure in the world of Connect Four! I have an awesome task for you: we’ve got 11 players, 88 games, and a…
Jan 31, 2024
Jan 31, 2024
In
Google Cloud - Community
by
jana_om
☁️GCP Data Engineering Project: Building and Orchestrating an ETL Pipeline for Online Food Delivery…This GCP Data Engineering project focuses on developing a robust ETL (Extract, Transform, Load) pipeline for the online food delivery…
Dec 16, 2023
1
Dec 16, 2023
1
Shashwat Pandey
Apache spark vs Apache Beam — What to choose & when ?Introduction: The Need for Big Data Frameworks
Aug 18, 2023
Aug 18, 2023
Varun Krishna Bhaskaran
Origins of Apache BeamIn the past two decades, the world has witnessed a data explosion. As data sets grow larger and more complex, traditional data processing…
May 10, 2023
May 10, 2023
Varun Krishna Bhaskaran
Apache Beam: Reading from, Writing to S3 in Python SDKIn today’s data-driven world, efficient data processing and storage are paramount. Apache Beam, a powerful and flexible tool, provides a…
May 25, 2023
May 25, 2023
Varun Krishna Bhaskaran
Custom Containers for Dataflow: Optimizing Workflows with Docker and GCP Artifact RegistryIn the world of data processing, optimizing workflows is crucial for achieving efficiency and scalability. Custom containers offer a…
Jul 10, 2023
1
Jul 10, 2023
1
Varun Krishna Bhaskaran
Building a Scalable Big Data ETL Pipeline II: Apache Beam Python SDK with MongoDB and S3 I/OThis is the second article on building a scalable ETL pipeline for the HM Land Registry Price Paid Data set. In the previous article…
Jun 15, 2023
Jun 15, 2023
 This story is no longer available
In
NE Digital
by
Nhat
Monitoring BigQuery costs in real time using Cloud Pub/Sub and Apache Beam
Jan 9, 2023
1
Jan 9, 2023
1