Build Streaming Data Pipelines on GCP with Dataflow (2026)

Dataflow isn’t just a managed Apache Beam runner; it’s a serverless, autoscaling engine that abstracts away cluster management for stream and batch processing, allowing you to focus on your data transformation logic.

Let’s see it in action. Imagine we have a stream of user click events, and we want to count how many clicks each user generates within a 5-minute tumbling window.

Here’s a Python snippet using Apache Beam:

import apache_beam as beam
from apache_beam.options.pipeline_options import PipelineOptions
from apache_beam.transforms.window import FixedWindows
import datetime

# Define pipeline options for Dataflow
pipeline_options = PipelineOptions(
    runner='DataflowRunner',
    project='your-gcp-project-id',
    region='us-central1',
    temp_location='gs://your-bucket/dataflow-temp',
    staging_location='gs://your-bucket/dataflow-staging',
    job_name='user-click-counter'
)

# Sample data (in a real scenario, this would come from Pub/Sub or Kafka)
# Each element is a tuple: (user_id, timestamp_in_seconds)
sample_data = [
    ('user1', 1678886400), # March 15, 2023 12:00:00 PM UTC
    ('user2', 1678886410),
    ('user1', 1678886425),
    ('user3', 1678886430),
    ('user1', 1678886450),
    ('user2', 1678886500),
    ('user1', 1678886700), # Still within the first window
    ('user3', 1678887000), # Start of the second window
    ('user1', 1678887010),
]

with beam.Pipeline(options=pipeline_options) as pipeline:
    (
        pipeline
        | 'CreateData' >> beam.Create(sample_data)
        | 'AddTimestamps' >> beam.Map(lambda x: beam.window.TimestampedValue(x[0], x[1]))
        | 'Window' >> beam.WindowInto(FixedWindows(size=300)) # 5-minute windows
        | 'CountClicks' >> beam.combiners.Count.PerElement()
        | 'FormatOutput' >> beam.Map(lambda x: f"User: {x[0]}, Clicks: {x[1]}")
        | 'Print' >> beam.Map(print)
    )

To run this, you’d save it as a Python file (e.g., dataflow_job.py) and execute:

python dataflow_job.py

Dataflow handles provisioning the necessary compute resources, distributing the work, and managing the state for windowing and aggregation. It spins up workers, assigns them processing tasks, and scales the number of workers up or down based on the workload. When the job finishes, it tears down the infrastructure.

The core problem Dataflow solves is the operational overhead of running distributed data processing systems. Instead of managing Kafka clusters, Spark clusters, or Flink clusters, you define your pipeline using Apache Beam, and Dataflow handles the execution. It provides a unified programming model for both batch and streaming data, meaning you can often use the same Beam pipeline code for both scenarios, just by changing the runner option.

Internally, Dataflow translates your Beam pipeline into a Directed Acyclic Graph (DAG) of operations. It then distributes these operations across a fleet of workers. For streaming pipelines, it uses a combination of checkpointing and state management to ensure fault tolerance and exactly-once processing guarantees. When you define a window (like FixedWindows(size=300)), Dataflow tracks which data elements belong to which window and aggregates them accordingly. The Count.PerElement() combiner efficiently aggregates counts within each window for each user. The TimestampedValue step is crucial for streaming; it explicitly assigns a processing time or event time to each element, which is then used by the windowing logic.

The job_name parameter is more than just a label; it’s used by Dataflow to identify and manage your job. If you submit a job with the same job_name and it’s still running, Dataflow will likely treat it as a subsequent update to that existing job, rather than starting a new one. This is key for managing pipeline updates and restarts.

What most people don’t realize is how Dataflow’s autoscaling works under the hood. It monitors metrics like CPU utilization and queue lengths of your worker machines. If the processing load increases, Dataflow automatically provisions more workers to handle the backlog. Conversely, if the load decreases, it scales down the workers to save costs. This dynamic scaling is a significant advantage over fixed-cluster systems, especially for unpredictable workloads. The autoscaling policies are configurable, allowing you to set minimum and maximum worker counts, as well as target CPU utilization.

The next concept to explore is handling late data in streaming pipelines.