Batch Processing Design Patterns

Explain common design patterns for large-scale batch processing.

Partition Parallelism

Split data by key (e.g., date, user ID range) and process each partition independently in parallel, significantly reducing total processing time.

Incremental Processing

Process only new/changed data since the last run instead of full reprocessing. Track a high watermark or the last processed max ID/timestamp.

Checkpoint and Fault Tolerance

Long-running batch jobs should checkpoint periodically (persist intermediate results). On failure, resume from the latest checkpoint instead of starting over.

Data Skew Handling

If certain keys have far more data than others (hot products, super users), some tasks run extremely slowly. Solutions: Salting (scatter hot keys), Broadcast Join (broadcast small tables).

Batch Size Optimization

Too small: high task scheduling overhead. Too large: memory pressure and expensive re-runs on failure. Tune based on data volume and available compute.

Output Consistency

When writing batch output to a target system, use atomic writes (e.g., write to a temp table, then RENAME/swap) to prevent readers from seeing partial results.