The author compares the cost and performance of various big data processing frameworks including BigQuery, Dataflow, and Spark for processing data in their organization's tiktok-like feed.
They generate a set of reproducible data and run benchmark tests locally and on GCP. They find that while BigQuery is cheaper, it has limitations in scalability and code sustainability.
Dataflow and Spark are more expensive due to framework bloat and parallelization overhead. The author suggests that with some tweaks, Spark can be optimized further.
