The author compares the cost and performance of various big data processing frameworks including BigQuery, Dataflow, and Spark for processing data in their organization's tiktok-like feed.
They generate a set of reproducible data and run benchmark tests locally and on GCP. They find that while BigQuery is cheaper, it has limitations in scalability and code sustainability.
Dataflow and Spark are more expensive due to framework bloat and parallelization overhead. The author suggests that with some tweaks, Spark can be optimized further.
Share with your friends and followers
Start blogging about your favorite technologies, reach more readers and earn rewards!
Join other developers and claim your FAUN account now!
Only registered users can post comments. Please, login or signup.