Building a Flink Self-Serve Platform on Kubernetes at Scale

Instacart handles over two trillion events a year in their data pipelines and adopted Apache Flink for real-time distributed processing in 2021.

They have used Flink for real-time decision-making, data augmentation, machine learning, and OLAP events ingestion. Running Flink on AWS' EMR was not sufficient to meet their high demand, so they built a self-serve Flink platform on Kubernetes.

The new platform has decreased operation costs, infra cost saving, and achieved auto failure recovery. The team learned that Flink service onboarding and operations should be streamlined without Kubernetes details: "Most of our platform users don’t have knowledge of Kubernetes, so we should abstract K8S details as much as possible."