Building Real-time Machine Learning Foundations at Lyft

Lyft had a Machine Learning Platform called LyftLearn, but it lacked support for streaming data in many of its systems. To address this, Lyft initiated the Real-time Machine Learning with Streaming initiative to enable developers to efficiently build and enhance models with streaming data. They identified three capabilities for real-time ML applications: real-time features, real-time learning, and event-driven decisions. By creating a common interface called RealtimeMLPipeline, they simplified integrating streaming into ML models, enabling faster iteration and deployment across development and production environments. The project led to the development of real-time ML use cases and garnered interest from various teams within Lyft. They faced technical challenges related to the complexity of streaming applications and made enhancements to the Flink stack.