Netflix’s “Psycho Pattern” stitched together Spark, Kafka, and Airflow into a relentless micro-batch pipeline. It tracked high watermarks for near-real-time threat detection—fast enough, sharp enough.
Then came the Flink switch. Lower latency? Sure. But it missed the mark. Signal quality stayed flat. Engineering costs climbed. The real problem wasn’t lag. It was garbage signals. False positives—not delay—were killing accuracy.
System shift: Teams are circling back. Stateful micro-batch is looking sturdier than streaming when you care more about truth than speed.
