Lessons from scaling PostgreSQL queues to 100K events

RudderStack cranked PostgreSQL up to 100K events/sec as a queuing engine. The secret sauce: tight tuning of job partitioning, smarter indexing, tuned VACUUM timing, and compaction that didn't choke.

Recursive CTEs stood in for loose index scans. Caching cut I/O repeats. They ditched byte slices to sidestep the hex-encoding tax—standardized everything to Go strings instead. Bulk inserts rode in through COPY.

System shift: Ditching Kafka for PostgreSQL isn’t just a clever hack—it’s part of a broader move. Some teams are trading complex pipelines for sturdy, transactional databases that can hold their own.