Join us

Parsing 1 Billion Rows in Bun/Typescript Under 10 Seconds

Parsing 1 Billion Rows in Bun/Typescript Under 10 Seconds

Bun tries to swallow files over 4GB and promptly chokes. The culprit? Its Buffer caps out at 4GB. The fix? Slice files into chunks under 4GB but keep the buffer lean, no more than 128KB, to keep things zippy. Pull out the big guns—workers. This move fires up all CPU cores, slashing processing time from 18.65s to 9.22s. And yep, go manual with UTF-8 parsing. It's Bun's single-threaded quirk, and it's kind of charming.


Let's keep in touch!

Stay updated with my latest posts and news. I share insights, updates, and exclusive content.

By subscribing, you share your email with @faun and accept our Terms & Privacy. Unsubscribe anytime.

Give a Pawfive to this post!


Only registered users can post comments. Please, login or signup.

Start blogging about your favorite technologies, reach more readers and earn rewards!

Join other developers and claim your FAUN.dev account now!

Avatar

The FAUN

@faun
A worldwide community of developers and DevOps enthusiasts!
Developer Influence
3k

Influence

302k

Total Hits

1

Posts