35 Million Hot Dogs: Benchmarking Caddy vs. Nginx

This blog post compares the performance of Caddy and Nginx as reverse proxies.

The author benchmarks the proxies in three areas: purely in-proxy “synthetic” responses, serving files, and reverse proxying.
They vary variables such as concurrent requests and default versus optimized settings to measure system resource utilization, HTTP response latency, total number of requests processed, and error rate.
The author used k6 as their HTTP test driver and measured resource use with psrecord.

These are some of the key findings:

The most striking piece of new knowledge for the author was learning about failure modes. Nginx will fail by refusing or dropping connections, Caddy will fail by slowing everything down. Is one better than the other? For certain use-cases, almost certainly. Some folks will want fail-fast, while others will want to keep accepting connections at all costs. The key point is that there is a difference.
As predicted, “synthetic” responses seem to be easiest, followed by small HTML file content, followed by reverse proxied requests, followed by large HTML content. Nginx’s caching behaviors let it really shine with static asset files (and, presumably, its sendfile capability).
Caddy pays the cost for garbage collection, but it’s not an oppressive cost (at least at “normal” traffic levels). Nginx uses an almost effortless amount of memory. Caddy peaked at almost 160MB real memory allocated for non-breaking tests, which may or may not be significant depending on what amount of total memory is available to the OS. It remains to be seen what specific code path is causing all this malloc/free, but my tests seem to point at whatever mechanism underlies reverse_proxy.
Caddy’s default configuration is good. We get all our cores unlocked and we rev up every resource available to use without any memory leaks. Nginx is scoped to a single core by default.