After weeks of investigation, it was discovered that the cause of a performance issue in a Kubernetes cluster was related to DNS. The investigation led to uncovering multiple issues, including exceeding the limit of concurrent DNS queries, missing caches, and the notorious "ndots" issue. By implementing various solutions such as increasing query limits, enabling DNS caching, and modifying hostnames, the performance improved significantly.
















