Optimizing ArgoCD repo-server to work with Kustomize in monorepo

But why an ArgoCD instance with 20 applications had this issue, while ArgoCD instance with 200+ applications didn’t? The difference lies in the usage.

Both ArgoCD instances are using a monorepo, but the affected one is using Kustomize in half of its applications, after checking the in-container memory usage I discovered that multiple instances of Kustomize are taking approximately 80 MB of memory for each process. Given there are 10 applications refreshed simultaneously, then the memory requirements flies up to an astounding level just for a few seconds, but that’s enough to grab a SIGKILL signal from the OOM Killer, that guy does not have mercy.

Everytime, when a commit is pushed to the repository the ArgoCD cache for that repository is wiped and rebuilt again — to build the cache ArgoCD needs to run Helm, Kustomize and other tools to render the manifests.

The solution

There are two steps which could be taken — thanks to ArgoCD’s flexibility and to it’s documentation.

1. Limit the number of concurrently processed manifests by repo-server

argocd-repo-server fork/exec config management tool to generate manifests. The fork can fail due to lack of memory and limit on the number of OS threads. The — parallelismlimit flag controls how many manifests generations are running concurrently and allows avoiding OOM kills.

In case you are using ArgoCD operator that does not allow appending commandline arguments to repo-server you can set an environment variable instead: ARGOCD_REPO_SERVER_PARALLELISM_LIMIT

2. Additionally to increase the performance of how the ArgoCD discovers changes in a monorepo we can set an annotation on Application level, to make ArgoCD aware of which commited files belongs to which Application so it will refresh state only of applications affected by the files actually commited

annotations:
# “.” resolves to path specified in .spec.source.path
argocd.argoproj.io/manifest-generate-paths: .