This would be a short story of a case, that happened to me. One of many development teams had reported synchronization failures in ArgoCD, after checking it we discovered that ArgoCD’s repo-server Pod was restarting a few times. Reason? OOMKilled, pretty popular case.
But why an ArgoCD instance with 20 applications had this issue, while ArgoCD instance with 200+ applications didn’t? The difference lies in the usage.
Both ArgoCD instances are using a monorepo, but the affected one is using Kustomize in half of its applications, after checking the in-container memory usage I discovered that multiple instances of Kustomize are taking approximately 80 MB of memory for each process. Given there are 10 applications refreshed simultaneously, then the memory requirements flies up to an astounding level just for a few seconds, but that’s enough to grab a SIGKILL signal from the OOM Killer, that guy does not have mercy.
Everytime, when a commit is pushed to the repository the ArgoCD cache for that repository is wiped and rebuilt again — to build the cache ArgoCD needs to run Helm, Kustomize and other tools to render the manifests.
The solution
There are two steps which could be taken — thanks to ArgoCD’s flexibility and to it’s documentation.
- Limit the number of concurrently processed manifests by repo-server
argocd-repo-server fork/exec config management tool to generate manifests. The fork can fail due to lack of memory and limit on the number of OS threads. The — parallelismlimit flag controls how many manifests generations are running concurrently and allows avoiding OOM kills.
In case you are using ArgoCD operator that does not allow appending commandline arguments to repo-server you can set an environment variable instead: ARGOCD_REPO_SERVER_PARALLELISM_LIMIT
2. Additionally to increase the performance of how the ArgoCD discovers changes in a monorepo we can set an annotation on Application level, to make ArgoCD aware of which commited files belongs to which Application so it will refresh state only of applications affected by the files actually commited
annotations:
# “.” resolves to path specified in .spec.source.path
argocd.argoproj.io/manifest-generate-paths: .
Share with your friends and followers
Start blogging about your favorite technologies, reach more readers and earn rewards!
Join other developers and claim your FAUN account now!
Damian Kęska
Platform Engineer
@keskad
User Popularity
71
Influence
7k
Total Hits
1
Posts
Read, Learn, Know, Teach
Hand curated newsletters for Developers, private Slack with like minded people, podcasts, job offers, news and more!
Only registered users can post comments. Please, login or signup.