[BUG] New pods shortly deleted, and the old pods remains. ArgoCD about reloader

[MuneebAijaz]

@qdrddr i think this would need more investigation on if the reason really is ArgoCD. and if it is, should it watch the Env field under deployments. if not, ArgoCD provides ways to ignore specific fields in specific resources

[shameemshah]

we are also facing the same issue

[MuneebAijaz]

@qdrddr the way reloader works is by updating env var, which triggers a deployment change and update it sent to the pods. Could there be any case that argocd is actively reverting the changes to the deployment done by reloader, in that case new pods will be deleted and old state will be persisted?

[qdrddr]

@MuneebAijaz would you recommend how to workaround this and continue using ArgoCD?

[qdrddr]

Could you point to the documents for this?

Also, I have doubts about the proposed workaround to set ArgoCD to skip checking parts of a resource, as even if ArgoCD ignores Envs, it still will see and kill extra containers with auto-prune settings regardless of changes in Env.

So basically, I cannot use Reloader with ArgoCD with enabled auto pruning. @MuneebAijaz

[qdrddr]

The problem is that Reloader creates additional containers before killing outdated containers to reduce impact, which is an excellent strategy. But ArgoCD, with enabled Pruning, notices an extra container and kills it before Reloader gets a chance to kill the outdated container. Resulting in the outdated containers remain unchanged.

Do you know if ArgoCD integration is needed here?

Ideas on how this can be fixed:

1. So ideally, in this scenario of Reloader creating a new container and then killing the old one and so on, instead, Reloader should tell ArgoCD to increase replicas by one, and then Reloader can kill the old containers one by one. So ArgoCD with enabled Pruning would create a new extra container and re-create those deleted by Reloader.
2. Alternatively, you can temporarily turn off auto pruning in ArgoCD for a given app and reload it once the Reloader is complete.
3. A not-ideal scenario that might also work is to check if there is more than one container in the replica set without creating an extra container, kill one by one, and wait till they are re-created by ArgoCD & k8s.

[MuneebAijaz]

I have doubts about the proposed workaround to set ArgoCD

yes, there are implications to that approach. but not the ones stated above.

The problem is that Reloader creates additional containers before killing outdated containers

Reloader doesnt do that, Reloader only updates ENV field in the parent resource (Deployment, Statefulset, Daemonset), and when an ENV is updated, Deployment etc are bound to propagate that change to the pods, so the parent resource spins up another ReplicaSet with new ENV, and Replicaset creates new pod with updated ENV. that is how update is done by Reloader.

Reloader itself doesnt manage pod/container lifecycle, to not have any effect on user's application, it relies on already set deployment strategy in Deployments to propagate that change.

[MuneebAijaz]

I will try to replicate the issue on my end, and get back to you.

[0123hoang]

@qdrddr Did you follow #333 ?
I changed from reloadStrategy: annotations to env-vars and problem gone.

[qdrddr]

@0123hoang Nope, the problem persists with reloadStrategy: env-vars

[Gatschknoedel]

I would debug this by disabling self-heal for the responsible argo app, let reloader do its thing and afterwards check the argo app. My guess is that the application is out of sync and argo is immediately reverting because of that.