Graduating ApplicationSets, and shipping it with ArgoCD about applicationset

[OmerKahani]

Hi, can we have a project / milestone with all the issues from the doc?
It will make it easier to follow the status

[jgwest]

As discussed at today's meeting, Thursday December 17th:

ApplicationSet Integration

Project Logistics

• We'll begin the process to graduate ApplicationSet repository from argoproj-labs/ to argoproj/, likely in the new year (will just be an 'Argo CD ecosystem' project, eg not a top-level Argo project)

Focus on repository QoL improvements, automated testing (unit/E2E), and documentation

• I've opened a set of new issues that aim to get AppSet ready for closer Argo CD integration
  • Adding new E2E tests (tests that target live k8s/argo cd) for supported generators
  • Investigate unit test coverage data, to identify if there are any unit test improvement areas
  • Writing additional end-user focused documentation, based on (and adding to) the great design proposal docs that folks have already written
  • Some additional GitHub commit/PR checks to improve repo QoL for devs/reviewers working in the codebase
  • For those reading this, if one of the issues jumps out at you, feel free to take a look!

Initial integration strategy

• Complete the code quality and testing, as above

• Complete the items required on the Argo CD 1.9 side, for ApplicationSet

  • Add needed API to reposerver in Argo CD, on behalf of ApplicationSet
  • Create a new kustomize configuration in Argo CD repo, that generates a new install.yaml-style yaml, containing a combined install of ArgoCD+ApplicationSet controller (see
    argo-cd/hack/update-manifests.sh). Allows end users to quickly kubectl apply themselves an install that contains both.

• At this stage:

  • The ApplicationSet codebase will live in argoproj/applicationset, remaining separate from the Argo CD codebase
  • ApplicationSet will ship as part of Argo CD via an optional kustomize-generated install YAML (see argo-cd/manifests) that allows users to easily install ArgoCD+AppSet together
    • It will also be possible to install ApplicationSet against an existing Argo CD install, after the fact
  • ApplicationSet to live in its own k8s Deployment, with a single replica (no need to support cluster sharding akin to application controller, at this time)

Post Release

• Once we've completed initial integration as described, and we've released a release of ArgoCD+AppSet, we will ask the community for feedback to help us validate ApplicationSet functionality with a wider user base
• We've completed initial discussions and gathered everyone's thoughts on whether merge the AppSet codebase into Argo CD, and whether to merge the Argo CD and ApplicationSet k8s artifacts.
  • As above, we are keeping both separate for the time being, and future decisions here will rely heavily on community feedback/usage on AppSet controller, and via shared agreement over the technical merits of integration.

Improving user self-service scenario with ApplicationSet controller

Also discussed at the meeting was @alexmt and @jessesuen's suggestions regarding an additional use case that it would be beneficial for ApplicationSet controller to handle.

There are a couple of user roles that are important here:

• Cluster administrators: Privileged administrators with full cluster access and full git org access. They are responsible for both cluster admin and Argo CD admin (creating Argo CD Applications and Projects for devs)
• Developers (cluster end users): Developers wishing to deploy applications to the cluster, via Argo CD, but not having full cluster access. For example, they might be limited to a particular namespace (and repo) and likely not able to install CRD, cluster roles, etc. Thus they need (in most cases) Argo CD Apps and Projects created for them.

The 'self-service' use case seeks to allow developers (cluster end users) greater flexibility to:

• Deploy multiple applications to a cluster
• Deploy to multiple clusters
  But, in both cases, empowering developers to do this on their own (by committing changes JSON/YAML manifests, containing AppSet template data, to a Git repo), without needing to involve a cluster administrator to create the Argo CD Applications/Projects on their behalf (as doing so requires privileged access, which devs lack).

Alex raised the question of whether end users should be able to create/update Argo CD Projects using application set controller (a 'project set'), especially in the case where projects didn't exist. For instance, in this case, would ApplicationSet throw an error if a targeted Argo CD Project didn't exist, or it would it create and manage it? Jesse noted that he didn't believe that application sets should span projects, as a general rule.

At present, the project field of an ApplicationSet can be parameterized with values from the generator, allowing one to target multiple projects from a single ApplicationSet.

@jopit noted allowing the cluster/list generator to target multiple applications would be beneficial, as presently these generators are more limited in their scope. Jesse noted that the ability to add labels to cluster secrets would be beneficial here (but that it is not currently supported in Argo CD).

Ultimately, the generators currently implemented should improve this use case to some degree, but the Git file generator (a WIP PR) should provide the greatest power/flexibility for end-users, with additional improvements described above to be considered further in the future.

[rajivml]

@sbose78 @jgwest since, app-of-apps will be eventually replaced with ApplicationSet, will ApplicationSet support application ordering as well i.e. the order in which the applications can be deployed, today the same can be achieved using app-of-apps sync-wave ordering but I didn't find anything about deployment ordering under the ApplicationSet design doc

[rajivml]

thanks @jgwest as long as there is an alternative it's fine to move over to ApplicationSet because for us, Application deployment ordering is a P1 requirement and that's we have onboarded to ArgoCD

[dgoodwin]

I apologize I can't make it today, but I can start attending in the new year if that is helpful.

[sbose78]

No worries, thank you for the note. See you soon!

[jgwest]

Hi @rajivml, the short answer is no, this is out of scope for AppliationSet functionality at this time, but another proposed solution is being tracked in this Argo CD issue.

For a longer answer, see the above link, plus this discussion of relationship between ApplicationSet and sync order in Argo community slack: https://argoproj.slack.com/archives/C014ZPM32LU/p1600274721058300

[jgwest]

I've created a new milestone to track our progress towards Argo CD integration. Within the milestone I've added all the items discussed on this tracking issue, the google doc, and related meetings, plus a few issues that have cropped up since then. If you had an open PR that is in review, I have added it to the milestone as well.

Note that this milestone is focused on tracking Argo CD integration, and should not be seen as gating the merge of items that are not on the list: PRs for other items continue to be welcome.

Finally, the list is fluid (more to be added as needed), so feel free to discuss on GitHub or in Argo community slack if there are additional items that you feel could/should be included.