redhat-developer / gitops-operator

An operator that gets you an ArgoCD for cluster configuration out-of-the-box on OpenShift along with the UI for visualizing environments.

OpenShift GitOps Operator

An operator that gets you an Argo CD for cluster configuration out-of-the-box on OpenShift along with the UI for visualizing environments.

Getting started

Making the operator available on the in-cluster OperatorHub

1. Add the following resource to your cluster:

apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: gitops-service-source
  namespace: openshift-marketplace
spec:
  displayName: 'Gitops Service by Red Hat'
  image: 'quay.io/<quay-username>/gitops-operator-index:v0.0.1'
  publisher: 'Red Hat Developer'
  sourceType: grpc

2. Go to the OperatorHub on OpenShift Webconsole and look for the "OpenShift GitOps" operator.

3. Install the operator in the openshift-gitops-operator namesapce using the defaults in the wizard, and optionally, select the checkbox to enable cluster monitoring on the namesapce. Wait for it to show up in the list of "Installed Operators". If it doesn't install properly, you can check on its status in the "Installed Operators" tab in the openshift-gitops-operator namespace.

4. To validate if the installation was successful, look for the route named cluster in the openshift-gitops namespace. Note: the namespace doesn't have to exist in advance, the operator creates it for you.

That's it! Your API route should be created for you. You don't need to expliclty create any operand/CR.

Contributing

1. Clone the repository.
2. Login to a cluster on your command-line.
3. Execute make install to apply the CRDs.
4. Execute make run to run the operator locally.

Tests

Unit tests

make test

e2e tests

make test-e2e

Re-build and Deploy

This operator currently deploys the following payload:

quay.io/<quay-username>/gitops-backend:v0.0.1

If that's all that you are changing, the following steps are not needed in development mode. You could update your image "payload" and re-install the operator.

Set the base image and version for building operator, bundle and index images.

export IMAGE=quay.io/<quay-username>/gitops-operator VERSION=1.8.0

1. Build and push the operator image.

make docker-build docker-push

2. Build and push the Bundle image ( operator + OLM manifests )

make bundle
make bundle-build bundle-push

3. Build and push the Index image

Install opm binary which is required to build index images

make opm

make catalog-build catalog-push

The Index image powers the listing of the Operator on OperatorHub.

GitOps Operator vs Argo CD Community Operator

Features	GitOps Operator	Argo CD Community Operator
Default Cluster Argo CD instance	✅	❌
Cluster Argo CD instances in namespaces defined by envrionment variable ARGOCD_CLUSTER_CONFIG_NAMESPACES	openshift-gitops	❌
Cluster Configuration RBAC/Policy Rules	All APIGroups,Resources,get,list,watch Verbs appended with admin ClusterRoles. Additional APIGroups: operator.openshift.io,user.openshift.io, config.openshift.io, console.openshift.io, machine.openshift.io, machineconfig.openshift.io, compliance.openshift.io, rbac.authorization.k8s.io, storage.k8s.io, etc.	All APIGroups,Resources,Verbs
Integrated with OpenShift Console Environments page for visualizing GitOps environments and applications	✅	❌
Air-gapped environments	OCP	❌
Installed tools	helm 3, kustomize	helm 2 and 3, kustomize, ksonnet
Single Sign-on	RHSSO, Dex	Keycloak, Dex
Redis Server	Redis 5, Secure connection is not yet supported	Redis 6
ArgoCDExport	❌	✅
Installation Modes	All Namepaces	Single, All Namespaces
Support for Kubernetes	❌	✅
Maintained by	Red Hat	Community

Migrate from Argo CD Community Operator to GitOps Operator

Please follow the migration guide when moving from the Argo CD Community Operator to the GitOps Operator.

Progressive Delivery

OpenShift GitOps from release v1.9.0 supports progressive delivery using Argo Rollouts.

Users can enable progressive delivery by using a new type of CRD called RolloutManager. A detailed usage guide for this new CRD is available here.

gitops-operator's Issues

the repo server does not create a SA, therefore automountServiceAccountToken: false

Describe the bug
The repo server does not create a service account, (unlike server, application-controller etc). If a service account is not defined, I believe the deployment is built with automountServiceAccountToken: false (https://github.com/argoproj-labs/argocd-operator/blob/9d60b042f3c7b60661944ef08f05ebc8e17b8403/controllers/argocd/deployment.go#L775), which in turn does not mount the SA token in /var/run/secrets/kubernetes.io/serviceaccount/token .

This seems to only be a problem in the openshift-gitops namespace on the openshift-gitops ACD (presumably due to the operator). I am trying to sort out whether this is intentional.
To Reproduce
Steps to reproduce the behavior:

1. Install from operator.
2. set mountsatoken and serviceaccount in ACD
3. Check deployment and see automountServiceAccountToken: false, which will prevent the token from mounting.
4. See error

Expected behavior
The ability to mount the SA token as takes place with other pods.
Or... allow modifying automountServiceAccountToken in the openshift-gitops ACD (unless this is intentional)

Screenshots

1. ACD with SA set, 2. Deployment

Additional context

cluster wide secret "<argocd-name>-default-cluster-config" being created causes "Out of Sync"

Issue / Bug Description

• We're using the openshift-gitops operator (v.1.1.2) as a "Parent" instance in a multi-instance setup where the ArgoCD in the openshift-gitops namespace is the parent instance, and it deploys (among other things) other instances of ArgoCD to local and remote clusters. This works fine, but we have observed some issues and have some questions.

• Our ArgoCD apps for deploying other instances of ArgoCD go to "Out of Sync" because of the presence of a secret (which we do not have in our synced Git repo). The secret is called -default-cluster-config, so for example my-argocd-default-cluster-config. I am able to prune this object to make the app status go to "Synced" but it is recreated afterwards.

• I wasn't able to find any docs that explain this secret and what causes it to be created so would appreciate any information. If it is not sensible to remove this secret, what workarounds are possible to prevent it from making our apps go out of sync?

Expected behavior

• "Out of Sync" requires pruning (not expected). What is this cluster wide secret and can its creation be prevented?

Additional context

• ACM installs GitOps operator on cluster; create ArgoCD resource on management cluster; ArgoCD reconciles App of Apps creates apps. This was not working in previous version (v1.1.2) either.

set default limits

Describe the bug
The default install does not set any limits or requests, which will make it fail when a ns has quotas.

  Warning  FailedCreate  17s (x7 over 2m58s)  replicaset-controller  (combined from similar events): Error creating: pods "pti-systemtest-redis-757955d999-xwbwb" is forbidden: failed quota: compute-resources: must specify limits.cpu,limits.memory,requests.cpu,requests.memory

To Reproduce
Steps to reproduce the behavior:

1. have quota for ns
2. create argo install
3. see that pods are not created

Expected behavior
Set default limits (which is seen as a good practice anyways)

Screenshots
N/A

Additional context
N/A

admin pw is salted

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

1. install gitops operator and create an argo CR with default settings to create an instance
2. introspect secret for admin
3. see that it is salted (i.e starts with $"

Expected behavior
A cleartext pw which can be used to login as admin

Screenshots
N/A

Additional context
N/A

ArgoCD image missing tar

Describe the bug
Applications with Helm charts defined in a chart repository fail to be deployed.

To Reproduce
Steps to reproduce the behavior:

1. Create an application with a hosted Helm chart, e.g.:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-application
spec:
  project: my-project
    repoURL: 'https://my-helmchart.repository.org/helm/'
    targetRevision: 0.1.0
    chart: my-chart
  destination:
    server: 'https://kubernetes.default.svc'
    namespace: my-namespace
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

2. Attempt to synchronize the application
3. See error

rpc error: code = Unknown desc = exec: "tar": executable file not found in $PATH

Expected behavior
Chart is downloaded as a "tgz" file and processed correctly.

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.

v1.2 DEX initContainers missing resource definition

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

1. upgrade to 1.2
2. allow the Dex rs to run
3. pods fail
4. Error creating: pods "openshift-gitops-dex-server-..." is forbidden: failed quota: openshift-gitops-compute-resources: must specify cpu,limits.cpu,limits.memory,memory

Expected behavior
Resources need to be defined for initcontainers

Screenshots

incorrect:
initContainers:
- name: copyutil
image: >-
registry.redhat.io/openshift-gitops-1/argocd-rhel8@sha256:df4b11a78ab8f8a3ee758a1a2b549c190ce1467cdd098a1c10468bbf6e76a596
command:
- cp
- '-n'
- /usr/local/bin/argocd
- /shared/argocd-dex
resources: {}

working:

spec:
restartPolicy: Always
initContainers:
- name: copyutil
image: >-
registry.redhat.io/openshift-gitops-1/argocd-rhel8@sha256:3e6521a3610b23dce99f4eb643171ac9172808e86f3ca4154f5d548a286bb95f
command:
- cp
- '-n'
- /usr/local/bin/argocd
- /shared/argocd-dex
resources:
limits:
cpu: 500m
memory: 256Mi
requests:
cpu: 250m
memory: 128Mi
Additional context

provide option to disable clusterrolebinding for multitenancy

issue

We would like to provide argocd multitenancy inside one cluster, provide multiple argocd server instances in different namespaces, hand the administrative responsibility to the respective teams. The upstream argocd implementation fully supports this: https://github.com/argoproj/argo-cd/tree/master/manifests and the operator also supports multiple instances of argocd but it creates a clusterrolebinding for the $argocdinstance-application-controller service account which gives global read rights.
This global read right is allowing the account to read into system or other tenant resources and based on the link and our testing it's not strictly necessary.

request
We would like to install argocd instances which have kubernetes rbac permissions only on their own namespaces and only the operations team can give rbac permissions to individual namespaces for the service account. It would be good if there will be a parameter for the argocd instance to enable/disable the clusterrolebinding of the service account.

Disabling name spaced mode for an argocd instance

Describe the bug

We have been using gitops operator with a single custom argocd ops instance that manages a whole cluster, everything from machineconfigs to configmaps in openshift-config, etc.

This worked fine in version 1.0
Somewhere around version 1.1 namespace isolation was added and namespaced mode was added and a namespace field was added to default-cluster-config secret causing our argocd to throw errors like "Cluster level Some-API-Object Some-Resource can not be managed when in namespaced mode"
As a workaround we could just remove the namespace field in the secret and it would start working again.
Now in version 1.2 if I delete the namespace field in the secret, the gitops-operator reconciles and adds the namespace field to the secret again so our argocd setup doesnt work anymore.

Can you advise on what to do?
Cant see any way to get around this in the docs https://docs.openshift.com/container-platform/4.8/cicd/gitops/ ...

Thanks ...

To Reproduce
Steps to reproduce the behavior:

1. Delete the namespace field from the default-cluster-config secret in namespace my-custom-argocd
2. oc delete pods --all -n my-custom-argocd
3. gitops-operator will show a whole lot of reconcile in the logs
4. Check the default-cluster-config secret again and the namespace field is back

Expected behavior
A way to remove the namespace field in the default-cluster-config secret

When GitOps operator is run locally (not installed via OLM), it does not correctly setup the 'argoproj.io' Role rules for the 'argocd-application-controller'

When GitOps operator is run locally, it does not correctly setup the 'argoproj.io' Role rules for the 'argocd-application-controller' pod/service account.

When all of these are true:

• You run the GitOps operator locally, for example via scripts/run_e2e_tests.sh or with make run-local
• You run it on a cluster that has not previously had the GitOps operator installed via OLM (and likewise not Argo CD operator)

You will find:

• That the argocd-application-controller created within the openshift-gitops namespace is missing permissions to read/write argoproj.io CRs.
• That the following error occurs in the argocd-application-controller pod log:
  time="2021-06-28T19:46:10Z" level=error msg="Error persisting normalized application spec: applications.argoproj.io \"(application name)\" is forbidden: User \"system:serviceaccount:openshift-gitops:openshift-gitops-argocd-application- controller\" cannot patch resource \"applications\" in API group \"argoproj.io\" in the namespace \"openshift-gitops\"" application=(application name)

This is due to how the role is configured, as of argoproj-labs/argocd-operator#318.

When OLM installs an operator, it adds read/write access to the admin ClusterRole. However, if you don't use OLM to install the GitOps operator, then the admin cluster role will NOT contain the Argo CD CR permissions. Thus the admin cluster role cannot be guaranteed to contain the necessary cluster CRDs.

Unfortunately, as of argoproj-labs/argocd-operator#318, the admin ClusterRole is appended to the openshift-gitops-argocd-application-controller namespace Role. So if the admin ClusterRole is missing the Argo CD CRs (because the operator was not installed via OLM), then the openshift-gitops namespace will be missing permissions to write to Argo CD CRs.

TLDR: we can't rely on the admin ClusterRole to necessarily contain the rules we need, because it is written to by OLM, and not every CRD is installed via OLM.

To reproduce:

1. Start with a clean cluster (one on which no one has previously installed GitOps operator):

   • To verify that you have a clean cluster, run the following command:
   • k get clusterrole/admin -o yaml | grep argoproj.io | wc -l
   • It should return 0

2. cd (path to gitops-operator)

3. Run kubectl apply -f deploy/crds to install the CRDs (ignore the error 'The GitopsService "example-gitopsservice" is invalid')

4. Run make run-local

5. Wait for the openshift-gitops namespace to be created, and wait for Argo CD to be installed in it.

6. Run kubectl get role/openshift-gitops-argocd-application-controller -n openshift-gitops -o yaml

   • Notice that it does NOT contain any rules for 'argoproj.io'

7. Delete the openshift-gitops namespace

8. Now install the latest OpenShift GitOp operator from OperatorHub

9. Wait for the openshift-gitops namespace to be created, and wait for Argo CD to be installed in it.

10. Run kubectl get role/openshift-gitops-argocd-application-controller -n openshift-gitops -o yaml

    • Notice that it DOES contain rules that reference to 'argoproj.io'

The difference between steps 6 and 10 is the issue.

Replace time.Sleep() in test suite

Is your task related to a problem? Please describe.

Currently test suite contains time.Sleep() which is not a optimize solution to wait for a status.

Describe the solution you'd like

Replace time.Sleep() with some kind of polling logic.

Describe alternatives you've considered

Additional context

gitops-operator/test/e2e/gitops_service_test.go

Line 158 in b34fd46

time.Sleep(5 * time.Second)

argocd version

Hi
I can't seem to find info about which version of argocd this operator builds on. Which version is it today? And, if not version 1.8 already, when do you plan to release that version with the gitops operator?

Typo in gitops operator description

Describe the bug
There is an extraneous space in the kam URL under the Features section of the operator description:
https://github.com/ redhat-developer/kam
It should be
https://github.com/redhat-developer/kam)

To Reproduce
Steps to reproduce the behavior:

1. From OperatorHub, select the GitOps operator and when the Page appears, check the last bullet under the Features section

Expected behavior
The link does not appear with the typo. The link to kam should appear.

Screenshots

make test-e2e target returns exit code zero on failure

Describe the bug
#127 (comment)

To Reproduce
make test-e2e reports exit code 0 on test failure

Expected behavior
make target should return none zero exit code on any test failure in e2e test

Application-scoped Argo CD doesn't work when managed through Cluster-scoped Argo CD

Describe the bug
When managing a secondary application-scoped ArgoCD instance through a primary ArgoCD instance, the namespace role bindings that the secondary application-scoped ArgoCD instance creates will get deleted by the primary ArgoCD instance.

To Reproduce
Steps to reproduce the behavior:

• Create a cluster-scoped ArgoCD instance
• Create a application-scoped ArgoCD instance
• Label a namespace with argocd.argoproj.io/managed-by=application-argocd
• Role bindings in the new namespace will get deleted by the cluster-scoped ArgoCD

This is because the application scoped ArgoCD instance will get deployed with a label that matches the cluster-scoped ArgoCD Application name

  labels:
    app.kubernetes.io/instance: <Cluster scoped Application name for deploying application scoped ArgoCD instance>

When you then annotate a namespace with argocd.argoproj.io/managed-by=application-argocd, the rolebindings that get created will inherit the app.kubernetes.io/instance label on the application-scoped ArgoCD instance.

This causes the cluster-scoped ArgoCD to pick them up and delete then because ArgoCD picks up instance labels.
https://argoproj.github.io/argo-cd/faq/#why-is-my-app-out-of-sync-even-after-syncing
argoproj/argo-cd#1482

Expected behavior
The ability to manage a secondary argocd instance from a primary argocd instance without conflict.

Avoid imagePullPolicy of Always - replace with IfNotPresent

Describe the bug
The images in use by argo/dex etc are immutable AFAIK, hence there is no need for using imagePullPolicy: Always. Using always makes the workload more sensitive to registries or intermediary proxies being unavailable (because the pod will not use the cached image in these cases - and rather ends up in an ImagePullBackoff). It also puts more strain on the serving image-registry. Lastly it makes pod starts slower as they need to do the pull to check.

A clear and concise description of what the bug is.

To Reproduce
N/A

Expected behavior
Use already cached image if available on node.

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.

ArgoCD CRD should support extraArgs such as kubectl-parallelism-limit

Is your feature request related to a problem? Please describe.
Command line arguments for the various ArgoCD component can not be set when using this Operator.

Describe the solution you'd like
Have the Operator support an extraArgs list under the controller section of the ArgoCD CRD and pass those as command line arguments to the ArgoCD application controller.

Additional context
This is a problem for reducing the default application controller kubectl-parallelism-limit value which can cause OoM errors when the controller Pod starts even with several GB of memory allocated.

Remove resource quota set for the default Argo CD Instance namespace.

Describe the bug
v1.2 sets resource quota openshift-gitops-compute-resources for the default Argo CD instance namespace. This is created with the intention to restrict the namespace with resources and ensure all the workloads are deployed with resource requests and limits.

Users have noted many issues with this default behavior.

1. gitops-operator is an openshift specific operator. So the resource quota is set with the scope "Not Terminating" to support the deployment of deployer pods(specific to deployment configs[DC]). Users have reported issue running this in combination with some machine learning pods which are of scope "Terminating".
2. Few users are using the default Argo CD instance for app delivery. In such cases, Argo CD instance may try to deploy the pods in openshift-gitops namespace which would lead to a failure if the pods does not have resource requests/limits set.
3. The current resource quota is not well tested and users might have to upscale and downscale the quota as per their requirements.

Expected behavior
Remove the resource quota on the default Argo CD instance namespace.

Additional context
NA

Do not install RHSSO OOTB for the default Argo CD instance in openshift-gitops namespace

Describe the bug
The current behavior installs and configures OpenShift Login OOTB using RHSSO for default Argo CD instance. However the team has agreed to revoke this decision as OpenShift Login OOTB does not work on Vanilla OpenShift clusters or clusters without User Management.

Acceptance Criteria:

1. RHSSO should be only Installed and Configured when user explicitly modifies the Argo CD Custom Resource to add .sso.provider: keycloak in the Spec.
2. This is valid for both "default Argo CD instance(openshift-gitops ns)" and "Argo CD Instance in any other namespaces"

To Reproduce
Steps to reproduce the behavior:

1. Run operator locally operator-sdk run local
2. Run oc get pods -n openshift-gitops to see that keycloak pods are running.
3. Go to Argo CD Console -> Login with Keycloak -> Login with OpenShift
4. If you dont have any User Management configured on you OpenShift Cluster you will realise this option is useless or you will end up with an unsuccessful attempt to login using kubeadmin user.

Expected behavior
Do not install RHSSO OOTB for the default Argo CD instance in openshift-gitops namespace

Screenshots
NA

Additional context
NA

Enable OCP 4.9 CI triggers

Keep an eye until #186 is fixed, then only we can enable OCP 4.9 CI triggers

ApplicationController unable to create Events with default install: Unable to create audit event: events is forbidden

Describe the bug

The ApplicationController is unable to create events, due to missing RBAC rules.

The following is printed in the ApplicationController pod log:

Unable to create audit event: events is forbidden: User \"system:serviceaccount:openshift-gitops:openshift-gitops-argocd-application-controller\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-gitops\"" application=guestbook dest-namespace=openshift-gitops dest-

To Reproduce

1. Get a fresh OpenShift cluster
2. Install GitOps Operator 1.1.2 from OperatorHub
3. Login to the cluster from the CLI, with oc/kubectl
4. Kubectl apply this:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: guestbook
  namespace: openshift-gitops
spec:
  destination:
    server: https://kubernetes.default.svc
    namespace: openshift-gitops
  project: default
  source:
    path: guestbook
    repoURL: https://github.com/argoproj/argocd-example-apps.git
    targetRevision: HEAD

5. Check the controller logs: kubectl logs pod/openshift-gitops-application-controller-0

You will see:
time="2021-06-28T20:46:15Z" level=error msg="Unable to create audit event: events is forbidden: User \"system:serviceaccount:openshift-gitops:openshift-gitops-argocd-application-controller\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-gitops\"" application=guestbook dest-namespace=openshift-gitops dest-server="https://kubernetes.default.svc" reason=ResourceUpdated type=Normal

Additional context

This was originally reported on May 5th on this bug #116, but I have pulled out the behaviour here as the rest of the behaviour in that bug is likely expected behaviour.

Reproduced on 1.1.2 installed from OperatorHub , and AFAICT reproducible on latest.

Operator does not have permissions to watch/create oAuthClient(was working till 1.2.1).

Describe the bug
Operator does not have permissions to watch/create oAuthClient. This seems to have missed during the operator-sdk upgrade(not very sure). This has an impact w.r.t enabling SSO using Keycloak.

E1005 07:42:30.372953       1 reflector.go:138] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1.OAuthClient: failed to list *v1.OAuthClient: oauthclients.oauth.openshift.io is forbidden: User "system:serviceaccount:openshift-operators:gitops-operator-controller-manager" cannot list resource "oauthclients" in API group "oauth.openshift.io" at the cluster scope

To Reproduce
Steps to reproduce the behavior:

1. Push the latest release candidate for 1.3.0_20 to OperatorHub.
2. Install the Operator
3. Uninstall the Dex and Install Keycloak.
4. While the Keycloak Pods gets created and all the configuration is created. Operator fails to create oAuthClient. Which will hamper the process of Login With OpenShift.

Expected behavior
Operator has permissions to manage OAuthClient.

Screenshots
NA

Additional context
NA

GitOps GA version could/should support OCP 4.6

Is your feature request related to a problem? Please describe.
I see that the support statement for the operator (https://access.redhat.com/support/policy/updates/openshift#gitops) only mentions OCP 4.7.

That is somewhat limiting given that OCP 4.6 is a EUS release that will be in many enterprise data-centers for at least another year or so.

Describe the solution you'd like
Assuming there are no technical hurdles (e.g. Argo versions incompatible with OCP 4.6 libraries, etc) , have the GA version of the GitOps operator supported on OCP 4.6.

Describe alternatives you've considered
The alternative is to convince customers in OCP 4.6 to run the GitOps operator in its preview format or wait to adopt the GitOps operator until they migrate to OCP 4.7.

Additional context
Add any other context or screenshots about the feature request here.

GitOps operator installed with incorrect permissions (unable to create resources)

Describe the bug

I've installed the GitOps Operator onto a new, uncustomized install of OpenShift 4.7(.5). I'm trying to deploy a simple webserver consisting of:

• A Namespace
• A Deployment
• A Configmap
• A Service
• A Route

The GitOps operator is able to create to create the Namespace and
the ConfigMap, but is unable to create the remaining resources due
to permission problems. E.g:

• deployments.apps is forbidden: User
  "system:serviceaccount:openshift-gitops:openshift-gitops-argocd-application-controller"
  cannot create resource "deployments" in API group "apps" in the
  namespace "oai-demo"
  

• routes.route.openshift.io is forbidden: User
  "system:serviceaccount:openshift-gitops:openshift-gitops-argocd-application-controller"
  cannot create resource "routes" in API group "route.openshift.io" in
  the namespace "oai-demo"
  

Additionally, in the log for the application controller pod, I also
see:

time="2021-05-05T12:15:09Z" level=error msg="Unable to create audit
event: events is forbidden: User
\"system:serviceaccount:openshift-gitops:openshift-gitops-argocd-application-controller\"
cannot create resource \"events\" in API group \"\" in the namespace
\"openshift-gitops\"" application=oai-demo-webserver
dest-namespace=openshift-gitops
dest-server="https://kubernetes.default.svc" reason=OperationCompleted
type=Warning

It looks as if there are a number of permissions not configured
correctly out of the box.

Expected behavior

I expect the GitOps operator to successfully deploy an application.

Increase the default resource memory requirement for Repo Server.

Describe the bug
From v1.2, All the Argo CD workloads in the default Instance namespace are deployed with resource requests/limits by default. The current resource requests and limits set on the repo server is:

Requests: v1.ResourceList{
	Memory: "256Mi",
        CPU:   "250m",
},
Limits: 
       Memory: "512Mi",
       CPU:    "1000m",

However, many users have reported OOM issues on the repo-server. Looking at the issues I think we would need to upgrade the memory from 512Mi to 1024Mi

To Reproduce
Steps to reproduce the behavior:

1. run the operator from the console or from the CLI using operator-sdk run local
2. Move to Workloads -> Deployment -> openshift-gitops-repo-server and look at the memory limit that is set.

Expected behavior
OOM crash should be avoided.

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.

Update the operator to use Go 1.16

Is your feature request related to a problem? Please describe.

The gitops operator uses an outdated version of Go 1.13

Describe the solution you'd like

Update the gitops operator to use 1.16. By using the latest version of Go, we could also replace statik with an inbuilt Go package embed to embed the Argo CD image during build time and use it in the console link resource.

Reference https://golang.org/pkg/embed/

Unable to install the ArgoCD operator after installing gitops operator

Description:
Unable to install the ArgoCD operator after installing gitops operator

Prerequisite:
Install the GitOps operator

Steps to Reproduce:

1. Select the ArgoCD namespace
2. Navigate to Operator Hub
3. Search and select the "ArgoCD" operator
4. click Install on side pane

Expected:
Install operator page should be displayed with create button enabled

Actual:
Operator Conflicts exists error displays

Default ClusterRole/Binding for argocd-application-controller does not include CRs

Describe the bug
Default ClusterRole/Binding for argocd-application-controller does not include CRs

To Reproduce
Steps to reproduce the behavior:

1. Install openshift-gitops
2. Install an operator
3. Try to sync a repo with CRs
4. See error

Expected behavior
The CRs should be included in the ClusterRole openshift-gitops-openshift-gitops-argocd-application-controller or at least some docs that clarifies how to add them to it so gitops can handle those resources.

Screenshots

Additional context
I have installed the strimzi operator with openshift-gitops, and the strimzi namespace is create automatically by the application that should install the CR.

Error in default clusterrole openshift-gitops-openshift-gitops-argocd-application-controller

Describe the bug
The clusterrole openshift-gitops-openshift-gitops-argocd-application-controller is referring to *.machineconfig.openshift.io instead of *.machineconfiguration.openshift.io

To Reproduce
run oc describe clusterrole openshift-gitops-openshift-gitops-argocd-application-controller | grep machine as an administrator.
Expected behavior
Ability to configure machine config pools and machine configs
Screenshots

# oc describe clusterrole openshift-gitops-openshift-gitops-argocd-application-controller | grep machine
  *.machine.openshift.io                       []                 []              [*]
  *.machineconfig.openshift.io                 []                 []              [*]
# oc api-resources| grep machineconfig
containerruntimeconfigs               ctrcfg                             machineconfiguration.openshift.io/v1          false        ContainerRuntimeConfig
controllerconfigs                                                        machineconfiguration.openshift.io/v1          false        ControllerConfig
kubeletconfigs                                                           machineconfiguration.openshift.io/v1          false        KubeletConfig
machineconfigpools                    mcp                                machineconfiguration.openshift.io/v1          false        MachineConfigPool
machineconfigs                        mc                                 machineconfiguration.openshift.io/v1          false        MachineConfig

Upgrade operator-sdk to v1.6.2

Is your task related to a problem? Please describe.

The version of operator-sdk currently used v0.19 is outdated without a question. We should refresh with the new operator-sdk to
Checkout what is new:
https://sdk.operatorframework.io/docs/building-operators/golang/migration/#what-is-new
and what has changed.
https://sdk.operatorframework.io/docs/building-operators/golang/migration/#what-was-changed

Describe the solution you'd like

To migrate, we can follow these steps.
https://sdk.operatorframework.io/docs/building-operators/golang/migration/

Describe alternatives you've considered

N/A. It is about time to upgrade. :-)

Additional context

chore: remove the internal-objects annotation for gitopsservice

Describe the bug
Currently, A user cannot update/delete gitopsservice(as it is hidden) from the UI. This is needed for the following reasons.

1. v1.3 supports the ability to run all the gitops workloads on Infra Node. This feature should be enabled in the gitopsservice CR. I believe many users would like to do this from the UI/Console.
2. Delete the default Argo CD instance.

Expected behavior

1. Can enable the feature to run workloads on Infra node from UI.
2. Delete the default Argo CD instance from the UI.

Screenshots
NA

Additional context
NA

SSO does not work when global proxy is configured for OCP

Is your feature request related to a problem? Please describe.
Having to extract a admin token from secrets isn't very user-friendly.
It would be better if the argo instance was setup with SSO using the existing oauth server in ocp.

Describe the solution you'd like
SSO using OCP oauth.

Describe alternatives you've considered
none in particular

Additional context
N/A

provide support for 3rd party tls certificates

issue

We would like to replace the operator generated self-signed certificates to signed by a trusted CA type certificates, like the OpenShift builtin solution (https://docs.openshift.com/container-platform/4.7/security/certificates/service-serving-certificate.html) or external application like cert-manager.io.

As far as I understand the crd (deploy/crds/argoproj.io_argocds_crd.yaml) and the argocd creation wizard the only supported method is to give a ca certificate with the key to the operator and it will take care of signing the certs.

This isn't a working method for us because as far as I understand we need to provide a ca cert and key which is trusted by end users browsers which we most likely can't provide as no public cert provider company will hand over a ca with key and of course we can't install our custom ca cert everywhere.

request

Please provide a config parameter for the operator where we can disable the creation and update of the tls secret by the operator so the users can provide the certificate with a 3rd party solution.
It would be good if there will be a parameter to define the name of the secret containing the tls certs in case the 3rd party solution doesn't support the definition of the necessary secret name and the cert manager solution creates the secret with different name than the argocd expects.

operator-sdk upgrade removed the import of openshift package from argocd-operator.

Describe the bug
gitops-operator consumes the openshift package from argocd-operator as a dependency as shown below.

gitops-operator/cmd/manager/main.go

Line 42 in f1df138

_ "github.com/argoproj-labs/argocd-operator/pkg/reconciler/openshift"

Somehow, operator-sdk upgrade removed this line from the main.go file, Which unsets the dependency.

To Reproduce
Steps to reproduce the behavior:
NA

Expected behavior
This is a very important dependency that needs to be added back.

NA

resource.customizations is deleted from argocd-cm configmap

Describe the bug

Applying resource.customizations via argocd-cm configmap only works for a minute or two before being removed from the configmap and returning to having no health check customisations. The resource.customizations key is left in the configmap with an empty value.

I see an earlier description of the same symptom in this issue, though I am seeing the problem with recent versions of OpenShift:

• OpenShift 4.6 + GitOps operator 1.0.0 (Argo CD 1.8.4)
• OpenShift 4.7 + GitOps operator 1.1.0 (ArgoCD 2.0.0)

To Reproduce

(assuming OpenShift 4.7, but exact same behavior in 4.6)

• Deploy the OpenShift GitOps operator (1.1.0 currently) and wait for all pods to be up and running (less than a minute)
• Patch the configmap/argocd-cm object with the sample from https://argo-cd.readthedocs.io/en/stable/operator-manual/health/

health_file=/tmp/argo-health.yaml
cat<<EOF > ${health_file}
data:
  resource.customizations: |
    cert-manager.io/Certificate:
      health.lua: |
        hs = {}
        if obj.status ~= nil then
          if obj.status.conditions ~= nil then
            for i, condition in ipairs(obj.status.conditions) do
              if condition.type == "Ready" and condition.status == "False" then
                hs.status = "Degraded"
                hs.message = condition.message
                return hs
              end
              if condition.type == "Ready" and condition.status == "True" then
                hs.status = "Healthy"
                hs.message = condition.message
                return hs
              end
            end
          end
        end

        hs.status = "Progressing"
        hs.message = "Waiting for certificate"
        return hs
EOF

kubectl patch configmap/argocd-cm \
            --namespace openshift-gitops \
            --type merge \
            --patch-file "${health_file}"

sleep 10

kubectl get configmap/argocd-cm --namespace openshift-gitops -ojsonpath='{.data.resource\.customizations}'
<empty line>

Expected behavior

kubectl get configmap/argocd-cm --namespace openshift-gitops -ojsonpath='{.data.resource\.customizations}'

should return the resource customization applied as a patch.

Screenshots

Version

argocd: v2.0.0+unknown
  BuildDate: 2021-04-09T04:17:22Z
  GitCommit: 
  GitTreeState: clean
  GoVersion: go1.15.7
  Compiler: gc
  Platform: linux/amd64
argocd-server: v2.0.0+unknown
  BuildDate: 2021-04-09T04:17:22Z
  GitTreeState: clean
  GoVersion: go1.15.7
  Compiler: gc
  Platform: linux/amd64
  Ksonnet Version: unable to determine ksonnet version: exec: "ks": executable file not found in $PATH
  Kustomize Version: unknown 1970-01-01T00:00:00Z
  Helm Version: v3.5.0+6.el8+g77fb4bd
  Kubectl Version: v0.20.4
  Jsonnet Version: v0.17.0

Logs

logs.zip

Upgrade to argocd v2.1

Is your feature request related to a problem? Please describe.
argoproj/argo-cd#5505 has some memory problems, these are fixed in 2.1.x

Describe the solution you'd like
Upgrade argocd to 2.1 by default in the operator.

Describe alternatives you've considered
N/A

Additional context
argoproj/argo-cd#5505

ArgoCD 2.0 support?

Since ArgoCD finally released their 2.0.0 version (https://github.com/argoproj/argo-cd/releases/tag/v2.0.0), is there any roadmap or a rough timeline to incorporate it into the GitOps Operator? An info about a target month or quarter would be sufficient for me.

v1.2: OOMKilled of openshift-repo-server after upgrade from 1.1.2

After upgrading from gitops operator 1.1.2 to 1.2.0, the repo-server crashloops with OOM ...

Removing the namespace resourcequota did not help.

Configure ArgoCD during deployment

Hello,

If I understand correctly the way the GitOps Operator works, the post-configuration of ArgoCD such as authentication with OIDC has to be done after the installation of the Operator itself through patching the ArgoCD onject. Is there a way to push the ArgoCD configuration while instancianting the GitOps Operator? I don't see much information in the GitOpsService CRDs.

The goal of this is to automate the installation of OCP + GitOps and avoid any post-install patching.

Thanks!

e2e tests fails running locally

Describe the bug
e2e tests fails locally. Operator when run locally is not able to create workloads for ArgoCD instances created to manage Cluster Configuration.

Reported and fixed in upstream:
argoproj-labs/argocd-operator#340

To Reproduce
Steps to reproduce the behavior:
Run operator locally

1. make test-e2e
2. log the operator -> oc logs deploy/gitops-operator -n openshift-operators

You will find the below error in logs

:"failed to initialise kube client","ArgoCD Namespace":"openshift-gitops","ArgoCD Name":"openshift-gitops","error":"unable to load in-cluster configuration, KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT must be defined",

Expected behavior
e2e tests should pass

Additional context
NA

Application-scoped Argo instance fails to sync when a managed namespace is deleted

Describe the bug
When using an "Application-scoped" argocd (e.g. one that is not scoped to All Namepsaces in a cluster), the GitOps operator dynamically maintains a list of "managed namespaces", which are namespaces that include the argocd.argoproj.io/managed-by label. This list of namespaces gets cached in argocd (Settings > Clusters > "in-cluster" > NAMESPACES). However, if one of these namespaces gets deleted, the operator doesn't seem to notice, and the namespace remains in that list. This breaks both the "CONNECTION STATE" in that cluster config, and also all Sync attempts after the namespace gets deleted. The sync will show errors like <argo service account> does not have <random verb> on <random resource type> in namespace "the-namespace-you-deleted".

To Reproduce
Steps to reproduce the behavior:

1. Deply an application scoped ArgoCD via the CR:

   apiVersion: argoproj.io/v1alpha1
   kind: ArgoCD
   metadata:
    name: argocd #name of the Argo CD instance
    namespace: foo #namespace where you want to deploy argocd instance
   spec:
    server:
      route:
        enabled: true #creates an openshift route to access Argo CD UI
   

2. Create two new namespaces and label them

   kubectl create namespace to-delete
   kubectl label namespace/to-delete argocd.argoproj.io/managed-by=foo
   kubectl create namesoace sample-app
   kubectl label namespace/sample-app argocd.argoproj.io/managed-by=foo
   

3. Delete the first namespace

   kubectl delete namepsace/to-delete
   

4. Apply an application to second namespace

   apiVersion: argoproj.io/v1alpha1
   kind: Application
   metadata:
     name: sample-app #app CR name
     namespace: foo #argocd instance namespace
   spec:
     destination:
       namespace: sample-app #namespace where app is deployed
       server: 'https://kubernetes.default.svc'
     source:
       path: app
       repoURL: 'https://github.com/redhat-developer/openshift-gitops-getting-started'
       targetRevision: HEAD
     project: default
   

5. Observe the following errors:

6. Continual sync errors under App Conditions for the sample app, complaining of permissions errors for the to-delete namespace

7. Connection errors under the cluster settings

Expected behavior
The namespace should be deleted from the list of manages namespaces in the cluster settings, and the app should sync successfully.

Dex service account destroyed by operator

Describe the bug
When configuring Openshift SSO on an ArgoCD instance (with dex), the operator fails to reconcile with the following error (service xxx-argocd-dex-server account not found) :

{"level":"error","ts": xxx,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"argocd-controller","request":"openshift-gitops/xxx","error":"ServiceAccount \"xxx-argocd-dex-server\" not found", ...

If I create manually the desired service account, it disappears - seems deleted by the operator when "reconciling service accounts"

To Reproduce

Patch the ArgoCD instance with the following file :

spec:
  dex:
    image: quay.io/redhat-cop/dex
    openShiftOAuth: true
    version: v2.22.0-openshift
  rbac:
    defaultPolicy: 'role:readonly'
    policy: |
      g, system:cluster-admins, role:admin
    scopes: '[groups]'

oc patch argocd argocd-cluster -n openshift-gitops --type merge --patch "$(cat argocd-openshift-sso-patch.yaml)"

Additional context

Cluster version : OKD 4.7.0-0.okd-2021-03-07-090821
Operator image : registry.redhat.io/openshift-gitops-1-tech-preview/gitops-rhel8-operator:v1.0.1-9

Version check does not work for installed operator-sdk

https://github.com/redhat-developer/gitops-operator/blob/master/scripts/run_e2e_tests.sh#L19

$ ./scripts/run_e2e_tests.sh 
+ E2E_TEST_NS=gitops-test
+ E2E_TEST_DIR=./test/e2e
+ ARGOCD_NS=openshift-gitops
+ DEPRACATED_ARGOCD_NS=openshift-pipelines-app-delivery
+ CONSOLE_LINK=argocd
+ echo 'Checking if operator-sdk is installed'
Checking if operator-sdk is installed
+ command -v operator-sdk
++ operator-sdk version
++ grep -Po '[0-9][^.]+'
++ head -1
usage: grep [-abcDEFGHhIiJLlmnOoqRSsUVvwxZ] [-A num] [-B num] [-C[num]]
        [-e pattern] [-f file] [--binary-files=value] [--color=when]
        [--context[=num]] [--directories=action] [--label] [--line-buffered]
        [--null] [pattern] [file ...]

[Deprecation Notice] Operator SDK has a new CLI and project layout that is aligned with Kubebuilder.
See `operator-sdk init -h` and the following doc on how to scaffold a new project:
https://v0-19-x.sdk.operatorframework.io/docs/golang/quickstart/
To migrate existing projects to the new layout see:
https://sdk.operatorframework.io/docs/building-operators/golang/project_migration_guide/

+ operator_sdk_version=
+ '[' -gt 17 ']'
./scripts/run_e2e_tests.sh: line 19: [: -gt: unary operator expected

+ oc new-project gitops-test
[...]

update the deprecated APIs

Describe the bug
kubernetes 1.22 is End of life for deprecated APIs.

The apiextensions.k8s.io/v1beta1 API version of CustomResourceDefinition will no longer be served in v1.22.

More information here:
https://kubernetes.io/blog/2021/07/14/upcoming-changes-in-kubernetes-1-22/#api-changes

To Reproduce
I was trying to install the GitOps 1.2 operator on a 4.9 cluster, but it failed to find one or more requirements:
api-server resource not found installing CustomResourceDefinition gitopsservices.pipelines.openshift.io: GroupVersionKind apiextensions.k8s.io/v1beta1, Kind=CustomResourceDefinition not found on the cluster. This API may have been deprecated and removed, see https://kubernetes.io/docs/reference/using-api/deprecation-guide/ for more information.

Expected behavior
update the deprecated APIs

Additional context
https://kubernetes.io/docs/reference/using-api/deprecation-guide/#v1-22

e2e tests fails intermittently due to timeouts.

Describe the bug
The default timeout for go tests is 10 mins. This may lead to test suite failure when tests run for more than 10 mins.

To Reproduce
Steps to reproduce the behavior:

1. Run the operator locally using make install && make run

Expected behavior
Timeout should not be 10 mins, Increase the timeout.

Additional context

(...)
2021-09-28T10:48:19.142+0200	INFO	controller_gitopsservice	Reconciling GitopsService	{"Request.Namespace": "", "Request.Name": "cluster"}
2021-09-28T10:48:19.142+0200	INFO	controller_gitopsservice	No ResourceQuota set for namespace	{"Request.Namespace": "", "Request.Name": "cluster", "Name": "openshift-gitops"}
coverage: [no statements]
panic: test timed out after 10m0s

Update opencontianers/runc to fix vulnerabilities reported by dependabot

Describe the bug
Dependabot reported a high severity vulnerability with an indirect dependency opencontainers/runc. This can be addressed by upgrading the version of runc to v1.0.0-rc95

gitopsservice is not namespaced, but the console requests api on openshift-github

Describe the bug
The gitops service is not namespaced. Sample once installed:

apiVersion: pipelines.openshift.io/v1alpha1
kind: GitopsService
metadata:
  creationTimestamp: "2021-04-03T20:55:40Z"
  generation: 1
  managedFields:
  - apiVersion: pipelines.openshift.io/v1alpha1
    fieldsType: FieldsV1
    fieldsV1:
      f:spec: {}
      f:status: {}
    manager: gitops-operator
    operation: Update
    time: "2021-04-03T20:55:40Z"
  name: cluster
  resourceVersion: "35378"
  selfLink: /apis/pipelines.openshift.io/v1alpha1/gitopsservices/cluster
  uid: d7520894-fa57-40c7-a939-fc01c37e7e10
spec: {}

However the console considers it namespace for searching at:

https://console-openshift-console.apps-crc.testing/api/kubernetes/apis/pipelines.openshift.io/v1alpha1/namespaces/openshift-gitops/gitopsservices?limit=250

Instead of

https://console-openshift-console.apps-crc.testing/api/kubernetes/apis/pipelines.openshift.io/v1alpha1/gitopsservices?limit=250

To Reproduce
Steps to reproduce the behavior:

1. Install an OpenShift 4.7.2 (I have used CRC)
2. Install openshift-gitops operator
3. After the install finishes ok, go to gitops services or all instances tab
4. See a 404 error

Expected behavior
The instances on those resources displayed

Screenshots

Additional context

Allow default ArgoCD instance to manage ResourceQuota, LimitRanges and CRDs.

Is your feature request related to a problem? Please describe.
I would like to to create ResourceQuota and LimitRange objects for the namespaces I create with default Argo CD instance also I would like to manage customresourcedefinitions to install Sealed Secrets or Kubernetes External Secrets.

So can we add the below permissions to the existing one's of default Argo CD instance.
ResourceQuotas
LimitRanges
CRDs

Describe alternatives you've considered
NA

Additional context
Currently users are creating a new role and rolebinding to provide the above permissions.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    argocd.argoproj.io/sync-options: Prune=false
  name: openshift-gitops-argocd-extended-clusterrole
rules:
  - apiGroups:
    - ""
    resources:
      - resourcequotas
      - limitranges
      # - serviceaccounts
    verbs:
      - '*'
  - apiGroups:
    - argoproj.io
    resources:
      - argocds
    verbs:
      - '*'
  - apiGroups:
    - bitnami.com
    resources:
      - sealedsecrets
    verbs:
      - '*'
  - apiGroups:
    - apiextensions.k8s.io
    resources:
      - customresourcedefinitions
    verbs:
      - '*'

Support for custom tools

Is your feature request related to a problem? Please describe.
I'd like to find a way to add custom tools to Argo CD when using this operator.
Describe the solution you'd like
In Argo CD one can add extra custom tools as described here: https://argoproj.github.io/argo-cd/operator-manual/custom_tools/
I'd like to be able to do that when using the Red Hat operator version of Argo CD

Describe alternatives you've considered
Either done by configuration at install time or be providing the ability to change path to the repo image to point to a customized image

Document steps to create, update and delete resource requests/limits for Argo CD workloads.

Describe the bug
Many users have reported the OOM issues on the Argo CD instance created by the gitops-operator. Most of these issues are on repo-server and application-controller. The issue can easily be fixed by updating the resource limits of the specific workload that failed.

The resource requests/limits can be updated for any or all the workloads by updating the Argo CD CR. However, there is no proper documentation which explains the steps to perform this.

Please create a well explained documentation to fix this issue.

Error in README instruction

There is an error in the instructions in the README.

On applying the gitops catalog source given in the README instructions we encounter Invalid value error.

The name Gitops-service-source should not have uppercase character G

➜ cat gitops.yaml                  
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: Gitops-service-source
  namespace: openshift-marketplace
spec:
  displayName: 'Gitops Service by Red Hat'
  image: 'quay.io/redhat-developer/gitops-backend-operator-index:v0.0.1'
  publisher: 'Red Hat Developer'
  sourceType: grpc

➜ k apply -f gitops.yaml
The CatalogSource "Gitops-service-source" is invalid: metadata.name: Invalid value: "Gitops-service-source": a DNS-1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')