In this article, you will learn how to manage multiple clusters with Argo CD and Advanced Cluster Management for Kubernetes. Advanced Cluster Management (ACM) for Kubernetes is a tool provided by Red Hat based on a community-driven project Open Cluster Management. I’ll show you how to use it with OpenShift to implement gitops approach for running apps across multiple clusters. However, you can as well deploy a community-driven version on Kubernetes.

If you are not familiar with Argo CD you can read my article about Kubernetes CI/CD with Tekton and ArgoCD available here.

Prerequisites

To be able to run that exercise you need to have at least two OpenShift clusters. I’ll run my both clusters on cloud providers: Azure and GCP. The cluster running on Azure will act as a management cluster. It means that we will install there ACM and Argo CD for managing both local and remote clusters. We can easily do it using operators provided by Red Hat. Here’s a list of required operators:

After installing the Advanced Cluster Management for Kubernetes operator we also need to create the CRD object MultiClusterHub. Openshift Console provides simplified way of creating such objects. We need to go to the operator details and then switch to the “MultiClusterHub” tab. There is the “Create MultiClusterHub” on the right corner of the page. Just click it and then create object with the default settings. Probably it takes some time until the ACM will be ready to use.

ACM provides dashboard UI. We can easily access it through the OpenShift Route (object similar to Kubrnetes Ingress). Just switch to the open-cluster-management namespace and display a list of routes. The address of dashborad is https://multicloud-console.apps.<YOUR_DOMAIN>.

The OpenShift GitOps operator atomatically creates the Argo CD instance during installation. By default, it is located in the openshift-gitops namespace.

The same before, we can access Argo CD dashboard UI through the OpenShift Route.

Add a Remote Cluster

In this section, we will use ACM dashboard UI for importing an existing remote cluster. In order to do that go to the to the “Clusters” menu item, and then click the “Import cluster” button. There are three different modes of importing a remote cluster. We will choose the method of entering server URL with API token. The name of our cluster on GCP is remote-gcp. It will act as a production environment, so we will add the env=prod label.

You can get server URL and API token from your OpenShift login command:

After importing a cluster we can display a list of managed clusters in the UI. The local-cluster is labelled with env=test, while the remote-gcp with env=prod.

All the clusters are represented by the CRD objects ManagedCluster. Let’s display them using the oc CLI:

$ oc get managedcluster              
NAME            HUB ACCEPTED   MANAGED CLUSTER URLS                                     JOINED   AVAILABLE   AGE
local-cluster   true           https://api.zyyrwsdd.eastus.aroapp.io:6443               True     True        25h
remote-gcp      true           https://api.cluster-rhf5a9.gcp.redhatworkshops.io:6443   True     True        25h

We can manage clusters individually or organize them as groups. The ManagedClusterSet object can contain many managed clusters. Our sample set contains two clusters used in this article: local-cluster and remote-gcp. Firstly, we need to create the ManagedClusterSet object.

apiVersion: cluster.open-cluster-management.io/v1alpha1
kind: ManagedClusterSet
metadata:
  name: demo
spec: {}

Then we need to label managed cluster with the cluster.open-cluster-management.io/clusterset label containing the name of ManagedClusterSet it should belong to. A single ManagedCluster can be a member of only a single ManagedClusterSet.

$ oc label managedcluster remote-gcp cluster.open-cluster-management.io/clusterset=demo
$ oc label managedcluster local-cluster cluster.open-cluster-management.io/clusterset=demo

Integrate ACM with Argo CD

In order to integrate Advanced Cluster Management for Kubernetes with OpenShift GitOps we need to create some CRD objects. In the first step we need to bind managed clusters to the target namespace where Argo CD is deployed. The ManagedClusterSetBinding object assigns ManagedClusterSet to the particular namespace. Our target namespace is openshift-gitops.

apiVersion: cluster.open-cluster-management.io/v1beta1
kind: ManagedClusterSetBinding
metadata:
  name: demo
  namespace: openshift-gitops
spec:
  clusterSet: demo

After that, we need to create the Placement object in the same namespace. Placement determines which managed clusters each subscription, policy or other definition affects. It can filter by the cluster sets or other predicates like, for example, labels. In our scenario, placement filters all the managed clusters included in the demo managed set, which have the env label with the test or prod value.

apiVersion: cluster.open-cluster-management.io/v1beta1
kind: Placement
metadata:
  name: demo-gitops-placement
  namespace: openshift-gitops
spec:
  clusterSets:
    - demo
  predicates:
    - requiredClusterSelector:
        labelSelector:
          matchExpressions:
            - key: env
              operator: In
              values:
                - test
                - prod

Let’s verify if the object has been succesfully created. Our placement should filter out two managed clusters.

$ oc get placement -n openshift-gitops
NAME                    SUCCEEDED   REASON                  SELECTEDCLUSTERS
demo-gitops-placement   True        AllDecisionsScheduled   2

Finally, we can create the GitOpsCluster object. It assigns managed clusters to the target instance of Argo CD basing on the predicates defined in the demo-gitops-placement Placement.

apiVersion: apps.open-cluster-management.io/v1beta1
kind: GitOpsCluster
metadata:
  name: demo-gitops-cluster
  namespace: openshift-gitops
spec:
  argoServer:
    cluster: local-cluster
    argoNamespace: openshift-gitops
  placementRef:
    kind: Placement
    apiVersion: cluster.open-cluster-management.io/v1beta1
    name: demo-gitops-placement

If everything worked fine you should see the following status message in your GitOpsCluster object:

$ oc get gitopscluster demo-gitops-cluster -n openshift-gitops \
    -o=jsonpath='{.status.message}'
Added managed clusters [local-cluster remote-gcp] to gitops namespace openshift-gitops

Now, we can switch to the Argo CD dashboard. We don’t have any applications there yet, but we just want to verify a list of managed clusters. In the dashboard choose the “Settings” menu item, then go to the “Clusters” tile. You should see the similar list as shown below.

Now, we can manage application deployment across multiple OpenShift clusters with Argo CD. Let’s deploy our first app there.

Deploy App Across Multiple Clusters

As the example we will that GitHub repository. It contains several configuration files, but our deployment manifests are available inside the apps/simple directory. We will deploy a simple Spring Boot app from my Docker registry. Here’s our deployment manifest:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sample-spring-kotlin
spec:
  selector:
    matchLabels:
      app: sample-spring-kotlin
  template:
    metadata:
      labels:
        app: sample-spring-kotlin
    spec:
      containers:
      - name: sample-spring-kotlin
        image: piomin/sample-spring-kotlin:1.4.8
        ports:
        - containerPort: 8080
          name: http

There is also Kubernetes Service definition in this directory:

apiVersion: v1
kind: Service
metadata:
  name: sample-spring-kotlin
spec:
  type: ClusterIP
  selector:
    app: sample-spring-kotlin
  ports:
  - port: 8080
    name: http

For generating Argo CD applications for multiple clusters we will use a feature called Cluster Decision Resource Generator. It is one of available generator in Argo CD ApplicationSet project. After we created the Placement object, ACM has automatically created the following ConfigMap in the openshift-gitops namespace:

kind: ConfigMap
apiVersion: v1
metadata:
  name: acm-placement
  namespace: openshift-gitops
data:
  apiVersion: cluster.open-cluster-management.io/v1beta1
  kind: placementdecisions
  matchKey: clusterName
  statusListKey: decisions

The ApplicationSet generator will read the kind placementrules with an apiVersion of apps.open-cluster-management.io/v1. It will attempt to extract the list of clusters from the key decisions (1). Then, it validates the actual cluster name as defined in Argo CD against the value from the key clusterName in each of the elements in the list. The ClusterDecisionResource generator passes the name, server and any other key/value in the resource’s status list as parameters into the ApplicationSet template. Thanks to that, we can use those parameters to set the name of Argo CD Application (2) and a target cluster (3). The target namespace for our sample app is demo. Let’s create it on all clusters automatically with Argo CD (4).

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: sample-spring-boot
  namespace: openshift-gitops
spec:
  generators:
    - clusterDecisionResource:
        configMapRef: acm-placement # (1)
        labelSelector:
          matchLabels:
            cluster.open-cluster-management.io/placement: demo-gitops-placement
        requeueAfterSeconds: 180
  template:
    metadata:
      name: sample-spring-boot-{{name}} # (2)
    spec:
      project: default
      source:
        repoURL: https://github.com/piomin/openshift-cluster-config.git
        targetRevision: master
        path: apps/simple
      destination:
        namespace: demo
        server: "{{server}}" # (3)
      syncPolicy:
        automated:
          selfHeal: false
        syncOptions:
          - CreateNamespace=true # (4)

Once you created the ApplicationSet obejct Argo CD will create Applications basing on the managed clusters list. Since, there are two clusters managed by Argo CD we have two applications. Here’s the current view of applications in the Argo CD dashboard. Both of vthem are automatically synchronized to the target clusters.

Also, we can switch to the ACM dashboard once again. Then, we should choose the “Applications” menu item. It displays the full list of all running applications. We can filter the application by the name. Here’s a view for our sample-spring-boot application.

We can see the details of each application.

We can also see the topology view:

And for example, display the logs of pods from the remote clusters from a single place – ACM dashboard.

Final Thoughts

Many organizations already operate multiple Kubernetes clusters in multiple regions. Unfortunately, operating a distributed, multi-cluster, multi-cloud environment is not a simple task. We need to use right tools to simplify it. Advanced Cluster Management for Kubernetes is a Red Hat proposition of such tool. In this article, I focused on the showing you how to apply GitOps approach for managing application deployment across several clusters. With RedHat you can integrate Argo CD with ACM to manage multiple clusters following GitOps pattern. OpenShift organizes that process from the beginning to the end. I think it shows the added value of OpenShift as an enterprise platform in your organizarion in comparison to the vanilla Kubernetes.