Key leanings from GitOps Enterprise: Learn how to use Argo CD in multi-tenant installations, how to create preview environments and more.

Categories

At first, it is important to differentiate the possible scenarios, or as the course calls it categories:

Category 1

Category 1 change very often as due to new releases

• Updating the container image version on the deployment manifest (maybe 80% of cases)
• Updating the container image AND some kind of configuration in a configmap or secret (maybe 15% of cases)
• Updating only the configuration to fine-tune a business or technical property (maybe 5% of the cases)

There is an artictle How to Model Your GitOps Environments and Promote Releases between Them which specifically talks about these

Category 2

As these reference the developer manifests (applications), which define the proper state, the manifests in Category 2 should be defined once and then forget about it.

Category 3 and 4

Same as category 1 or 2 respectively, only focusing on infrastructure applications.

• Developers usually don’t care about infrastructure manifests
• Usually change when there is an update of the infrastructure application or fine-tune the configuration

Anti Patterns

Anti-pattern 1 — Mixing different types of manifests

Keep in mind, an application shall be installable locally using kustomize or helm, without ArgoCD in place. If that’s possible then you are probably doing it right.

So even so ArgoCD offers the ability to have different types of manifest as part of your Application/Application Set, you shall maintain a very clear separation between Kubernetes (category 1) and Argo CD resources (category 2).

As part of source you can have helm or kustomize. Here some rules

• helm

  • Do not mix parameters, values, valuesObject, and valuesFiles.
  • Preferable use valuesFiles and reference the values file in the git repo
  • Do not use parameter overrides.
  • Use Helm Umbrella charts that allow you to both reference other charts and override specific values on them.

• kustomize

  • Save your Kustomize values as an overlay instead of hardcoding them in the Application CRD

Anti-pattern 2 – Working at the wrong abstraction level

Following Anti-pattern 1, remember that Argo CD application CRDs should only point to the main Kubernetes manifests/helm charts + value files. So Application CRD shall be static and there should not be a need to even change TargetRevision. Changes should always happen on the underlying Kubernetes manifests. Thus also enables a clear history of events for what an application did.

Anti-pattern 3 – Using templating at different/multiple levels

Templating of Application CRDs should not be used. A classical example would be using a helm chart containing Application CRDs which themselves point to the Helm charts of the Kubernetes manifests. If templating is necessary, use Application Sets instead.

Anti-pattern 4 – Not using Application Sets

ApplicationSets allow you to generate multiple Applications CRDs. This scales easily if you increase the clusters for example.

Best practice – Use the three-level structure

The starting point should be a 3-level structure as shown in the image below

• Lowest level we have the Kubernetes manifests that define how the application runs (category 1 of manifests)
• One level above, we have the Application Set as explained in the previous section. These wrap the main Kubernetes manifests into Argo CD applications (category 2 of manifests).
• Last, as an optional component you can group all your application sets in an App-of-App that will help you bootstrap a completely empty cluster with all apps.

Notice how simple this pattern is:

• There are only 3 levels of abstraction.
• Each level is completely independent of everything else.
• Helm and Kustomize are only used once at the Kubernetes manifests and nowhere else.

So let’s have a look at https://github.com/kostis-codefresh/many-appsets-demo

├── apps                <- Level 3
├── appsets             <- Level 2
├── docs
├── README.md
└── root-argocd-app.yml <- Level 1

If we look into more details, we can see each app has a folder under apps (apps/<name-of-app>/envs/<name-of-env>) containing the manifests:

├── apps
│   ├── billing
│   │   ├── base
│   │   │   ├── deployment.yml
│   │   │   ├── kustomization.yml
│   │   │   └── service.yml
│   │   └── envs
│   │       ├── prod-eu
│   │       │   ├── deployment.yml
│   │       │   ├── kustomization.yml
│   │       │   ├── replicas.yml
│   │       │   ├── settings.yml
│   │       │   └── version.yml
│   │       └── prod-us
│   │           ├── deployment.yml
│   │           ├── kustomization.yml
│   │           ├── replicas.yml
│   │           ├── settings.yml
│   │           └── version.yml
│   ├── fake-invoices
│   │   ├── base
│   │   │   ...
│   │   └── envs
│   │       └── qa
│   │           ...
│   ├── invoices
│   │   ├── base
│   │   │   ...
│   │   └── envs
│   │       ├── prod-eu
│   │       │   ...
│   │       ├── prod-us
│   │       │   ...
│   │       ├── qa
│   │       │   ...
│   │       └── staging
│   │           ...
│   ├── orders
│   │   ...
│   └── payments
│       ...
...

In the appsets folder, we keep all our application sets:

├── appsets
│   ├── my-prod-appset.yml
│   ├── my-qa-appset.yml
│   └── my-staging-appset.yml

If you check the structure in details you will see

• Deploy an existing application to a new environment -> Create a new Kustomize overlay. No Argo CD changes are needed.
• Remove an application from an environment -> Delete the respective Kustomize overlay. No Argo CD changes are needed.
• Create a brand new application -> Commit the K8s manifests in a new folder under “apps”. No Argo CD changes are needed.
• Create a new environment called “integration” based on the qa one -> Copy/modify the qa application set to a new file for “integration”. In the next sync Argo CD will create the new combinations for all applications that have an “integration” overlay.
• Add a new cluster -> Connect the cluster to Argo CD and all applicationsets that reference it will automatically deploy their applications to that cluster as well.
• Move a cluster to a different environment -> Simply add/edit a new label on the cluster for the respective application set.

Organizing of code

As ArgoCD does not deal with source code, so some concepts/techniques do not make sense. Following the anti-pattern, hence it’s also important to separate source code and deployment manifests.

Considering that, one is tempted to move all manifests into a single code repository. This approach suffers from several scalability and performance issues as the number of Argo CD applications grows

it has several limitations not only in the way that Argo CD detects commits but also how the Git repository deals with conflicts and retries from all the different workflows that interface with the Git repository as a deployment source.

The recommendation is to have multiple Git repositories. Ideally one for each team or each department plus one for infrastructure. Also important is, that you don’t mix infrastructure applications and developer applications in the same Git repo.

Preview environments

Following Kubernetes Deployment Antipatterns the best way to deal with changes, is to have dynamic environments for testing, so a developer can test their new features in isolation. Argo CD offers a way to create dynamic environments by using the Pull Request Generator. It can monitor any Git repository for pull requests and for any feature (in parallel) the following process applies

1. A developer creates a branch on the source Git repository
2. The developer creates a pull request on the branch (usually pending tests and feedback from their team).
3. Argo CD notices the pull request on the source code Git repository and creates a new application for it.
4. The Pull Request Generator takes the manifests from the infrastructure repository and templates/modifies them according to the pull request in the source code repo.
5. A new application gets created and deployed in the Kubernetes cluster.
6. The developer accesses the new application, reviews it, runs unit tests, shares it, etc.
7. When the pull request gets merged/closed, Argo CD automatically discards the preview environment.

There are also some caveats with this approach

• It depends on your ci tooling, basically each PR shall create a docker image that is pushed to the registry and tagged with the Git hash of the source code.
• Preview environment only work for a single application (the one that has the pull request). If a preview environment shall contain different applications from different Git repositories, then the scenario is not possible.
• Preview environments work for updating new container tags, but if the configuration also changes (e.g. new input parameter), then the scenario also does not work.

This is something I want to dig deeper but for now it’s fine to know that the possibility exist.

Promoting to different GitOps environments

About helm charts

When using helm there are basically two approaches to choose from - your company should focus on a single one and not mix them:

1. Have a Helm chart for every application you develop. In this case the Helm chart has 2 versions. One version keeps history about the Chart itself and the second version (appVersion in the Chart.yml file) keeps track of the application contained in the Chart.
2. Have a reusable “single” chart that is targeted at many applications. It doesn’t hold a specific application on its own. You always need to combine it with values that define the image/tag/configuration.

Use hierarchial value files

When it comes to define environment specific values, we shall use the values hierarchy supported by Helm: Have multipe value files (from common values to environment specific), which are then combined by helm. This way you don’t duplicate your configuration multiple times. You can always override values in the lower level:

├── values
│   └── common-values.yam
│       └── all-prod-envs.yaml
│           └── specific-prod-cluster.yaml

As for the helm cli, the same applies for Argo CD: Provide the value files in expected order and remember, the last one wins. So in an Application CRD you would specify

...
    helm:
      valueFiles:
      - common-values.yaml
      - all-prod-envs.yaml
      - specific-prod-cluster.yaml

Helm charts in git vs. helm charts in registry

It’s recommended to consume helm charts from git rather registries, mainly due to the following reasons

• Helm repositories (and OCI artifacts) are mutable. This means that you might download Helm chart 1.2.3 today with different contents than what you downloaded yesterday as version 1.2.3
• There is no built-in auditing or history for changes that happened in the chart. If you use Git you get these features for “free”
• If you use your own Git repo, you can monitor and optionally approve all changes that happen on the chart, which improves security and integrity.

However - especially for infrastructure applications - you probably rely on charts from the registry. Then use the multi source feature with at least 2 sources

• One contains your external Helm chart
• Second one contains your Helm values.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: application_name
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
  - list:
      elements:
      - cluster: a
        clusterid: a
        targetRevision: x.x.x.x
      - cluster: b
        clusterid: b
        targetRevision: x.x.x.x

  template:
    metadata:
      name: application_name-{{.env}}
      namespace: argocd
    spec:
      destination:
        namespace: application_name
        name: "{{.cluster}}"
      project: infrastructure
      sources:
      - repoURL: https://scm.example.com/infrastructure.git
        targetRevision: HEAD
        ref: values
      - repoURL: https://helm.example.com/
        targetRevision: "{{.targetRevision}}"
        chart: application_name
        helm:
          releaseName: application_name
          valueFiles:
          - $values/application_name/{{.env}}/values.yaml
          - $values/application_name/values.yaml

The above example shows an ApplicationSet, but same is also possible for Applications. What we usually also do is make the targetRevision of the helm chart parametrized, so combined with the generators list we can individually update the helm chart per cluster.

Saving environment configurations on their own files

The generator list in application sets can become very long, especially the more enviornment you have. It may be better to extract this useful information in individual files by using the Git file generator:

  generators:
  - git:
      repoURL: https://github.com/kostis-codefresh/multi-sources-example.git
      revision: HEAD
      files:
      - path: "appsets/4-final/env-config/**/config.json"

appsets/4-final/env-config/prod/us/config.json would look like this:

{
    "env": "prod-us",
    "region": "us",
    "type": "prod",
    "version": "prod",
    "chart": "0.1.0"
}

That’s it for now, there are some more stuff to come.