Azure AD Workload Identity allows you the use of a Managed Identity to access resources in Azure

What is Azure AD Workload Identity?

Azure AD Workload Identity allows you the use of a Managed Identity to access resources in Azure

Azure AD Workload Identity for Kubernetes integrates with the capabilities native to Kubernetes to federate with external identity providers

The kubernetes cluster becomes a token issuer, which issues tokens to Kubernetes Service Accounts. These service account tokens can be configured to be trusted on Azure AD applications or user-assigned managed identities.

Following the Quick Start shows you how to setup and test it. However, why not make it a bit more concrete and use Workload Identity with External-Secrets

What is External-Secrets Operator and why do we need it?

Kubernetes secrets contain sensitive information that should be kept securely, such as passwords, API keys, tokens. Without secrets, it is difficult to use Kubernetes for most environments and workloads:

• Backend applications might need access to the database to process and store data
• Frontend applications might require API keys to integrate with third-party APIs, and the Kubernetes cluster might need access to pull container images from private container registries

Usually secrets are stored directly in Kubernetes as secrets:

A Secret is an object that contains a small amount of sensitive data such as a password, a token, or a key.

External Secrets Operator is a Kubernetes operator that integrates external secret management systems like AWS Secrets Manager, HashiCorp Vault, Google Secrets Manager, Azure Key Vault and much more. This allows you to manage the secrets outside of Kubernetes, directly in these systems - with adequate access control - while the External Secrets Operator provide them to the workloads running on Kubernetes.

Benefits of Using External Secret Operators are clear:

• Enhanced Security
  • Secrets are not stored directly in the Kubernetes cluster
  • External secret managers provide advanced security features like automatic rotation, access logging, and encryption
• Centralized Secret Management
  • Centralized approach simplifies secret management across multiple clusters and environments
  • Consistent policies and practices can be enforced
    • Principle of Least Privilege: Grant minimal access necessary to secrets
    • Regular Audits and Monitoring: Periodically review secret access and usage
    • Use of Kubernetes RBAC: Leverage Kubernetes Role-Based Access Control to manage access to secrets
    • Automatic Secret Rotation: Implement policies for automatic rotation of secrets to minimize risks
• Scalability and Flexibility
  • Easily manage secrets for large-scale, dynamic environments.
  • Supports multiple secret backends, allowing for flexibility in choosing the right tool for the job.

Setup for Multi Tenancy

When deploying External Secrets Operator it is important to understand the organizational needs and then set it up according to the guidelines on Multi Tenancy, In our case we decided to use a SecretStore per Namespace

This means the administrator has to provide a secretStore per namespace. While this ensure that developers only can see their secrets and not all secrets, it is more complex to handle, as for each namespace we need a secretStore and the respective credentials.

As we are using Rancher and it’s Rancher Projects our setup forsees, that we have one SecretStore per project, means the namespaces belonging to a project share the same. Still we have to deploy the SecretStore to each ns.

So basically what do we need for a team to use

1. a keyvault in azure -> once per project, setup by admins
2. a managed identity, which can read from the keyvault -> once per keyvault, setup by admins
3. a SecretStore -> for each namespace, using MI_clientId from step 2
4. a ServiceAccount which will be “federated” with the managed idendity -> for each namespace
5. a federated credential -> for each namespace

Allowing the teams to do step 3-5 will help them to be more dynamically. Using Rancher Projects already ensures only the team which has permissions to access the project, so giving them permissions to manage SecretStore should be fine. Besied the SecretStore, they also need to deploy a ServiceAccount So for each namespace they have to deploy this:

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: azure-store
spec:
  provider:
    # provider type: azure keyvault
    azurekv:
      authType: ManagedIdentity
      # Optionally set the Id of the Managed Identity, if multiple identities are assigned to external-secrets operator
      identityId: "<MI_clientId>"
      # URL of your vault instance, see: https://docs.microsoft.com/en-us/azure/key-vault/general/about-keys-secrets-certificates
      vaultUrl: "https://my-keyvault-name.vault.azure.net"

apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-serviceaccount
  namespace: my-namespace

This gives the developers the most flexibility. But wait we still have to do the fedration. This step is essential, as it connects the ServiceAccount with the Managed Identity, so the ServiceAccount can access the secrets.

As we manage our cloud resources with Terraform we will use federated_identity_credential to manage this federation. With that, developers can initiate the federation, which will only be granted after a peer review. This will ensoure adequate controll on which identity is used for which namespace.

Conclusion and next steps

WHile we believe this is a nice and flexible solution for developers to be in control to use External Secrets Operator in their namespaces, the last step is still not ideal - at least not in clusters where developers frequently create new namespace - it breaks of the flow as objects in step 3 and 4 can be done in continuous deployment, while step 5 is done in another place.

We are thinking to automatize step 5 by either something like an admission controller or a Kubernetes operator. I did not find yet something existing, so we might need to implement something. Stay tuned for updates.