Terraform secrets with SOPS and Azure Keyvault

We are heavily using Terraform and and also Azure. However until now, we left out certain things cause they contain secrets which we don't want to expose in the code. SOPS is a nice solution to solve that problem and keep things together what belongs together.

What is SOPS? #

SOPS stands for Secrets OPerationS, and is an open-source text file editor that encrypts/decrypts YAML, JSON, ENV, INI and BINARY formats and encrypts with AWS KMS, GCP KMS, Azure Key Vault, age, and PGP.

If you want more details to SOPS as such you can also have a look at A Comprehensive Guide to SOPS: Managing Your Secrets Like A Visionary, Not a Functionary (gitguardian.com).

What do you need? #

As we will use azure-key-vault: you will need

1. access to the respective key-vault

2. sops binary

3. Encrypting/decrypting with Azure Key Vault requires the resource identifier for a key. This has the following form:

   https://${VAULT_URL}/keys/${KEY_NAME}/${KEY_VERSION}
   

Sops only encrypts the secrets, so the advantage is you can have key-value pairs in a yaml or json file, and while the key is readable, the value part is encrypted. This also allows to address particular keys - we will see this later.

Setup #

1. We setup a keyvault (Key Vault names are globally unique) - you can do this using azure cli (as per example) or prefferable using terraform

2. We create a key - you can do this using azure cli (as per example) or prefferable using terraform

   resource "azurerm_key_vault_key" "sops-key" {
     name = "sops-key"
     key_vault_id = azure-key-vault.terraform-kv.id
     key_type     = "RSA"
     key_size     = 2048
   
     key_opts = [
       "decrypt",
       "encrypt",
     ]
   }
   

Encryption and Decryption #

Once we have the key created and configured, then we can use it for encryption. I usually grab it as follows

sub_name=sub-terraform
keyvault_name=terraform-kv
keyvault_subscription=$(az account subscription list --query "[? displayName=='$sub_name'].subscriptionId | [0]" | sed "s/\"//g")
sopskey=$(az keyvault key show --name sops-key --vault-name $keyvault_name --subscription $keyvault_subscription --query key.kid | sed "s/\"//g")

Now you can encrypt a file using:

sops --encrypt --azure-kv $sopskey test.yaml > test.enc.json

And decrypt it using:

sops --decrypt test.enc.json

As mentioned in the beginning, sops only encrypts the values not the keys, so assuming the content of test.json looks as follows:

{
    "key1": "secret1"
    "key2": "secret2"
}

Encryption will result in something like

{
    "key1": "ENC[AES256_GCM,data:QdlQDvuZbx+3w1E=,iv:YUhT2wfJZ/u39Gag27iD6x8oiQ+DOpSfGAkQ7jEyTIU=,tag:0Q5lWcE0oP77r+swKckCqQ==,type:str]",
    "key2": "ENC[AES256_GCM,data:UiiRDR92negQ3y6hD+sfeCm+LirN/kT+,iv:Z5HaUeXEdv2WzgD2kByGP0rg/fX9TgF2ckIf9S6hbq8=,tag:W5WZLRl2gzj/v5WwbdxkHQ==,type:str]",
    "sops": {
        "kms": null,
        "gcp_kms": null,
        "azure_kv": [
            {
                "vault_url": "https://kv-.vault.azure.net",
                "name": "sops-key",
                "version": "ac618eb34afc41d1914eb64cf0f30cee",
                "created_at": "2023-09-25T06:42:14Z",
                "enc": "XXXXXXXXXXXX"
            }
        ],
        "hc_vault": null,
        "age": null,
        "lastmodified": "2023-09-25T06:42:17Z",
        "mac": "ENC[AES256_GCM,data:XXXXXXX,iv:YYYYYYY,tag:ZZZZZZZ,type:str]",
        "pgp": null,
        "unencrypted_suffix": "_unencrypted",
        "version": "3.8.0"
    }
}

Usage in Terraform #

We need carlpett/sops-provider. As part of your terraform code, you have to define the encrypted file from above

data "sops_file" "test-secret" {
  source_file = "test.enc.yaml"
}

As they keys can be accessed you can e.g. use the secret from key1 as follows

output "db-password" {
  value = data.sops_file.test-secret.data["key1"]
}

You can also encrypt whole files, e.g. private keys (not .yaml or .json)

sops --encrypt --azure-kv $sopskey sslcertificate.key > sslcertificate.enc.key

These have then to be declared using input_type=raw …

data "sops_file" "sslcertificate" {
  source_file = "sslcertificate.enc.key"
  input_type  = "raw"
}

… accessed using .raw

resource "kubernetes_secret" "certificate" {
  ...

  data = {
    "tls.crt" = file("${path.module}/sslcertificate.pem")
    "tls.key" = data.sops_file.sslcertificate.enc.key
  }

  type = "kubernetes.io/tls"
}

Conclusion #

Using SOPS with an external keyvault, really simplifes managing secret data together with your terraform code, without exposing them. So you can still make your code visible to the rest of your team(s)/company without compromising on leaking secrets.