Terraform secrets with SOPS and Azure Keyvault
Posted in development on October 24, 2023 by Adrian Wyssmann ‐ 3 min read
We are heavily using Terraform and and also Azure. However until now, we left out certain things cause they contain secrets which we don’t want to expose in the code. SOPS is a nice solution to solve that problem and keep things together what belongs together.
What is SOPS?
SOPS stands for Secrets OPerationS, and is an open-source text file editor that encrypts/decrypts YAML, JSON, ENV, INI and BINARY formats and encrypts with AWS KMS, GCP KMS, Azure Key Vault, age, and PGP.
If you want more details to SOPS as such you can also have a look at A Comprehensive Guide to SOPS: Managing Your Secrets Like A Visionary, Not a Functionary (gitguardian.com).
What do you need?
As we will use azure-key-vault: you will need
access to the respective key-vault
sops binary
Encrypting/decrypting with Azure Key Vault requires the resource identifier for a key. This has the following form:
https://${VAULT_URL}/keys/${KEY_NAME}/${KEY_VERSION}
Sops only encrypts the secrets, so the advantage is you can have key-value pairs in a yaml or json file, and while the key is readable, the value part is encrypted. This also allows to address particular keys - we will see this later.
Setup
We setup a keyvault (Key Vault names are globally unique) - you can do this using azure cli (as per example) or prefferable using terraform
We create a key - you can do this using azure cli (as per example) or prefferable using terraform
resource "azurerm_key_vault_key" "sops-key" { name = "sops-key" key_vault_id = azure-key-vault.terraform-kv.id key_type = "RSA" key_size = 2048 key_opts = [ "decrypt", "encrypt", ] }
Encryption and Decryption
Once we have the key created and configured, then we can use it for encryption. I usually grab it as follows
sub_name=sub-terraform
keyvault_name=terraform-kv
keyvault_subscription=$(az account subscription list --query "[? displayName=='$sub_name'].subscriptionId | [0]" | sed "s/\"//g")
sopskey=$(az keyvault key show --name sops-key --vault-name $keyvault_name --subscription $keyvault_subscription --query key.kid | sed "s/\"//g")
Now you can encrypt a file using:
sops --encrypt --azure-kv $sopskey test.yaml > test.enc.json
And decrypt it using:
sops --decrypt test.enc.json
As mentioned in the beginning, sops only encrypts the values not the keys, so assuming the content of test.json
looks as follows:
{
"key1": "secret1"
"key2": "secret2"
}
Encryption will result in something like
{
"key1": "ENC[AES256_GCM,data:QdlQDvuZbx+3w1E=,iv:YUhT2wfJZ/u39Gag27iD6x8oiQ+DOpSfGAkQ7jEyTIU=,tag:0Q5lWcE0oP77r+swKckCqQ==,type:str]",
"key2": "ENC[AES256_GCM,data:UiiRDR92negQ3y6hD+sfeCm+LirN/kT+,iv:Z5HaUeXEdv2WzgD2kByGP0rg/fX9TgF2ckIf9S6hbq8=,tag:W5WZLRl2gzj/v5WwbdxkHQ==,type:str]",
"sops": {
"kms": null,
"gcp_kms": null,
"azure_kv": [
{
"vault_url": "https://terraform-kv.vault.azure.net",
"name": "sops-key",
"version": "ac618eb34afc41d1914eb64cf0f30cee",
"created_at": "2023-09-25T06:42:14Z",
"enc": "XXXXXXXXXXXX"
}
],
"hc_vault": null,
"age": null,
"lastmodified": "2023-09-25T06:42:17Z",
"mac": "ENC[AES256_GCM,data:XXXXXXX,iv:YYYYYYY,tag:ZZZZZZZ,type:str]",
"pgp": null,
"unencrypted_suffix": "_unencrypted",
"version": "3.8.0"
}
}
Usage in Terraform
We need carlpett/sops-provider. As part of your terraform code, you have to define the encrypted file from above
data "sops_file" "test-secret" {
source_file = "test.enc.yaml"
}
As they keys can be accessed you can e.g. use the secret from key1
as follows
output "db-password" {
value = data.sops_file.test-secret.data["key1"]
}
You can also encrypt whole files, e.g. private keys (not .yaml or .json)
sops --encrypt --azure-kv $sopskey sslcertificate.key > sslcertificate.enc.key
These have then to be declared using input_type=raw
…
data "sops_file" "sslcertificate" {
source_file = "sslcertificate.enc.key"
input_type = "raw"
}
… accessed using .raw
resource "kubernetes_secret" "certificate" {
...
data = {
"tls.crt" = file("${path.module}/sslcertificate.pem")
"tls.key" = data.sops_file.sslcertificate.enc.key
}
type = "kubernetes.io/tls"
}
Conclusion
Using SOPS with an external keyvault, really simplifes managing secret data together with your terraform code, without exposing them. So you can still make your code visible to the rest of your team(s)/company without compromising on leaking secrets.