Filter out elements from a json object in in Terraform

Posted on October 19, 2022 by Adrian Wyssmann ‐ 4 min read

We would like to manage our own Azure Policy Initiatives with Terraform. Our own initatives shall be based on existing ones, but without deprecated policies.

What are Azure Policy Initiatives

Azure Policy allows you audit your infrastructure on Azure and enforce compliance.

Through its compliance dashboard, it provides an aggregated view to evaluate the overall state of the environment, with the ability to drill down to the per-resource, per-policy granularity. It also helps to bring your resources to compliance through bulk remediation for existing resources and automatic remediation for new resources.

Policies are grouped in initiatives:

An initiative definition is a collection of policy definitions that are tailored toward achieving a singular overarching goal.

There are several builtin initiatives which are constantly updated. But this also means, some of the policies part of the initiative are deprecated.

How to manage Policy Initiatives with Terraform

azurerm_policy_set_definition form the azurerm provider is the one resource, that manages policy sets or also know as policy initiatives. While the field parameters is used to provide the policy definition as a JSON object. AS mentioned in the introduction, we want to have our own initiatives - based on existing ones - but without deprecated policies.

Let’s say we want to create an initiative based AzureSecurityCenter initiative, we can use the [azure_policy_set_definition datasource] to access an existing policy

data "azurerm_policy_set_definition" "azuresecuirtybenchmark" {
  display_name = "Azure Security Benchmark"
}

We can then access the parameter as follows

data.azurerm_policy_set_definition.azuresecuirtybenchmark.parameters

How to remove deprecated policies using Terraform

So far so good, so let’s see what we get:

locals {
  decoded_policy_set_definition_params  = jsondecode(data.azurerm_policy_set_definition.azuresecuirtybenchmark.parameters)
}

output "demo" {
    value = local.decoded_policy_set_definition_params 
}

This will show you the content:

Changes to Outputs:
  + demo = {
      + ASCDependencyAgentAuditLinuxEffect                                                                   = {
          + allowedValues = [
              + "AuditIfNotExists",
              + "Disabled",
            ]
          + defaultValue  = "AuditIfNotExists"
          + metadata      = {
              + description = "Enable or disable Dependency Agent for Linux VMs"
              + displayName = "Audit Dependency Agent for Linux VMs monitoring"
            }
          + type          = "String"
...

From there let’s iterate over the root elements, by adding this

  filtered_policy_set_definition_params  = [
    for v in decoded_policy_set_definition_params : v.type
  ]

However terraform plan results in this error

Error: Invalid reference

  on local.tf line 18, in locals:
  18:     for v in decoded_policy_set_definition_params : s.type

A reference to a resource type must be followed by at least one attribute

This is cause the decoded_policy_set_definition_params is local, hence

  filtered_policy_set_definition_params  = [
    for v in local.decoded_policy_set_definition_params : v
  ]

Works and if we adjust the output as follows

output "demo" {
    value = local.filtered_policy_set_definition_params 
}

We still get the same output:

Changes to Outputs:
  + demo = [
      + {
          + allowedValues = [
              + "AuditIfNotExists",
              + "Disabled",
            ]
          + defaultValue  = "AuditIfNotExists"
          + metadata      = {
              + description = "Enable or disable Dependency Agent for Linux VMs"
              + displayName = "Audit Dependency Agent for Linux VMs monitoring"
            }
          + type          = "String"
        },
...

Now we want to filter out all elements which contain

metadata     = {
  deprecated  = true

Following this post the recommended way is to use [lookup], which

retrieves the value of a single element from a map, given its key. If the given key does not exist, the given default value is returned instead.

As we want all that don’t have the attribute metadata.deprecated, we would use this

for v in local.decoded_policy_set_definition_params : s if lookup(v.metadata, "deprecated", null) == null

The result contains all elements, but we are missing the key of the element (the policy name)

Changes to Outputs:
  + demo = [
      + {
          + allowedValues = [
              + "AuditIfNotExists",
              + "Disabled",
            ]
          + defaultValue  = "AuditIfNotExists"
          + metadata      = {
              + description = "Enable or disable Dependency Agent for Linux VMs"
              + displayName = "Audit Dependency Agent for Linux VMs monitoring"
            }
...

So when iterating over the json object we have to read carefully what it tells [here][result types]

The type of brackets around the for expression decide what type of result it produces.
[] produces a tuple
{} result is an object and you must provide two result expressions that are separated by the => symbol

So the adjusted expression is

  filtered_policy_set_definition_params  = {
    for k, v in local.decoded_policy_set_definition_params : k=>v if lookup(v.metadata, "deprecated", null) == null
  }

Which results in this

  + demo = {
      + ASCDependencyAgentAuditLinuxEffect                                                                   = {
          + allowedValues = [
              + "AuditIfNotExists",
              + "Disabled",
            ]
          + defaultValue  = "AuditIfNotExists"
          + metadata      = {
              + description = "Enable or disable Dependency Agent for Linux VMs"
              + displayName = "Audit Dependency Agent for Linux VMs monitoring"
            }
          + type          = "String"
...

Now we can use this in our azurerm_policy_set_definition as an jsonencode-ed string:

resource "azurerm_policy_set_definition" "Example" {
  name                = "Example-Policy-Set"
  policy_type         = "Custom"
  display_name        = "Example-Policy-Set"
  management_group_id = data.azurerm_management_group.Sandboxes.id
  parameters          = jsonencode(local.filtered_policy_set_definition_params)
  dynamic "policy_definition_reference" {
    for_each = data.azurerm_policy_set_definition.test.policy_definition_reference
    content {
      parameter_values     = policy_definition_reference.value.parameter_values
      policy_definition_id = policy_definition_reference.value.policy_definition_id
      reference_id         = policy_definition_reference.value.reference_id

    }
  }
}

Wyssmann Engineering

Title here

Filter out elements from a json object in in Terraform

What are Azure Policy Initiatives

How to manage Policy Initiatives with Terraform

How to remove deprecated policies using Terraform