Manage Rancher Projects in Terraform

Posted on November 2, 2022 by Adrian Wyssmann ‐ 6 min read

While working on managing our Rancher clusters is the management of Rancher projects. I want to talk about the approach I have taken, which may be useful to you as well.

What is a project?

Rancher is using projects as an additional layer:

A project is a group of namespaces, and it is a concept introduced by Rancher. Projects allow you to manage multiple namespaces as a group and perform Kubernetes operations in them. You can use projects to support multi-tenancy, so that a team can access a project within a cluster without having access to other projects in the same cluster.

In terms of hierarchy:

Clusters contain projects
Projects contain namespaces

There are several things you can do on projects

Assign users to a group of namespaces (i.e., project membership).
Assign users specific roles in a project. A role can be owner, member, read-only, or custom.
Assign resources to the project.
Assign Pod Security Policies.

Each cluster already come with 2 pre-defined projects:

How to manage projects with Terraform?

Generic approach

The obvious resource to do that is rancher2_project, which allows you to manage a project and it’s resources. However, if you also want to manage the permissions, you also need

rancher2_role_template - Provides a Rancher v2 Role Template resource. This can be used to create Role Template for Rancher v2 and retrieve their information
rancher2_project_role_template_binding - Provides a Rancher v2 Project Role Template Binding resource. This can be used to create Project Role Template Bindings for Rancher v2 environments and retrieve their information

I consider projects, relatively static, so we can easily have multiple blocks like this

resource "rancher2_project" "default" {
  name                            ="Default"
  description                     = "Default project created for the cluster",
  cluster_id                      = rancher2_cluster.cluster.id
  container_resource_limit {
      limits_cpu      = 10m
      limits_memory   = 10Mi
      requests_cpu    = 2
      requests_memory = 2Gi
  }
  resource_quota {
    project_limit {
      requests_storage = 5000Mi
    }
    namespace_default_limit {
      requests_storage = 500Mi
    }
  }
}

Then we also need the rancher2_role_template and rancher2_project_role_template_binding, which will lead to a lot of resources and relatively tedious if you have a lot of projects and role-bindings. So my aim is to manage projects and the related role bindings together, also so that developers which are eventually not so deep into terraform can easily adjust limits and so.

How I manage projects

So I decided to have a local map, which contains all projects and it’s configuration in a file called projects.tf

locals {
  projects = {
    "default" = {
      name                       = "Default",
      description                = "Default project created for the cluster",
      limits_cpu                 = "2000m",
      limits_memory              = "1000Mi",
      requests_cpu               = "10m",
      requests_memory            = "10Mi",
      requests_storage_project   = "5000Mi",
      requests_storage_namespace = "500Mi"
    },
    "system" = {
      name                       = "System",
      description                = "System project created for the cluster",
      limits_cpu                 = "2000m",
      limits_memory              = "1000Mi",
      requests_cpu               = "10m",
      requests_memory            = "10Mi",
      requests_storage_project   = "3000Gi",
      requests_storage_namespace = "80Gi"
    },
  }
}

Projects can be dynamically created using a fore_each

resource "rancher2_project" "pr" {
  for_each = local.projects
  name                            = each.value.name
  description                     = each.value.description
  cluster_id                      = rancher2_cluster.cluster.id
  container_resource_limit {
      limits_cpu      = each.value.limits_cpu
      limits_memory   = each.value.limits_memory
      requests_cpu    = each.value.requests_cpu
      requests_memory = each.value.requests_memory
  }
  resource_quota {
    project_limit {
      requests_storage = each.value.requests_storage_project
    }
    namespace_default_limit {
      requests_storage = each.value.requests_storage_namespace
    }
  }
}

How I manage roles

First we have to understand that roles are defined in the local cluster, and in order to create the bindings, we need the role_template_id of these roles. As each cluster has it’s own state file, we need first make the current cluster aware of the remote state of the local cluster, in file data.tf. Assuming we use azurerm as backend it would look like this:

data "terraform_remote_state" "local-state" {
  backend = "azurerm"

  config = {
    use_azuread_auth     = true
    tenant_id            = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
    subscription_id      = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
    resource_group_name  = "rg-rancher"
    storage_account_name = "sa-rancher"
    container_name       = "rancher"
    key                  = "rancher-local"
  }
}

We also need to expose the role_template_ids in a way, we can use them, hence we have this output.tf

output "roleids" {
  description = "IDs for roles"
  value = {
    custom-read-only=rancher2_role_template.custom-read-only.id
    custom-project-member=rancher2_role_template.custom-project-member.id
    custom-cluster-read-only=rancher2_role_template.custom-cluster-read-only.id
    custom-global-read-only=rancher2_global_role.custom-global-read-only.id
    cluster-owner="cluster-owner"
    admin="admin"
  }
}

Now we can access them using the rolename, so for example custom-read-only-role will be accessed like this:

data.terraform_remote_state.playground-local-state.outputs.roleids["custom-read-only"]

As a next step, as I already have the list local.projects, I extend it with a map memebers

locals {
  projects = {
    "default" = {
      name                       = "Default",
      description                = "Default project created for the cluster",
      limits_cpu                 = "2000m",
      limits_memory              = "1000Mi",
      requests_cpu               = "10m",
      requests_memory            = "10Mi",
      requests_storage_project   = "5000Mi",
      requests_storage_namespace = "500Mi"
         "all-ro" = {
            name="K8s_ALL_ReadOnly"
            role="custom-project-member"
         }
         "ro" = {
            name="K8s_Default_ReadOnly"
            role="custom-read-only"
         }
         "pm" = {
            name="K8s_Default_ProjectMember"
            role="custom-project-member"
         }
      }
    },
    "system" = {
      name                       = "System",
      description                = "System project created for the cluster",
      limits_cpu                 = "2000m",
      limits_memory              = "1000Mi",
      requests_cpu               = "10m",
      requests_memory            = "10Mi",
      requests_storage_project   = "5000Mi",
      requests_storage_namespace = "500Mi"
      members = {
         "all-ro" = {
            name="K8s_ALL_ReadOnly"
            role="custom-project-member"
         }
         "ro" = {
            name="K8s_System_ReadOnly"
            role="custom-read-only"
         }
         "pm" = {
            name="K8s_System_ProjectMember"
            role="custom-project-member"
         }
      }
    },
    },
  }
}

At last I create the role binding dynamically, but first I need to create an iterable list with all data from the members element of each project. Hence I iterate over all projects and grab the required inf

locals {
   prtbs = flatten([
      for projectname,project in local.projects : [
         for role, details in project.members: {
            role_name          = "${projectname}-${role}"
            project_name       = "${projectname}"
            role_template_id   = data.terraform_remote_state.local-state.outputs.roleids["${details.role}"]
            group_principal_id = "activedirectory_group://CN=${details.name},OU=K8S,OU=Groups,DC=wyssmann,DC=intra"
         }
      ]
   ])
}

Which will get me this list

local.prtbs
[
  {
    "group_principal_id" = "activedirectory_group://CN=K8s_ALL_ReadOnly,OU=K8S,OU=Groups,DC=wyssmann,DC=intra
    "project_name" = "default"
    "role_name" = "default-all-ro"
    "role_template_id" = "rt-xxxxx"
  },
  {
    "group_principal_id" = "activedirectory_group://CN=K8s_Default_ProjectMember,OU=K8S,OU=Groups,DC=wyssmann,DC=intra
    "project_name" = "default"
    "role_name" = "default-pm"
    "role_template_id" = "rt-xxxxx"
  },
  {
    "group_principal_id" = "activedirectory_group://CN=K8s_Default_ReadOnly,OU=K8S,OU=Groups,DC=wyssmann,DC=intra
    "project_name" = "default"
    "role_name" = "default-ro"
    "role_template_id" = "rt-xxxxx"
  },
  {
    "group_principal_id" = "activedirectory_group://CN=K8s_ALL_ReadOnly,OU=K8S,OU=Groups,DC=wyssmann,DC=intra
    "project_name" = "system"
    "role_name" = "system-all-ro"
    "role_template_id" = "rt-xxxxx"
  },
  {
    "group_principal_id" = "activedirectory_group://CN=K8s_System_ProjectMember,OU=K8S,OU=Groups,DC=wyssmann,DC=intra
    "project_name" = "system"
    "role_name" = "system-pm"
    "role_template_id" = "rt-xxxxx"
  },
  {
    "group_principal_id" = "activedirectory_group://CN=K8s_System_ReadOnly,OU=K8S,OU=Groups,DC=wyssmann,DC=intra
    "project_name" = "system"
    "role_name" = "system-ro"
    "role_template_id" = "rt-xxxxx"
  },
]

From this least I use for_each to create a resource for each binding

resource "rancher2_project_role_template_binding" "prtb" {
  for_each           = { for entry in local.prtbs : "${entry.role_name}" => entry }
  name               = "${each.key}"
  project_id         = rancher2_project.pr["${each.value.project_name}"].id
  role_template_id   = each.value.role_template_id
  group_principal_id = each.value.group_principal_id
}

Conclusion

When working with a lot of objects of the same type, it fastly becomes tedious to manually create multiple similar resource blocks, and adjusting the values of each of the blocks without errors. I find it easier to have a easy manageable list/map and create resources dynamically. Luckily [Terraform] supports that, although in the beginning I struggled quite a bit to get the loops and inner loops correctly. Once you manage that, you have a lot of possibilities to make your config files much more manageable and readable.

Wyssmann Engineering

Title here