While working on managing our Rancher clusters is the management of Rancher projects. I want to talk about the approach I have taken, which may be useful to you as well.
What is a project?
Rancher is using projects as an additional layer:
A project is a group of namespaces, and it is a concept introduced by Rancher. Projects allow you to manage multiple namespaces as a group and perform Kubernetes operations in them. You can use projects to support multi-tenancy, so that a team can access a project within a cluster without having access to other projects in the same cluster.
In terms of hierarchy:
- Clusters contain projects
- Projects contain namespaces
There are several things you can do on projects
- Assign users to a group of namespaces (i.e., project membership).
- Assign users specific roles in a project. A role can be owner, member, read-only, or custom.
- Assign resources to the project.
- Assign Pod Security Policies.
Each cluster already come with 2 pre-defined projects:
How to manage projects with Terraform?
Generic approach
The obvious resource to do that is rancher2_project, which allows you to manage a project and it’s resources. However, if you also want to manage the permissions, you also need
- rancher2_role_template - Provides a Rancher v2 Role Template resource. This can be used to create Role Template for Rancher v2 and retrieve their information
- rancher2_project_role_template_binding - Provides a Rancher v2 Project Role Template Binding resource. This can be used to create Project Role Template Bindings for Rancher v2 environments and retrieve their information
I consider projects, relatively static, so we can easily have multiple blocks like this
resource "rancher2_project" "default" {
name ="Default"
description = "Default project created for the cluster",
cluster_id = rancher2_cluster.cluster.id
container_resource_limit {
limits_cpu = 10m
limits_memory = 10Mi
requests_cpu = 2
requests_memory = 2Gi
}
resource_quota {
project_limit {
requests_storage = 5000Mi
}
namespace_default_limit {
requests_storage = 500Mi
}
}
}
Then we also need the rancher2_role_template and rancher2_project_role_template_binding, which will lead to a lot of resources and relatively tedious if you have a lot of projects and role-bindings. So my aim is to manage projects and the related role bindings together, also so that developers which are eventually not so deep into terraform can easily adjust limits and so.
How I manage projects
So I decided to have a local map, which contains all projects and it’s configuration in a file called projects.tf
locals {
projects = {
"default" = {
name = "Default",
description = "Default project created for the cluster",
limits_cpu = "2000m",
limits_memory = "1000Mi",
requests_cpu = "10m",
requests_memory = "10Mi",
requests_storage_project = "5000Mi",
requests_storage_namespace = "500Mi"
},
"system" = {
name = "System",
description = "System project created for the cluster",
limits_cpu = "2000m",
limits_memory = "1000Mi",
requests_cpu = "10m",
requests_memory = "10Mi",
requests_storage_project = "3000Gi",
requests_storage_namespace = "80Gi"
},
}
}
Projects can be dynamically created using a fore_each
resource "rancher2_project" "pr" {
for_each = local.projects
name = each.value.name
description = each.value.description
cluster_id = rancher2_cluster.cluster.id
container_resource_limit {
limits_cpu = each.value.limits_cpu
limits_memory = each.value.limits_memory
requests_cpu = each.value.requests_cpu
requests_memory = each.value.requests_memory
}
resource_quota {
project_limit {
requests_storage = each.value.requests_storage_project
}
namespace_default_limit {
requests_storage = each.value.requests_storage_namespace
}
}
}
How I manage roles
First we have to understand that roles are defined in the local cluster, and in order to create the bindings, we need the role_template_id of these roles. As each cluster has it’s own state file, we need first make the current cluster aware of the remote state of the local cluster, in file data.tf. Assuming we use azurerm as backend it would look like this:
data "terraform_remote_state" "local-state" {
backend = "azurerm"
config = {
use_azuread_auth = true
tenant_id = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
subscription_id = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
resource_group_name = "rg-rancher"
storage_account_name = "sa-rancher"
container_name = "rancher"
key = "rancher-local"
}
}
We also need to expose the role_template_ids in a way, we can use them, hence we have this output.tf
output "roleids" {
description = "IDs for roles"
value = {
custom-read-only=rancher2_role_template.custom-read-only.id
custom-project-member=rancher2_role_template.custom-project-member.id
custom-cluster-read-only=rancher2_role_template.custom-cluster-read-only.id
custom-global-read-only=rancher2_global_role.custom-global-read-only.id
cluster-owner="cluster-owner"
admin="admin"
}
}
Now we can access them using the rolename, so for example custom-read-only-role will be accessed like this:
data.terraform_remote_state.playground-local-state.outputs.roleids["custom-read-only"]
As a next step, as I already have the list local.projects, I extend it with a map memebers
locals {
projects = {
"default" = {
name = "Default",
description = "Default project created for the cluster",
limits_cpu = "2000m",
limits_memory = "1000Mi",
requests_cpu = "10m",
requests_memory = "10Mi",
requests_storage_project = "5000Mi",
requests_storage_namespace = "500Mi"
"all-ro" = {
name="K8s_ALL_ReadOnly"
role="custom-project-member"
}
"ro" = {
name="K8s_Default_ReadOnly"
role="custom-read-only"
}
"pm" = {
name="K8s_Default_ProjectMember"
role="custom-project-member"
}
}
},
"system" = {
name = "System",
description = "System project created for the cluster",
limits_cpu = "2000m",
limits_memory = "1000Mi",
requests_cpu = "10m",
requests_memory = "10Mi",
requests_storage_project = "5000Mi",
requests_storage_namespace = "500Mi"
members = {
"all-ro" = {
name="K8s_ALL_ReadOnly"
role="custom-project-member"
}
"ro" = {
name="K8s_System_ReadOnly"
role="custom-read-only"
}
"pm" = {
name="K8s_System_ProjectMember"
role="custom-project-member"
}
}
},
},
}
}
At last I create the role binding dynamically, but first I need to create an iterable list with all data from the members element of each project. Hence I iterate over all projects and grab the required inf
locals {
prtbs = flatten([
for projectname,project in local.projects : [
for role, details in project.members: {
role_name = "${projectname}-${role}"
project_name = "${projectname}"
role_template_id = data.terraform_remote_state.local-state.outputs.roleids["${details.role}"]
group_principal_id = "activedirectory_group://CN=${details.name},OU=K8S,OU=Groups,DC=wyssmann,DC=intra"
}
]
])
}
Which will get me this list
local.prtbs
[
{
"group_principal_id" = "activedirectory_group://CN=K8s_ALL_ReadOnly,OU=K8S,OU=Groups,DC=wyssmann,DC=intra
"project_name" = "default"
"role_name" = "default-all-ro"
"role_template_id" = "rt-xxxxx"
},
{
"group_principal_id" = "activedirectory_group://CN=K8s_Default_ProjectMember,OU=K8S,OU=Groups,DC=wyssmann,DC=intra
"project_name" = "default"
"role_name" = "default-pm"
"role_template_id" = "rt-xxxxx"
},
{
"group_principal_id" = "activedirectory_group://CN=K8s_Default_ReadOnly,OU=K8S,OU=Groups,DC=wyssmann,DC=intra
"project_name" = "default"
"role_name" = "default-ro"
"role_template_id" = "rt-xxxxx"
},
{
"group_principal_id" = "activedirectory_group://CN=K8s_ALL_ReadOnly,OU=K8S,OU=Groups,DC=wyssmann,DC=intra
"project_name" = "system"
"role_name" = "system-all-ro"
"role_template_id" = "rt-xxxxx"
},
{
"group_principal_id" = "activedirectory_group://CN=K8s_System_ProjectMember,OU=K8S,OU=Groups,DC=wyssmann,DC=intra
"project_name" = "system"
"role_name" = "system-pm"
"role_template_id" = "rt-xxxxx"
},
{
"group_principal_id" = "activedirectory_group://CN=K8s_System_ReadOnly,OU=K8S,OU=Groups,DC=wyssmann,DC=intra
"project_name" = "system"
"role_name" = "system-ro"
"role_template_id" = "rt-xxxxx"
},
]
From this least I use for_each to create a resource for each binding
resource "rancher2_project_role_template_binding" "prtb" {
for_each = { for entry in local.prtbs : "${entry.role_name}" => entry }
name = "${each.key}"
project_id = rancher2_project.pr["${each.value.project_name}"].id
role_template_id = each.value.role_template_id
group_principal_id = each.value.group_principal_id
}
Conclusion
When working with a lot of objects of the same type, it fastly becomes tedious to manually create multiple similar resource blocks, and adjusting the values of each of the blocks without errors. I find it easier to have a easy manageable list/map and create resources dynamically. Luckily [Terraform] supports that, although in the beginning I struggled quite a bit to get the loops and inner loops correctly. Once you manage that, you have a lot of possibilities to make your config files much more manageable and readable.