Blog on Wyssmann Engineering

User Bitwarden as ssh agent

Tue, 13 Jan 2026 10:08:00 +0200

Instead of managing ssh keys locally, you can use Bitwarden as a ssh-agent which IMHO brings some nice benefits, I want to show you.

Working with ssh (at least at home) you create your keys with ssh-keygen and place them in $HOME/.ssh. With Bitwarden the process is different:

At first you need to ensure you have bitwarden application installed on your system and app is running.
As a second step you have to create an SSH key in the WebApp or the Desktop app:

NixOS for all my system - part 2

Sun, 11 Jan 2026 10:08:00 +0200

I my last post I showed you how to configure a remote host. In this post I want to reduce the manual interaction with the remote host - remember I had to do the manual setup on the target system.

Overall approach

The 3 first steps are basically the same, I have to get the target system ready

Download latest iso image
Burn image to an usb stick
Boot notebook from usb stick
Install the remote system

Now the next steps is crucial, running the target system with an usb stick comes with ssh running. Unfortunately neither root nor default user nixos have a password set. If you use the image downloaded from the website you still have to interact with the remote system briefly by running sudo passwd nixos or sudo passwd root.

NixOS for all my system

Sat, 15 Nov 2025 10:08:00 +0200

I introduced you to Nixos in my previous post. This is an amazing way of managing your linux in a declarative way with easy rollback.

For my other nodes in my home lab I use(d) Ansible and Terraform which also allows me to create reproducible environments. However, I don’t want to use different tools, so why not use Nixos everywhere.

Overall approach

As a starting point on my journey, I will install Nixos on a spare notebook manually and then apply further changes from the config I have locally on my developer machine. So what I did:

Chezmoi externals behind a coorporate proxy

Tue, 17 Jun 2025 09:06:00 +0200

As you know I use https://wyssmann.com/blog/2022/08/chezmoi-a-very-cool-tool-to-manage-your-dotfiles/ to manage my dotfiles. One problem I face is using it in corparate environment with proxy that uses kerberos authentication. Just adding the external urls to .chezmoiexternal.toml will not work ase there is no native kerberos support.

Luckily there is curl which supports Kerberos authentication with –negotiate. In order to ensure that curl works by default with --netgotiate I add the following content into ~/.curlrc

--proxy-negotiate
-u :

The I create a script, that would run at the beginning e.g. run_before_get_packages.sh.tmpl:

NixOS my new linux distro

Sun, 01 Jun 2025 10:08:00 +0200

Reason for NixOs

Some months ago I got a new Notebook and took the chance to checkout NixOS. I used Archlinux for many years and was very happe. However as DevOps/Platform Engineer I am used to make things reproducible - I use(d) Ansible and Terraform which allows me to create repoducible environments. I even [used Ansible][https://gitlab.com/papanito/devenv] for some aspcets. Then I stumbled upon NixOs which states

Declarative builds and deployments. Nix is a tool that takes a unique approach to package management and system configuration. Learn how to make reproducible, declarative and reliable systems.

How to use separate environment configuration files ArgoCD

Thu, 13 Feb 2025 14:58:00 +0200

Problem

Our current argo setup is as follows:

platform-tooling
├argocd
| └applications
│   ├application_name1.yaml
│   └application_name2.yaml
├application_name1
│ ├env1
│ │ └values.yaml
│ ├dev
│ │ └values.yaml
...

application_nameX is usually an applicatioSet which ensures the application is deployed to all clusters. Currently this file contains a repetitive configuration:

generators:
- list:
 elements:
 - cluster: cluster-a
 clusterid: c-abcde
 project: p-abcde
 env: env1
 notifyChannel: channel-a
 targetRevision: x.x.x.x
 
 - cluster: cluster-b
 clusterid: c-bcdef
 project: p-bcdef
 env: env2
 notifyChannel: channel-b
 targetRevision: x.x.x.x
 ...

This list grows the more cluster one has. It also does not follow the DRY-principle. So, following one of my last posts I started to use environment configurations files. So we have the common config in a folder argocd/cluster-config which contains a <environment>.yaml containing standard config. For example env1.yaml

Copy of Failed to get the data key required to decrypt the SOPS file: The provided grant has expired due to it being revoked

Thu, 23 Jan 2025 08:00:00 +0200

As of a sudden Terraform secrets with SOPS and Azure Keyvault does not work anymore and you will get an error when trying to decrypt:

sops --decrypt ./secrets/secrets.enc.json > ./secrets/secrets.json
Failed to get the data key required to decrypt the SOPS file.

Group 0: FAILED
 https://mykv.vault.azure.net/keys/sops-key/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx: FAILED
 - | failed to decrypt sops data key with Azure Key Vault key
 | 'https://mykv.vault.azure.net/keys/sops-key/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx':
 | DefaultAzureCredential: failed to acquire a token.
 | Attempted credentials:
 | EnvironmentCredential: missing environment variable
 | AZURE_TENANT_ID
 | WorkloadIdentityCredential: no client ID specified. Check
 | pod configuration or set ClientID in the options
 | ManagedIdentityCredential: managed identity timed out. See
 | https://aka.ms/azsdk/go/identity/troubleshoot#dac for more
 | information
 | AzureCLICredential: ERROR: AADSTS50173: The provided grant
 | has expired due to it being revoked, a fresh auth token is
 | needed. The user might have changed or reset their password.
 | The grant was issued on '2024-06-21T14:08:41.7484466Z' and
 | the TokensValidFrom date (before which tokens are not valid)
 | for this user is '2024-11-04T11:11:10.0000000Z'. Trace ID:
 | xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx Correlation ID:
 | yyyyyyyy-yyyy-yyyy-yyyy-yyyyyyyyyyyy Timestamp: 2025-01-22
 | 13:55:44Z
 | Interactive authentication is needed. Please run:
 | az login --scope https://vault.azure.net/.default
 |
 | AzureDeveloperCLICredential: please run "azd auth login"
 | from a command prompt to authenticate before using this
 | credential

Recovery failed because no master key was able to decrypt the file. In
order for SOPS to recover the file, at least one key has to be successful,
but none were.

While usually I did run az auth login it seems that now you need Azure Developer CLI. Hence the solution for above problem ist

Velero using Azure Managed Identity

Mon, 20 Jan 2025 14:00:00 +0200

What is velero?

velero is a backup and restore tha allows you to

Take backups of your cluster and restore in case of loss.
Migrate cluster resources to other clusters.
Replicate your production cluster to development and testing clusters.

What we want to do?

We are currently running velero. However we want to use Azure Managed Identity. Currently our setup is that we have 2 different resource groups

1 for the storage account
1 for aks (disk snaphost).

Looking trough the documentation, it is not very clear to me on how to specify different resource groups using ./credentials-velero. Actually it’s quite simple. So if you are installing velero with helm you need to define the following in the values.yaml

DevOps vs. Platform Engineering

Mon, 06 Jan 2025 10:18:00 +0200

Platform Engineering is the new kid on the block. But I have seen, there is some confusion with the term Devops. Here I want to give a quick clarification and also some cool links where you can read about Platform Engineering

DevOps vs. Platform Engineering

DevOps is a software development approach that promotes collaboration between development and operations teams
Platform engineering gives DevOps teams a centralized platform for their tools and workflows.

Platform engineering teams take over the design, implementation and maintenance of these tools and workflows. The platform team uses tool experts to understand developer needs, select the best tools for the required tasks, perform integrations and automations, and troubleshoot and maintain the established platform over time.

How to organize application and application sets in ArgoCD

Wed, 01 Jan 2025 10:00:00 +0200

Key leanings from GitOps Enterprise: Learn how to use Argo CD in multi-tenant installations, how to create preview environments and more.

Category	Description	Type	Change Frequency	Target User	What they are for
1	Developer Kubernetes Manifest	Helm, Kustomize or plain manifest in git	Very often	Developers mostly	Describe the state of any application to any of your organization environments (QA/Staging/Production etc)
2	Developer Argo CD Manifest	Argo CD app and Application Set	Almost never	Operators/Developers	Policy configurations referencing the source of truth for an application i.e Category 1
3	Infrastructure Kubernetes manifests	Usually external Helm charts	Sometimes	Operators	Describe the state of any infrastructure application (e.g. logging, monitoring, …) to any of your organization environments (QA/Staging/Production etc)
4	Infrastructure Argo CD manifests	Argo CD app and Application Set	Almost never	Operators	cy configurations referencing the source of truth for an infrastructure application i.e Category 3

Ship Rancher API Audit Logs from AKS clusters

Thu, 24 Oct 2024 10:00:00 +0200

As reader of my blog you know we are using Rancher logging app. While we migrated the Rancher (Upstream) cluster from RKE to AKS, we cannot use the built in log collection and shipping for audit logs.

External Secrets Operator and Azure Identity Workload

Wed, 18 Sep 2024 10:00:00 +0200

What is Azure AD Workload Identity?

Azure AD Workload Identity allows you the use of a Managed Identity to access resources in Azure

Azure AD Workload Identity for Kubernetes integrates with the capabilities native to Kubernetes to federate with external identity providers

The kubernetes cluster becomes a token issuer, which issues tokens to Kubernetes Service Accounts. These service account tokens can be configured to be trusted on Azure AD applications or user-assigned managed identities.

Azure Managed Identity

Sun, 01 Sep 2024 10:00:00 +0200

Managing secrets and credentials manually is cumbersome and error prone so what if you can make that easier? At least in azure you can do so using managed identities.

What is managed identities?

Resources usually need secrets (secrets7credentials/certificates/keys) to communicate with other resources. While (in Azure) you can securely store these in an Azure Key Vault, as a developer you still have to configure your applications accordingly - means you probably copy around these secrets. With Managed identities this is not necessary anymore, as it will provide an automatically managed identity in Microsoft Entra ID for applications. An application can use this to connect to resources, by obtaining Microsoft Entra tokens without having to manage any credentials.

Terraform complains with "EvalSymlinks: too many links" when on a Windows roaming profile

Wed, 31 Jan 2024 10:00:00 +0200

Working with terraform on Windows can be a pain, even more when using remote profiles.

Problem

While try to do a terraform init you get the following error

│ Error: Failed to install provider
│
│ Error while installing hashicorp/azurerm v3.83.0: failed to compute
│ checksum for
│ C:\Users\papa~1\AppData\Local\Temp\4\terraform-provider1701116033:
│ EvalSymlinks: too many links

The problem seem to occur if the profile is a roaming user profiles.

Solution

Use Local Folder

This happens on Windows servers, where the profile is not local. To fix it you need a local folder where you have read access, and then configure the following variables either in PS or bash

Terraform secrets with SOPS and Azure Keyvault

Tue, 24 Oct 2023 08:00:00 +0200

We are heavily using Terraform and and also Azure. However until now, we left out certain things cause they contain secrets which we don’t want to expose in the code. SOPS is a nice solution to solve that problem and keep things together what belongs together.

What is SOPS?

SOPS stands for Secrets OPerationS, and is an open-source text file editor that encrypts/decrypts YAML, JSON, ENV, INI and BINARY formats and encrypts with AWS KMS, GCP KMS, Azure Key Vault, age, and PGP.

Improvement of Atlantis workflow with azure by setting no_proxy automatically

Thu, 19 Oct 2023 15:00:00 +0200

While we are using Atlantis to to deploy changes to the Cloud infrastructure, we had the issue, that after each new setup we had to re-deploy the atlantis instance, cause we had to extend the no_proxy environment variable.

Why update the `no_proxy`?

Generally access to azure resources is going through public endpoints at first. Especially for sensitive stuff like keyvault and storage, this shall happen through private endpoints.

While traffic to public endpoints have to be routed through the webproxy, for privat links we have a direct connection, so it shall not go trough the proxy. So for each private link, the fqdn has to be added to the no_proxy so that calls from atlantis (or terraform) are redirected properly.

Git-hooks - Automatically check for secrets in your code

Mon, 16 Oct 2023 08:00:00 +0200

Who did not once in his/her life commit secrets to the code and instantly regretted it? Well we can use git-hooks to avoid that.

What is it?

Based on Git-hooks - a practical example with tf docs I extended the git-hooks so it check for secrets and aborts the commit if secrets are found. All you neds it After that also install gitleaks, which is a SAST tool for

detecting and preventing hardcoded secrets like passwords, api keys, and tokens in git repos. Gitleaks is an easy-to-use, all-in-one solution for detecting secrets, past or present, in your code.

Git-hooks - a practical example with tf docs

Wed, 27 Sep 2023 08:12:28 +0200

You are working on terraform code and want to ensure your documentation is updated after you made some changes. Why don’t you use git-hooks for that?

A long time ago I wrote about git-hooks, today I want to give you a practicle example on how I ensure my terraform documetation stays up-to-date.

terraform-docs

Not sure if you every heard of terraform-docs, but it’s a tool which generates terraform modules documentation in various formats. You can define a config file .terraform-docs.yml which defines how your documentations shall look like. My $HOME\.terraform-docs.yml looks as follows:

Hetzner Cloud referral (again)

Mon, 15 May 2023 07:40:12 +0200

As a long-time user of Hetzner as my hosting provider, I am very satisfied of their services. The do not provide a bunch of interesting offers but also an auction where you can reuse hardware for a very good price.

Server hardware can be reused even if a product is terminated. This is reasonable in terms of both economic and ecological aspects. Benefit from this advantage.

For the ones of you looking for a cloud service - similar to Google Cloud or Amazon - I recommend to look into the Hetzner Cloud which offers a very competitive pricing. If you decide to give it a try, you may use my Hetzner Referral Link so you get € 20 in cloud credits.

Reusable Github Workflows

Sat, 15 Apr 2023 12:17:56 +0200

Do you have multiple projects of the same topic, whicch use the same workflows? Then you might have a look into reusable workflows.

I have multiple ansible roles and I want to use the same workflows for all of these roles. This is when reusable workflows comes into play:

Rather than copying and pasting from one workflow to another, you can make workflows reusable. You and anyone with access to the reusable workflow can then call the reusable workflow from another workflow.

Blog on Wyssmann Engineering

User Bitwarden as ssh agent

NixOS for all my system - part 2

Overall approach

NixOS for all my system

Overall approach

Chezmoi externals behind a coorporate proxy

NixOS my new linux distro

Reason for NixOs

How to use separate environment configuration files ArgoCD

Problem

Copy of Failed to get the data key required to decrypt the SOPS file: The provided grant has expired due to it being revoked

Velero using Azure Managed Identity

What is velero?

What we want to do?

DevOps vs. Platform Engineering

DevOps vs. Platform Engineering

How to organize application and application sets in ArgoCD

Categories

Category 1

Ship Rancher API Audit Logs from AKS clusters

External Secrets Operator and Azure Identity Workload

What is Azure AD Workload Identity?

Azure Managed Identity

What is managed identities?

Terraform complains with "EvalSymlinks: too many links" when on a Windows roaming profile

Problem

Solution

Use Local Folder

Terraform secrets with SOPS and Azure Keyvault

What is SOPS?

Improvement of Atlantis workflow with azure by setting no_proxy automatically

Why update the no_proxy?

Git-hooks - Automatically check for secrets in your code

What is it?

Git-hooks - a practical example with tf docs

terraform-docs

Hetzner Cloud referral (again)

Reusable Github Workflows

Why update the `no_proxy`?