Introduction to Saltstack

Intro #

Salt is a modular configuration management system which uses a domain specific language to manage and automate your infrastructure. Most common format is YALM and Jinja for templating – but other formats like json are supported as well – see here and here. Salt commands and states run the same regardless the target system (os, physical hardware or cloud). As mentioned it is modular so it has a pluggable subsystem with a variety of plugins.

Although Saltstack supports agentless setup the most common setup is the one with master and slaves. The slaves are called minions and are commanded and controlled from one or more central command server(s) called master.

Commands are normally issued to the minions (via the master) by calling a client script simply called, `salt`.
Salt features a plugable transport system to issue commands from a master to minions. The default transport is ZeroMQ.

To communicate, minions have to subscribe to masters using a secure and encrypted connection via port 4505 and 4506. Interesting to know is that connections are initiated by the Salt minion, so no incoming ports opn the minions have to be opened (reduction of the attack vector). You may read this to get a bit more details.

Terminology #

Here a summary and reference to the most important terms

Salt fileserver and fileserver backend #

Salt comes with a simple file server with the main intention to present files for use in the Salt state system. The file server is a stateless ZeroMQ server that is built into the Salt master and can integrate different file server backends to allow Salt use files from different resources – it also supports the use of multiple backends, which in case used, all files are merged into a single virtual filesystem. Per default the fileserver backend serves files from the Master’s local filesystem. A backend is also called roots and fileserver environments are defined using the file_roots configuration option.

file_roots:
  base:
  - /srv/salt/
  dev:
  - /srv/salt/dev/services
  prod:
  - /srv/salt/prod/services
  - /srv/salt/prod/states

Although there are different backends I belief is worthwhile to spend some moments with the gitfs backend.

gitfs backend #

The gitfs backend allows Salt to serve files from git repositories. It can be enabled by adding git to the fileserver_backend list, and configuring one or more repositories in gitfs_remotes.

fileserver_backend:
  - git

Interestingly you can have multiple repository which can contain the same or different files. Salt will attempt to lookup the requested file from each gitfs remote repository in the order in which they are defined in the configuration. Also good to know, is that branches and tags become Salt fileserver environments.

For details checkout the official documentation: Git Fileserver Backend Walktrough

Environments #

An environment is directory structure containing a top file and a set of state files. Each environment – yes there can be multiple – have a top.sls file which defines the targets of the environments (hosts or groups of hosts). The targets itself defines the states to be applied to the targets. Per default the salt file server use the mandatory environment base, which must be defined and is used to download files when no environment is specified. Environments allow for files and sls data to be logically separated, but environments are not isolated from each other.

The Salt file server is an environment aware file server. This means that files can be allocated within many root directories and accessed by specifying both the file path and the environment to search. The individual environments can span across multiple directory roots to create overlays and to allow for files to be organized in many flexible ways.

Below you can see an example which defines three environments, whereas each has its own directory assigned to it:

# Example configuration with multiple environments YAML
file_roots:
  dev:
    - /srv/salt/dev
  qa:
    - /srv/salt/qa
  prod:
    - /srv/salt/prod

The top file then references it:

# Example top.sls file with reference to the three environments YAML
dev:
  'webserver*':
    - webserver
  'db*':
    - db
qa:
  'webserver*':
    - webserver
  'db*':
    - db
prod:
  'webserver*':
    - webserver
  'db*':
    - db

What the above basically says is that we have in each environment webservers which get the webserver state and db servers which gets the db state. In case of git as backend, we learned above that also branches and tags are environments. So let’s assume we branche our dev environment into a feature/dev branch we could call the state.show-top to show the top data the minions will use

[saltuser@saltmaster ~]$ sudo salt '*' state.show_top saltenv=feature/test

(!) Minions may be pinned to a particular environment by setting the environment value in the minion configuration file

States #

A state is a re-usable configuration template written in YAML that describes everything required to put a system component or application into a known configuration. Each state is represented with a state file ending with .sls. Each state has one or more sections that call a state function and it’s relevant parameters. Each such funtion is ointroduced with a unique identifier

java_jdk:
  archive.extracted:
    - name: D:/java
    - source: https://nexus.intra/service/local/repo_groups/wyssmann/content/com/wyssmann/java/jdk/1.8.0_162/jdk-1.8.0_162-bin.zip
    - source_hash: 2c21a3356055d02d0784bd501a7f1a06e1c54b68

Above state is called jdk(.sls) and contains a unique identifier java_jdk. It uses the archive.extracted funtion to download and extract a customized Java package into D:/java ona windows minion. You can now apply the state to the minion by running following command on the master which also gives you immediately feedback on the activities salt performs.

[saltuser@saltmaster ~]$ sudo salt 'wintest' state.show_top saltenv=feature/test
....
                  - jdk1.8.0_162/release
                  - jdk1.8.0_162/THIRDPARTYLICENSEREADME-JAVAFX.txt
                  - jdk1.8.0_162/THIRDPARTYLICENSEREADME.txt
  
Summary for wintest
------------
Succeeded: 1 (changed=1)
Failed:    0
------------
Total states run:     1
Total run time:   7.594 s

States can do a lot like managing files, services and so forth and even get more flexible when using variables. Therefore I recommend further readings:

• Fundamentals – Tutorial for Salt basics
• States Modules – How to write states
• How do I use Salt states? – Complete tutorial which also covers complex states, templating etc.
• Remote execution – Running pre-defined or arbitrary commands on remote hosts

Pillars #

As learned above the fileserver configures the root for the state tree by file_roots, or in case of git_backend by gitfs_remotes, there are similar functions for pillars: pillar_roots or git_pillar. This also means that pillars must not be in a subdirectory of the state tree and pillar data is not bound to sls files but also can be retrieve from external sources. As for states tree, there is a top file top.sls declared in which enironment, which minion receives which data – the data is defined on the Salt Master and the passed trough only to the relevant minions.

# Example pillar top.sls YAML
base:
  '*':
    - packages
dev:
  'os:Debian':
    - match: grain
    - servers

The above example defined that all minions will have the pillar data found in the packages pillar and in the dev environment all minions which run Debian as their os – this is called grains and I will explain the concept later – will also have the pillar data found in the servers. Pillar data are key:value-pairs so the packages pillar may look like this

# Example pillar 'packages' YAML
{% if grains['os'] == 'RedHat' %}
apache: httpd
git: git
{% elif grains['os'] == 'Debian' %}
apache: apache2
git: git-core
{% endif %}

The example show that depending on some minion specifics – the os in this case – the package name for apache or git is different. Now the data can be used in a state files or modules via the pillar dictionary

# Example usage of pillar data in state file YAML
apache:
  pkg.installed:
    - name: {{ pillar['apache'] }}

As stated here there are two ways to get pillar data. So assume we have a more complex structure like this

# Example pillar with complex structure YAML
example:
  name: "example"
  user:
    name: "testuser"
    password: "test"

Pillar data can be access in either of these ways:

# Example pillar with complex structure YAML
name: {{ salt['pillar.get']('example:user:name') }}
# or
name: {{ pillar['example']['user']['name'] }}

It should be noted that within templating, the pillar variable is just a dictionary. This means that calling pillar.get() inside of a template will just use the default dictionary .get() function which does not include the extra : delimiter functionality. It must be called using the above syntax (salt['pillar.get']('foo:bar:baz','qux')) to get the salt function, instead of the default dictionary behavior.

Source: https://docs.saltstack.com/en/latest/topics/pillar/index.html

The following is important to know:

• When updating pillar data its a good practice to trigger the minion to fetch the data form the master

  salt '*' saltutil.refresh_pillar
  

• For testing or for ad hoc management, you can pass Salt pillar values directly on the command line using

  salt '*' state.apply ftpsync pillar='{"ftpusername": "test", "ftppassword": "0ydyfww3giq8"}'
  

For more specific checkout the tutorial, pillar walktrough and the documentation.

Grains #

Grains is a salt interface to derive information about the underlying system like os, IP address and so forth. It is available to Salt modules and components. All grains are resolve to lowercase letters. Grains can also be assigned to a minion either in the minion config file or in the /etc/salt/grains file on the minion.

[saltuser@saltmaster ~]$ salt '*' grains.ls
saltmaster.intra:
    - SSDs
    - biosreleasedate
    - biosversion
    - cpu_flags
    - cpu_model
    - cpuarch
    - disks
    - dns
...

Grains can be used to define “matching” rules in states or pillars – I recommend to read trough the linked pages to get a better understanding. A simple example can already be seen under the chapter “Pillar” where the respective pillar data depends on the os of the minion. Further reading

• fundamental components and concepts that you need to understand to use Salt
• index of salt commands