Introduction to Saltstack

Posted in automation on April 16, 2018 by Adrian Wyssmann ‐ 10 min read


Salt is a modular configuration management system which uses a domain specific language to manage and automate your infrastructure. Most common format is YALM and Jinja for templating – but other formats like json are supported as well – see here and here. Salt commands and states run the same regardless the target system (os, physical hardware or cloud). As mentioned it is modular so it has a pluggable subsystem with a variety of plugins.

Saltstack Architecture
Saltstack Architecture

Although Saltstack supports agentless setup the most common setup is the one with master and slaves. The slaves are called minions and are commanded and controlled from one or more central command server(s) called master.

Commands are normally issued to the minions (via the master) by calling a client script simply called, `salt`.
Salt features a plugable transport system to issue commands from a master to minions. The default transport is ZeroMQ.

To communicate, minions have to subscribe to masters using a secure and encrypted connection via port 4505 and 4506. Interesting to know is that connections are initiated by the Salt minion, so no incoming ports opn the minions have to be opened (reduction of the attack vector). You may read this to get a bit more details.


Here a summary and reference to the most important terms

Term Description Reference
Master Commander and controller of salt minions configuring the salt master
Minion daemon, receives commands from a remote Salt master. architecture salt-minion
Publisher Process is responsible for sending commands over the designated transport to connected minions. architecture
State A reusable declaration that configures a specific part of a system. Each state is defined using a state declarationStates are written down in salt files – a file with an SLS extension that contains one or more state declarations. SaltStack Configuration Management Tutorial
State Declaration A top level section of a state file that lists the state function calls and arguments that make up a state. Each state declaration starts with a unique ID. SaltStack Configuration Management Tutorial
State Functions Commands that you call to perform a configuration task on a system. [SaltStack Configuration Management Tutorial](
Formula A collection of Salt state and Salt pillar files that configure an application or system component. Most formulas are made up of several states spread across multiple state files. SaltStack Configuration Management Tutorial
Pillar Pillars is the way saltstack provides custom variables and data for minions as key:value-pairs. This data can be ports, file paths, configuration parameters and passwords. Pillars are stored in a pillar file – a file with an SLS extension. SaltStack Configuration Management TutorialStoring Static Data in the Pillar
Grains Grains are static information SaltStack collects about the underlying managed system. SaltStack collects grains for the operating system, domain name, IP address, kernel, OS type, memory, and many other system properties.You can add your own grains to a Salt minion by placing them in the /etc/salt/grains file on the Salt master, or in the Salt minion configuration file under the grains section. SaltStack Fundamentals TutorialGrains
Environments A directory hierarchy which contain a top file and a set of state files to configure systems. the top file
Target A grouping of machines which will have a set of states applied to them. the top file
Runners Salt runners are convenience applications executed with the salt-run command. Documentation “Runners”

Salt fileserver and fileserver backend

Salt comes with a simple file server with the main intention to present files for use in the Salt state system. The file server is a stateless ZeroMQ server that is built into the Salt master and can integrate different file server backends to allow Salt use files from different resources – it also supports the use of multiple backends, which in case used, all files are merged into a single virtual filesystem. Per default the fileserver backend serves files from the Master’s local filesystem. A backend is also called roots and fileserver environments are defined using the file_roots configuration option.

  - /srv/salt/
  - /srv/salt/dev/services
  - /srv/salt/prod/services
  - /srv/salt/prod/states

Although there are different backends I belief is worthwhile to spend some moments with the gitfs backend.

gitfs backend

The gitfs backend allows Salt to serve files from git repositories. It can be enabled by adding git to the fileserver_backend list, and configuring one or more repositories in gitfs_remotes.

  - git

Interestingly you can have multiple repository which can contain the same or different files. Salt will attempt to lookup the requested file from each gitfs remote repository in the order in which they are defined in the configuration. Also good to know, is that branches and tags become Salt fileserver environments.

For details checkout the official documentation: Git Fileserver Backend Walktrough


An environment is directory structure containing a top file and a set of state files. Each environment – yes there can be multiple – have a top.sls file which defines the targets of the environments (hosts or groups of hosts). The targets itself defines the states to be applied to the targets. Per default the salt file server use the mandatory environment base, which must be defined and is used to download files when no environment is specified. Environments allow for files and sls data to be logically separated, but environments are not isolated from each other.

The Salt file server is an environment aware file server. This means that files can be allocated within many root directories and accessed by specifying both the file path and the environment to search. The individual environments can span across multiple directory roots to create overlays and to allow for files to be organized in many flexible ways.

Below you can see an example which defines three environments, whereas each has its own directory assigned to it:

# Example configuration with multiple environments YAML
    - /srv/salt/dev
    - /srv/salt/qa
    - /srv/salt/prod

The top file then references it:

# Example top.sls file with reference to the three environments YAML
    - webserver
    - db
    - webserver
    - db
    - webserver
    - db

What the above basically says is that we have in each environment webservers which get the webserver state and db servers which gets the db state. In case of git as backend, we learned above that also branches and tags are environments. So let’s assume we branche our dev environment into a feature/dev branch we could call the to show the top data the minions will use

[saltuser@saltmaster ~]$ sudo salt '*' state.show_top saltenv=feature/test

(!) Minions may be pinned to a particular environment by setting the environment value in the minion configuration file


A state is a re-usable configuration template written in YAML that describes everything required to put a system component or application into a known configuration. Each state is represented with a state file ending with .sls. Each state has one or more sections that call a state function and it’s relevant parameters. Each such funtion is ointroduced with a unique identifier

    - name: D:/java
    - source: https://nexus.intra/service/local/repo_groups/wyssmann/content/com/wyssmann/java/jdk/1.8.0_162/
    - source_hash: 2c21a3356055d02d0784bd501a7f1a06e1c54b68

Above state is called jdk(.sls) and contains a unique identifier java_jdk. It uses the archive.extracted funtion to download and extract a customized Java package into D:/java ona windows minion. You can now apply the state to the minion by running following command on the master which also gives you immediately feedback on the activities salt performs.

[saltuser@saltmaster ~]$ sudo salt 'wintest' state.show_top saltenv=feature/test
                  - jdk1.8.0_162/release
                  - jdk1.8.0_162/THIRDPARTYLICENSEREADME-JAVAFX.txt
                  - jdk1.8.0_162/THIRDPARTYLICENSEREADME.txt
Summary for wintest
Succeeded: 1 (changed=1)
Failed:    0
Total states run:     1
Total run time:   7.594 s

States can do a lot like managing files, services and so forth and even get more flexible when using variables. Therefore I recommend further readings:


As learned above the fileserver configures the root for the state tree by file_roots, or in case of git_backend by gitfs_remotes, there are similar functions for pillars: pillar_roots or git_pillar. This also means that pillars must not be in a subdirectory of the state tree and pillar data is not bound to sls files but also can be retrieve from external sources. As for states tree, there is a top file top.sls declared in which enironment, which minion receives which data – the data is defined on the Salt Master and the passed trough only to the relevant minions.

# Example pillar top.sls YAML
    - packages
    - match: grain
    - servers

The above example defined that all minions will have the pillar data found in the packages pillar and in the dev environment all minions which run Debian as their os – this is called grains and I will explain the concept later – will also have the pillar data found in the servers. Pillar data are key:value-pairs so the packages pillar may look like this

# Example pillar 'packages' YAML
{% if grains['os'] == 'RedHat' %}
apache: httpd
git: git
{% elif grains['os'] == 'Debian' %}
apache: apache2
git: git-core
{% endif %}

The example show that depending on some minion specifics – the os in this case – the package name for apache or git is different. Now the data can be used in a state files or modules via the pillar dictionary

# Example usage of pillar data in state file YAML
    - name: {{ pillar['apache'] }}

As stated here there are two ways to get pillar data. So assume we have a more complex structure like this

# Example pillar with complex structure YAML
  name: "example"
    name: "testuser"
    password: "test"

Pillar data can be access in either of these ways:

# Example pillar with complex structure YAML
name: {{ salt['pillar.get']('example:user:name') }}
# or
name: {{ pillar['example']['user']['name'] }}

It should be noted that within templating, the pillar variable is just a dictionary. This means that calling pillar.get() inside of a template will just use the default dictionary .get() function which does not include the extra : delimiter functionality. It must be called using the above syntax (salt['pillar.get']('foo:bar:baz','qux')) to get the salt function, instead of the default dictionary behavior.

The following is important to know:

  • When updating pillar data its a good practice to trigger the minion to fetch the data form the master

    salt '*' saltutil.refresh_pillar
  • For testing or for ad hoc management, you can pass Salt pillar values directly on the command line using

    salt '*' state.apply ftpsync pillar='{"ftpusername": "test", "ftppassword": "0ydyfww3giq8"}'

For more specific checkout the tutorial, pillar walktrough and the documentation.


Grains is a salt interface to derive information about the underlying system like os, IP address and so forth. It is available to Salt modules and components. All grains are resolve to lowercase letters. Grains can also be assigned to a minion either in the minion config file or in the /etc/salt/grains file on the minion.

[saltuser@saltmaster ~]$ salt '*'
    - SSDs
    - biosreleasedate
    - biosversion
    - cpu_flags
    - cpu_model
    - cpuarch
    - disks
    - dns

Grains can be used to define “matching” rules in states or pillars – I recommend to read trough the linked pages to get a better understanding. A simple example can already be seen under the chapter “Pillar” where the respective pillar data depends on the os of the minion. Further reading