Creating an event-driven Infrastructure with Salt Reactor

In this September of Salt I have yet another salty article for you. Today we take a quick look at the Reactor subsystem of SaltStack. The Salt Reactor is a component of the Salt Master daemon that watches for a given list of events and reacts by triggering additional procedures. The events can be sent by Salt itself during software configuration, an admin on the command line interface or whatever comes to your mind.

Example:

  1. You or your cloud controller creates a new VM
  2. The VM boots and starts the preinstalled Minion daemon
  3. The Master detects this new Minion, sends a notification to the admins and tells the new Minion to install an Apache httpd webserver on its host
  4. The Minion receives this command, installs the software and notifies the Master that the webserver has been started and is ready for traffic
  5. The Master tells all Minions running a loadbalancer to trigger a reconfiguration to add the new webserver to the list of available backends

Sounds like a lot of work? Nope! We are going to build a PoC for this setup now in less than 30 minutes!

Preparation

Before we start creating our event-driven infrastructure make sure you have enough hardware resources to run two VMs (each 1 CPU + 512MB RAM) on your workstation or server.

Start both VMs, install an operating system (like Debian 7 or CentOS 7) and install a Salt Minion (v2015.8.0). One VM also needs a Salt Master (v2015.8.0).

Configuring the Salt Reactor

We are configuring the Salt Reactor by updating the Salt Master configuration file and telling both Minions to contact the Salt master at a given IP address:

The autosign file will contain a pattern of VM hostnames whose verification requests will be accepted automatically (salt-key --accept-all). The directory /srv/salt/states/ will contain all Salt state files and /srv/salt/reactor/ will contain all Reactor files.

You can see that the reactor setting is a list of event name patterns where each of them specifies a list of Reactor files like basic.sls. Reactor files a very similar to Salt State files except that they are interfacing the Python class LocalClient that is used to communicate with the Salt Master.

Let's take a look at the events and the configured reactions:

Event name patternEvent DescriptionReaction
salt/minion/*/starta Minion daemon startedRun basic.sls which appends a string to the file /tmp/salt.reactor.log (cmd.run) and tells the new minion to apply Salt states from the state file saltbox.reactor_apache_http from /srv/salt/states/ (state.sls) .
salt/job/*/ret/*a Minion completed a jobRun job_ret.sls which appends a string to the file /tmp/salt.reactor.log (cmd.run).
frontend/loadbalancer/pool/updatea Minion sent a custom event from a state file or CLIRun lb_pool_update.sls which appends a string to the file /tmp/salt.reactor.log (cmd.run). Hint: We could start another state.sls job for all loadbalancing hosts here.

Reactor files can contain YAML+jinja or can be written in pure Python. The minion hostname (FQDN) is printed using {{ data['id'] }} and {{ data['data']['new_web_server_ip'] }} shows the access of custom event data.

Now that we have reactor files in place, we are going to deploy an arbitrary Salt state file:

These states ensure that the package httpd is installed, the file /etc/httpd/conf/httpd.conf is owned by root:root and the file mode 644 is set. The service httpd should be running.
For every change on the service state like a service restart/start, an event is sent with the tag/name frontend/loadbalancer/pool/update and the webserver IP address as custom event data.

Reactor in Action

After you have deployed all of our files (see above) we should be able to see the Reactor in action. Create two terminal sessions to (1) run the command salt-run state.event pretty=True to watch for all Salt events and to (2) tail -f /tmp/salt.reactor.log.

Start or restart the salt-minion on the second VM e.g. with service salt-minion restart and see for entries in the logfile /tmp/salt.reactor.log.

You should see something like this:

When you restart the Salt minion another time you only see the log entry containing minion started. Since the service httpd is already running, no frontend/loadbalancer/pool/update event should be sent to the Master.

If you have problems getting the Reactor working or would like to do a very-quick start, simply switch to saltbox-vagrant that comes with a preconfigured Salt environment including a Reactor configuration.

Have fun!

Comments