Current Challenges when working with the SaltStack Systems Management Platform

People often hear me saying only the best about SaltStack, but there are few things that should be known by everyone who is starting to get into touch with SaltStack or is considering using it in the future. I'm often asked about the positive aspects of Salt, so this blog posts aims to describe five challenges when working with SaltStack that I've noticed in the last years. Because you know, nobody is perfect (except Chuck Norris, he is Nobody).

The downsides of super-actively developed Open Source projects (1)

At the time of writing this there are ~ 9,200 issues tracked in the main project at Github, ~ 15,650 closed pull requests, ~ 1,130 contributors and ~ 5,300 Github users gave the project a Github star. Compared to other configuration management systems that are hosted on Github, you see SaltStack is based on these metrics one of the largest projects in this area. Needles to say that many very big companies are using it and contributing a lot of valuable code. The monthly project statistics should show you what I mean with super-actively.

Given the huge amount of code contributions you might get a vague idea of what is required to maintain such a large project. Although the amount of people that are writing automated tests is increasing very fast it is very common that the main development branch contains minor to major code changes that make it hard to develop on Salt. Often you need to wait for some days to have a working develop branch again to branch of. Fortunately there are many developers from the community but also SaltStack Inc. itself that are actively working on fixing these issues.

IMHO PRs sometimes don't receive as much review attention as they need. Although SaltStack Inc. is continuously improving the review process, code is merged into the develop or release branches that cause regressions or are of some kind of bad quality. I think Salt is on a good way to decrease such problems in the future. The employees of SaltStack Inc. are doing a good work and are always busy with reviewing a lot of PRs everyday.

SaltStack requires a well-implemented patch/ release management (2)

Another thing that might be caused by the fact that Salt is an extremely active project is that the release workflow of the release team has many gaps. Let me show you some things I realized in the last 1-2 years:

Test every major Salt release extensively before updating in production

Major releases that bring an incredible list of bug fixes, feature additions and improvements are tested by the community and SaltStack employees. Unfortunately sometimes it happens that new bugs are discovered a few hours/ days after releasing a new Salt version. Normally they are fixed within hours in source code, so you know very soon how to hotfix it. It is very annoying that there are some regression bugs that take months to fix. In such cases you need to provide a PR yourself to have it available in the next Salt release.

A rule that applies to every other software is very important to apply when using Salt: Test every major Salt release extensively before updating in production

You might need an individual Salt release management strategy

If you touch Salt internals on a regular basis to fix bugs or add enhancements you obviously take care of software packaging yourself. This applies also when you always want to use the latest version of Salt. Why? Sometimes it takes weeks after official Salt releases to receive packages for Enterprise Linux (RHEL/ CentOS) and GNU/ Linux Debian based derivatives. The reason for this is that some packages are maintained by the community itself on a volunteer basis.

It is a honourable work to spend personal free-time to maintain these packages but IMHO it is an unacceptable circumstance that people are unable to test and use Salt after it has been officially released to the public domain. Official DEBs/ RPMs should be maintained by SaltStack Inc. itself to preserve direct software access without creating own packages or installing from source. This problem needs to be fixed ASAP.

Fundamental features tend to change (3)

Most of the code is improved on a regular basis which is good. Unfortunately sometimes interfaces/ usage changes which requires changing important pieces of your Salt setup. I admit this happens very rarely for the last 1-2 years. As a more or less active contributor I see how the various workflows improve over the last years. The increasing amount of SaltStack Inc. employees are taking the whole open source project to a very professional level. I think Salt stability improves with the increase of commercial gain for the SaltStack Inc. company that has obviously benefits for the community, too.

Great possibilities come with great responsibilities (4)

Issues like #12483 show that it's sometimes necessary to check the use cases of features and whether they exactly work as expected. In this particular issue people discovered that the Salt branch/ environment merging isn't working as stated in the documentation. It took some time for the community to understand the problem and provide some proposals. It's still not closed but personally I don't remember that this issue affected me in any kind.

Another example where developers and users need some iterations of improvements is the development of Salt Formulas which is nothing more than a folder with a defined sub-folder structure containing states managing any resources of an operating system/ cloud/ software/ and so on.

As you see in my MySQL formula, Elasticsearch formula or in the officially maintained formulas on Github, you can use Salt state files with a template engine like Jinja to create very powerful state files. Given these facts you can also transform your formulas into a bunch of trash that isn't readable/ maintainable anymore. There are some official formulas on Github that you better don't use as a base to learn writing Salt state files. Simply concentrate on your initial goals and don't hesitate to leave room for improvements that can be applied when you really need them.

Don't take all best practices literally (5)

Even though I started to document my own best practices I encourage Salt beginners to question all those best practices since not everything on the Net will be useful for your particular project. This isn't Salt-specific but applies for example on the advice maintaining a lookup table/ map in Jinja syntax (that stated to be hard to read and maintain for me).


Salt has so many characteristics that makes it a great helper to manage today's challenges of infrastructure requirements:

  • Modularity
  • Several interfaces that make it easy to integrate in today's open source datacenter
  • Extensibility on various levels
  • Software design principles that are ready for the future
  • ...

Although this article highlights negative aspects of Salt my overall conclusions are that Salt is going to be one of the big players in systems management in the future. It is already in use by the enterprise that decided to use Salt in important areas that have direct business influence. If you plan to make your infrastructure ready for the next years you should definitely take a substantial look at SaltStack.

I hope this helps some engineers and /(Sys|Dev|Cloud|IT)?Ops/ that plan to use SaltStack in their datacenter.