Configuration Management Systems using a Declarative Programming Model do not scale


I'm a SysOp/ Systems Engineer for ~ 4 years now. I've primarily worked on Debian, CentOS and RHEL. I have more or less basic knowledge about programming with program and script languages like bash, Python, C++, PHP, ruby, PowerShell and VBScript. I started to setup and manage Linux servers using small shell scripts, continued with writing a (damn) large and time-consuming configuration management system using bash scripting language. Then moved to C++, added many features and then finally discovered a configuration management project that is actually worth spending learning-time on it and has a considerable number of community members/contributors.

I've worked 1-2 years with this configuration management system so far, changed my employer and then started to spend time with another configuration management system for customers.

Both mgmt systems differ in major ways. The first acts as system management platform, providing various subsystems like remote execution on CLI/REST API and configuration management. The other mgmt system focuses on configuration management. This articles covers a specific topic with configuration management (a.k.a CM) context only.

Configuration Management explained

When I'm speaking about configuration management systems I mean software (mostly FOSS) that applies changes to a running operating system. Such changes can be:

  • install a webserver/mailserver/file editor/driver
  • create local system users/groups
  • configure the webserver by creating configuration files with a specific content
  • start the webserver so that it handles incoming HTTP requests

Most of the CM systems that I know are used by writing files that describe the changes that we want to apply to operating systems/servers. Every CM system requires a specific language syntax for change description/definition.

A basic workflow looks like the following:

  1. You write files that describe system changes and/or the desired state of a resource (software package, file, user, etc.)
  2. The CM system reads the files and translates them into an internal ToDo list (do task X then task Y then task Z)
  3. The CM system processes the ToDo list on a defined set of hosts

A few existing CM systems are:

  • (R)?ex (pronounced Rex)
  • Ansible
  • CFEngine
  • Chef
  • Puppet
  • SaltStack

Describing System Changes

There are two popular concepts describing system changes:

  1. A step-by-step list which changes should be applied in a specific order
  2. An unordered list of changes with relations and dependencies to each other

Example for concept (1):

  1. Install a package with the name apache2, choose the latest version (apt-get install apache2)
  2. Then add a local system user with the username john (useradd -s /bin/zsh -m john)
  3. Then create a file with the file path /var/www/index.html (touch /var/www/index.html), set the local user john and the group admins as file owner (chown john:admins /var/www/index.html), make it modifiable by the user john only (chmod 644 /var/www/index.html), set the words Hello World! as file content (echo "Hello World!" > /var/www/index.html)
  4. Then start a program with the name apache2 and set it into background mode (service apache2 start)

Let's compare this with concept (2):

  • Add a local system user with the username john after a package with the name apache2 has been installed and before creating a file with the file path /var/www/index.html
  • Start a program with the name apache2 after creating a file with the file path /var/www/index.html
  • Install a package with the name apache2 before adding a local system user with the username john
  • Create a file with the file path /var/www/index.html after adding a local system user with the username john and before starting a program with the name apache2

Concept (1) is called imperative, concept (2) is called declarative and is no April's fool.

Let's specify these changes using language syntax that is provided by two of the CM systems that I mentioned above:

A CM system that primarily uses the imperative model:

A CM system that uses the declarative model:

While the imperative model reads the definitions how they are written in the file, the declarative model doesn't respect the order of the definitions (whether they are correct or not) and tries to define the step-by-step list automatically. Arguments like before and require helps the file parser to find dependencies. One of the dependencies that we would have defined using before and require can be determined by the parser automatically:

When a file /var/www/index.html should be owned by the user john we need to create the user john before specifying the user john as owner of the file /var/www/index.html.

Sounds clever and useful? Yes it is! But it's one of the very few examples where implicit dependencies can be determined by the parser itself. Did I ever mentioned that most of the software code compilers people write for the last decades didn't made it into production? A German would say Das Leben ist kein Ponyhof.

A statement I found on the Net tells you the truth:

A fast-moving SaaS on a schedule of continuous rollouts will appreciate the flexibility and scalability of a well-implemented declarative CM solution.


This statement uses the term well-implemented declarative CM solution. I haven't seen a well-implemented declarative CM solution so far. Instead I see large-sized IT projects spending lots of money by trying to define explicit dependencies between resources like the ones above to setup a large number (200-1000) of servers. It hurts when it scales became one of my favorite phrases. A declarative language seems easier to learn since you don't have to worry about the resource definition order. On the other hand you need know how to deploy complex software that requires a long list of configuration steps beforehand. As larger your project grows as larger will be the hacks that define explicit resource dependencies.


This fact leads to my conclusion that configuration management systems that use declarative programming to define resources or system changes only simply don't scale when it comes to large projects - whatever large means. As far as I can say after working for two years with imperative and declarative models, using the declarative model in large projects is difficult to handle/maintain, time-consuming and expensive when it comes to money.