Using Packer & Vagrant to bootstrap a testing infrastructure (Part 3)

In the previous blog post of this series we saw how to configure Packer to create images that can be used by Vagrant to setup new VMs. In this final blog post we create an example Vagrant environment to deploy Elasticsearch and Kibana.

Components

Project Git Repository

Like most of my existing vagrant environments you will find the following files on my Github account called demo-elasticsearch-vagrant.

Packer/ Vagrant image

We won't use a self-created Packer image because in this blog post I will focus on Vagrant and how it helps us to bootstrap a small infrastructure based on the ELK stack. The public image that will be used is chef/debian-7.8.

Vagrant

The main actor comes in version 1.7.2 along with a prepared set of provision shell scripts and a generic Vagrantfile. I use it for almost all Vagrant projects and provides a Vagrant configuration file called nodes.yaml. VirtualBox 4.3.24 will be used as hypervisor.

SaltStack

I'm going to use the configuration management system of my systems management platform of my choice (what a sentence..) which is SaltStack.

Elasticsearch & Kibana

The software that will be installed by Salt is the search & analytics server Elasticsearch and the visualization webinterface Kibana. To learn how to use Kibana you may want to read an extensive Kibana 4 Tutorial by two of my colleagues Anna and Tim Roes. The SaltStack formulas that setup Java, Elasticsearch and Kibana are available on Github, too.

The plan

We start with creating a Git repo, add the Vagrant template iac-vagrant as Git submodule (tip: always use Git submodules with caution) and create a standard hierarchy of directories that will be used by iac-vagrant.

After we've done this we'll define Salt pillar data (parameters for formulas, a.k.a CMDB) and define a Salt top file that tells Salt which formulas to use. We download tarballs from the Internet containing Java and Kibana.

The final step is to execute vagrant up and Vagrant will (hopefully) do the rest for us.

Preparing the Vagrant environment

The first step is to create a Git Repo, add iac-vagrant as submodule, create a bunch of directories & files and download Java and Kibana:

Now let's add more Git repos as submodules: A repo containing custom Salt modules and all needed Salt formulas that will do black magic.

Finally let Git download submodules of our submodules (that will take some time). :-)

We add Salt pillar data and a top file that will tell Salt which formulas and configuration data to use:

Now place an iac-vagrant configuration file called nodes.yaml in the root directory of the Git repo:

Spinning up the VM

We've done everything that is needed to create the Vagrant VM and let Salt install the ELK stack. Change into the vagrant directory and let the game begin:

It's now time to make coffee or tea. You've gained ~ 5 minutes of free time. :)

Hooray, there we are. Vagrant has cloned the base image, created a VirtualBox VM and executed the provision scripts. The latter had installed and configured some basic tools and Salt. Let's SSH into the VM and call the function state.highstate of the Salt state execution module in test mode:

Salt just tells us it could have applied all states. Now enter god mode and remove the parameter test=True. Time for the second cup of coffee/ tea:

This command took 2 minutes (for me). We now should have a running Elasticsearch instance listening on 0.0.0.0:9200.

Using Elasticsearch and Kibana

Let's feed our hungry Elasticsearch instance with some data from mockaroo.com that I've modified to work with the Elasticsearch bulk API:

You can stop the import stream after some minutes to have enough documents.

Salt has installed the Elasticsearch kopf plugin for us. Determine the IP address of the interface eth1 using ip address and open http://<IP_ADDRESS_OF_ETH1>:9200/_plugin/kopf/#!/cluster in your web browser:

kopf-02

To display those documents in our browser we run Kibana on the command line and send it to background (CTRL + Z). Unfortunately I haven't added an init script to the formula yet, so for our small PoC we need to live with executing Kibana manually.

You should now be able to reach Kibana in your browser on http://<IP_ADDRESS_OF_ETH1>:5601. Kibana asks us to create a default index pattern. We set the pattern/ name sampledata.

kibana-sampledata-index-pattern-02

You can now open the Discover page and study the schema of our documents. Given this sample data we can create some nice visualisations like the following:

First names by country Employees working in companies by countries Birth rate in countries per 3 years

Birth rate in countries per year Birth rate statistics per 10 years

Summary

We've successfully deployed a small Elasticsearch/ Kibana stack using Vagrant! That was easy, right? The steps we've done were:

  1. Initialized a new Git repository
  2. Downloaded Java and Kibana
  3. Added iac-vagrant and Salt modules/ formulas as Git submodules
  4. Added directories and files that are used by Salt to install Java, Elasticsearch and Kibana
  5. Added a config file for iac-vagrant
  6. Created the Vagrant VM using vagrant up
  7. Executed the Salt function state.highstate to install the whole software stack
  8. Executed the Kibana binary and sent the process into the background
  9. Added sample data using curl
  10. Created visualisations in Kibana in our web browser

We now can do whatever we want with Elasticsearch and Kibana. When you broke something or want to start from scratch you simply need to vagrant destroy elastic1 && vagrant up in the vagrant directory or use Vagrant VirtualBox snapshot-plugin.

The Git repo that I've created can be cloned from Github using git clone https://github.com/bechtoldt/demo-elasticsearch-vagrant.git --recursive. You need to download Java and Kibana manually using wget (see above).

Feel free to leave a comment with your feedback or ask a question if something didn't work for you. Please also report typos and missing commas. :P

Comments