In the previous blog post of this series we saw how to configure Packer to create images that can be used by Vagrant to setup new VMs. In this final blog post we create an example Vagrant environment to deploy Elasticsearch and Kibana.
- Project Git Repository
Like most of my existing vagrant environments you will find the following files on my Github account called
- Packer/ Vagrant image
We won't use a self-created Packer image because in this blog post I will focus on Vagrant and how it helps us to bootstrap a small infrastructure based on the ELK stack. The public image that will be used is chef/debian-7.8.
The main actor comes in version 1.7.2 along with a prepared set of provision shell scripts and a generic Vagrantfile. I use it for almost all Vagrant projects and provides a Vagrant configuration file called
nodes.yaml. VirtualBox 4.3.24 will be used as hypervisor.
I'm going to use the configuration management system of my systems management platform of my choice (what a sentence..) which is SaltStack.
- Elasticsearch & Kibana
The software that will be installed by Salt is the search & analytics server Elasticsearch and the visualization webinterface Kibana. To learn how to use Kibana you may want to read an extensive Kibana 4 Tutorial by two of my colleagues Anna and Tim Roes. The SaltStack formulas that setup Java, Elasticsearch and Kibana are available on Github, too.
We start with creating a Git repo, add the Vagrant template
iac-vagrant as Git submodule (tip: always use Git submodules with caution) and create a standard hierarchy of directories that will be used by
After we've done this we'll define Salt pillar data (parameters for formulas, a.k.a CMDB) and define a Salt top file that tells Salt which formulas to use. We download tarballs from the Internet containing Java and Kibana.
The final step is to execute
vagrant up and Vagrant will (hopefully) do the rest for us.
Preparing the Vagrant environment
The first step is to create a Git Repo, add
iac-vagrant as submodule, create a bunch of directories & files and download Java and Kibana:
Now let's add more Git repos as submodules: A repo containing custom Salt modules and all needed Salt formulas that will do black magic.
Finally let Git download submodules of our submodules (that will take some time). :-)
We add Salt pillar data and a top file that will tell Salt which formulas and configuration data to use:
Now place an
iac-vagrant configuration file called
nodes.yaml in the root directory of the Git repo:
Spinning up the VM
We've done everything that is needed to create the Vagrant VM and let Salt install the ELK stack. Change into the vagrant directory and let the game begin:
It's now time to make coffee or tea. You've gained ~ 5 minutes of free time. :)
Hooray, there we are. Vagrant has cloned the base image, created a VirtualBox VM and executed the provision scripts. The latter had installed and configured some basic tools and Salt. Let's SSH into the VM and call the function
state.highstate of the Salt
state execution module in test mode:
Salt just tells us it could have applied all states. Now enter god mode and remove the parameter
test=True. Time for the second cup of coffee/ tea:
This command took 2 minutes (for me). We now should have a running Elasticsearch instance listening on
Using Elasticsearch and Kibana
Let's feed our hungry Elasticsearch instance with some data from mockaroo.com that I've modified to work with the Elasticsearch bulk API:
You can stop the import stream after some minutes to have enough documents.
Salt has installed the Elasticsearch kopf plugin for us. Determine the IP address of the interface
ip address and open
http://<IP_ADDRESS_OF_ETH1>:9200/_plugin/kopf/#!/cluster in your web browser:
To display those documents in our browser we run Kibana on the command line and send it to background (CTRL + Z). Unfortunately I haven't added an init script to the formula yet, so for our small PoC we need to live with executing Kibana manually.
You should now be able to reach Kibana in your browser on
http://<IP_ADDRESS_OF_ETH1>:5601. Kibana asks us to create a default index pattern. We set the pattern/ name
You can now open the Discover page and study the schema of our documents. Given this sample data we can create some nice visualisations like the following:
We've successfully deployed a small Elasticsearch/ Kibana stack using Vagrant! That was easy, right? The steps we've done were:
- Initialized a new Git repository
- Downloaded Java and Kibana
iac-vagrantand Salt modules/ formulas as Git submodules
- Added directories and files that are used by Salt to install Java, Elasticsearch and Kibana
- Added a config file for
- Created the Vagrant VM using
- Executed the Salt function
state.highstateto install the whole software stack
- Executed the Kibana binary and sent the process into the background
- Added sample data using curl
- Created visualisations in Kibana in our web browser
We now can do whatever we want with Elasticsearch and Kibana. When you broke something or want to start from scratch you simply need to
vagrant destroy elastic1 && vagrant up in the vagrant directory or use Vagrant VirtualBox snapshot-plugin.
The Git repo that I've created can be cloned from Github using
git clone https://github.com/bechtoldt/demo-elasticsearch-vagrant.git --recursive. You need to download Java and Kibana manually using wget (see above).
Feel free to leave a comment with your feedback or ask a question if something didn't work for you. Please also report typos and missing commas. :P