Log Searching and Analysis with Tutum and an ELK

tutum and elk
It’s 3 am and you get a text notifying you that your website is not responding. You roll out of bed, double check your website and sure enough, you are seeing a 500 error too.

What do you do now?

I hope you don’t want to log into each and every machine and start tailing log files from every service you run. There is a much easier way to do it!

The ELK stack is an open source set of three runtime components, an Elasticsearch server/cluster, a log collector and parser known as Logstash, and a search and reporting tool known as Kibana. These three services combine to give you a realtime, searchable view of all the logs from all your services in one place. A hugely valuable tool at 3am when your website is down!

tl;dr

We are going to setup an ELK system using Tutum’s Stack feature. If you want to skip the nitty gritty, then head to the end of the article where you can find the complete stack YAML code, and setup your ELK stack in about a minute.

Before we begin

To follow this tutorial we are going to need a Tutum account and at least one node setup and running in that account. This tutuorial will work with more nodes just as easily as it does with one node. Your node should have at least 1GB of RAM.

For an introduction to setting up your first node, please read this article.

You may also want to familiarize yourself with Tutum Stack‘s. You can find more documentation here.

We are going to be creating this system inside what Tutum calls a Stack. This will allow us to group services that work together in a single place. At the end of the tuturial we will have a single YAML file that we can use to bring up an ELK stack on a Tutum cluster in just a few clicks.

Elasticsearch

First thing we need is a place to store and also to search our log files. For this task we will use Elasticsearch. It has the ability to scale to many nodes, but for the purpose of this tutuorial we will be using a single Elasticsearch node.

Heading over to the Docker Hub we can find out what Elasticsearch images are available and how to use them. A quick search reveals that there is an Official Repo for Elasticsearch.

According to the README the image is configured with a volume at /usr/share/elasticsearch/data. This is where Elasticsearch will store our log data. We want to make this data persist if the container were to stop or be terminated. We also want to make sure we can access this data in order to back it up or to move it from one node to another should we need to.

The current Docker best practice is to store data for a persistent node inside a “Data Volume Container“. Think of it as a named storage locker for your data. In the future we can create other containers that reference this data volume container in order to back it up.

Let’s put what we know so far into a Stack YAML file.

elasticsearch-data:
image: 'alpine'
command: '/bin/true'
volumes:
- '/data/elasticsearch:/usr/share/elasticsearch/data'

elasticsearch:
image: 'elasticsearch:1.5'
volumes_from:
- 'elasticsearch-data'

Our elasticsearch-data section defines a Service within our Stack. It is using an image named alpine. This is a stripped down version of Linux (based on Busybox) with only the basic commands available. It is a decent place to start for a Data Volume Container. I also wanted to mention it here, as it could be a great starting point for other Docker images that you might create.

We tell the elasticsearch-data service to start up with the command /bin/true, which effectively tells it to start and immedeatly stop. That’s ok! A Data Volume Container does not need to be running for another container to access its data.

We also tell the elasticsearch-data service to create a volume on the host machine at /data/elasticsearch and plug that into the container at /usr/share/elasticsearch/data. Any other container that includes the volumes from elasticsearch-data will be able to access the data at /usr/share/elasticsearch/data.

That’s it for Elasticsearch! We should now have a fully functioning Elasticsearch server ready to go. Feel free to launch a new stack with the above YAML file. You can optionally open a port to test your installation by changing the elasticsearch service definition above to read:

elasticsearch:
image: 'elasticsearch:1.5'
ports:
- '9200:9200'
volumes_from:
- 'elasticsearch-data'

Launch that and then go to the service page and click on the Endpoints link at the top to find it’s IP address. Then open that address in your browser and you should see something like this:

{
"status" : 200,
"name" : "Ant Man",
"cluster_name" : "elasticsearch",
"version" : {
"number" : "1.5.2",
"build_hash" : "62ff9868b4c8a0c45860bebb259e21980778ab1c",
"build_timestamp" : "2015-04-27T09:21:06Z",
"build_snapshot" : false,
"lucene_version" : "4.10.4"
},
"tagline" : "You Know, for Search"
}

Logstash

Now it’s time to start actually collecting logs from our Tutum services and nodes.

Going back to the Docker Hub and searching for “logstash” reveals that there is an official Logstash image. Hopefully you are seeing how great a resource the Docker Hub is. You can create complex systems extreemly fast by using this library of community and officially created images.

Now that we have an image to use, let’s checkout their README so we can learn how to use it. It shows us that we can configure Logstash from the command line. This is great because we won’t need to create our own image with our custom configuration inside.

After looking at the Logstash documentation I learned that Logstash doesn’t support reading Docker logs out of the box. But it does support reading from a syslog stream. Thankfully there is a great program out there called logspout from Glider Labs that can read our Docker log steam and send them to a syslog daemon.

Let’s add to our stack YAML file from above with our new Logstash service, configured to listen for syslog messages and to send our logs to Elasticsearch:

logstash:
image: 'logstash'
command: 'logstash -e ''input { syslog { port => 12345 type => "docker" } } output { elasticsearch_http { host => "elasticsearch" } }'''
expose:
- '12345/udp'
links:
- elasticsearch

In our logstash config above we set the command for the service to include our configuration. You will notice that we set an input for syslog and set it to listen on port 12345. We also set the Logstash type to be docker so that we can be clear where these logs are coming from. To round out the configuration we add an output section that will output all the logs to our Elasticsearch server over HTTP using the elasticsearch_http output plugin.

Pay close attention to the host setting above. We set it to elasticsearch. When logstash starts up it will use a DNS lookup to find the IP address for elasticsearch. How does it know where the elasticsearch host is? Glad you asked!

The links section tells Docker and Tutum that we want to make the hostname elasticsearch map to the elasticsearch service in our stack. This is a powerful tool that allows us to build contaners without knowing ahead of time what IP addresses will be used for each service.

Logspout

Now that we have Logstash setup to recieve log messages we need Logspout to send the Docker logs from our nodes.

This time we will use the Logspout image from Glider Labs.

This is a pretty simple service, the only thing it needs is a connection to our Docker daemon so that it can read the logs and a connection to the Logstash server so it can forward the logs.

Let’s add Logspout to our stack YAML:

logspout:
image: 'gliderlabs/logspout'
entrypoint: '/bin/sh'
command: '-c ''/bin/logspout syslog://$LOGSTASH_PORT_12345_UDP_ADDR:12345'''
deployment_strategy: every_node
volumes:
- '/var/run/docker.sock:/tmp/docker.sock'
links:
- logstash

There are several interesting things going on here. Let’s tackle the entrypoint and command lines first.

By default the logspout image defines an entrypoint that forces the container to execute the /bin/logspout program directly. Unfortunately, we can’t use the hostname method that we used before. Logspout is unable to resolve the hostname.

The workaround to this is to use an environment variable. Tutum injects environment variables into our container when we set up links in addition to the hostnames.

In our logspout configuration above we set the entrypoint to /bin/sh so that we can inject environment variables into our command. We then set the command to execute /bin/logspout with the correct options.

Another special part of the logspout service above is the deployment_strategy setting. This tells Tutum that we want to be sure that this service will start a container on all of our nodes. This ensures that we will collect the logs from all our nodes, even if we add new ones later on. Awesome!

Kibana

Finally we need a convenient way to view and search our logs. The last element to our ELK stack is Kibana.

Checking out the Docker Hub I didn’t find an official Kibana image, but I did find one that will suit our needs: marcbachmann/kibana4.

The only configuration for this Kibana image that we need is to set the ELASTICSEARCH environment variable to point to our Elasticsearch service.

Here is the stack YAML to setup our kibana service:

kibana:
image: marcbachmann/kibana4
environment:
- ELASTICSEARCH=http://elasticsearch:9200
ports:
- '5601:5601'
links:
- elasticsearch

We can assign any number of environment variables to our services using the environment setting. The rest of this stack YAML should look familiar. We link the service to our elasticsearch service, which enables the elasticsearch hostname to resolve to the Elasticsearch server. We also set a public port so that we can access Kibana from the internet.

Putting it all together

That should be it! We can put all the configuration into one YAML file and deploy the stack on our cluster.

elasticsearch-data:
image: 'alpine'
command: '/bin/true'
volumes:
- '/data/elasticsearch:/usr/share/elasticsearch/data'

elasticsearch:
image: 'elasticsearch:1.5'
ports:
- '9200:9200'
volumes_from:
- 'elasticsearch-data'

logstash:
image: 'logstash'
command: 'logstash -e ''input { syslog { port => 12345 type => "docker" } } output { elasticsearch_http { host => "elasticsearch" } }'''
expose:
- '12345/tcp'
- '12345/udp'
links:
- elasticsearch

logspout:
image: 'gliderlabs/logspout'
entrypoint: '/bin/sh'
command: '-c ''/bin/logspout syslog://$LOGSTASH_PORT_12345_UDP_ADDR:12345'''
deployment_strategy: every_node
volumes:
- '/var/run/docker.sock:/tmp/docker.sock'
links:
- logstash

kibana:
image: marcbachmann/kibana4
environment:
- ELASTICSEARCH=http://elasticsearch:9200
ports:
- '5601:5601'
links:
- elasticsearch

After all the services in the stack have started (it’s OK for the elasticsearch-data service to show as Stopped) we should be able to access the Kibana interface at this address: http://kibana.stack.user.svc.tutum.io:5601/. Be sure to replace stack with the stack name you used and user with your user ID in the domain name.

Kibana will require a small amount of configuration the first time you launch it.

It will ask you to “Configure an index pattern”.

Click the checkbox next to: Use event times to create index names.

Then change the Time-field name to equal @timestamp.

Click Create. And you should be good to go!

Tagged with: , , , , ,
Posted in Tutorial
18 comments on “Log Searching and Analysis with Tutum and an ELK
  1. Here is a cleaned up YAML of the final Stack:

  2. Krzysiek says:

    Reblogged this on Wolny strzelec czy szybki snajper? and commented:
    Pamiętacie ELK? W tutum można całą paczkę uruchomić sobie w chmurze. Póki co za darmo 🙂

  3. herzogmunk says:

    Absolutely AWESOME!

  4. herzogmunk says:

    PS @ianneub … on tutum `elasticsearch-data` has to be `elasticsearchdata`

    • Did you get an error trying to use the service name with the dash? That’s odd as the above YAML works for me from a straight copy and paste:

      • herzogmunk says:

        Yeah, sorry my fault. I tried to run it _locally_ and got the errors there
        `docker-compose -p elk -f docker-compose.yml up -d`
        `Invalid service name “elasticsearch-data” – only [a-zA-Z0-9] are allowed

  5. This is awesome! 2 questions though.
    1. The kibana endpoint seems to be open for everyone. Is there a way to add authentication?
    2. Is there a way to limit (set maxsize) the amount of data that is stored in the elastic search data container?

    • Ian Neubert says:

      The way that I solved the open endpoint problem was to add in an oauth container that sits in front of kibana and proxies traffic to it. Such as https://github.com/ianneub/docker-google-auth-proxy

      Here is an example:

      As far as limiting the size of a container or a volume. I don’t believe that is currently possible with Docker. I’m not 100% sure though, so I would recommend checking elsewhere to find a good answer 🙂

  6. Stuart says:

    Really excellent guide. One thing I found is that I had to change the logstash command to the following:

    command: ‘logstash -e ”input { syslog { port => 12345 type => “docker” } } output { elasticsearch { host => “elasticsearch” } }”’

    This seems to be a result of a deprecation to elasticsearch_http.

  7. lucj says:

    Very good article, thanks !
    What would you recommend if a more complexe logstash configuration is needed ? Passing it to the command argument could be hard. How would you copy it over to the Tutum’s host ?

    • Ian Neubert says:

      I would create my own logstash image. In that image I would add my custom configuration. The stack would remain essentially the same, except that I would change the logstash component to use my custom image.

      • lucj says:

        Thanks, that makes sense. I use docker-elk compose file but the init step of copying the configuration to the node is a little bit annoying. Thanks a lot.

  8. jparkercaa says:

    Very cool article. I’m looking to productionize something like this but I don’t see an obvious way to scale the elastic cluster (or any other disk-bound service) to multiple containers. I’m thinking of building a service per elastic host, each with it own volume (and volume service I suppose) but I would love a more elegant solution. What am I missing?

  9. None of the YAML files work for me. I try to deploy and the logstash containers dies after a few moments.

  10. Hi Awesome article thanks for sharing!

    I needed to change the logstash command to make it work

    https://gist.github.com/duncanhunter/1e1076ef94b82e7eb250.js

  11. max says:

    Thanks for the article, this is very awesome!

    I got everything working (+ google oauth reverse proxy) but only see logs from some of the “logspouted” nodes and not the others. Can’t see any errors in logstash or the instances of logspout… 😒 any ideas other way to debug something like that? Thanks again for the writeup!

  12. Los bancos colombianos sⲟn agiotistas, no le ԁɑn intereses al
    ahorrador sino máѕ bien gastos ⲣor sus cuentas
    y servicios, y terminan con los ahorros ɗе laѕ persinas
    menos pudientes.

Leave a Comment

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Categories
%d bloggers like this: