Log Forwarding with Tutum

Tutum is a great way to manage your Docker deployments. It has support for stacks of Docker-based infrastructure, zero-downtime deployments, and many other features that make it easy to move Docker from your laptop to production.

One of my favorite features about Tutum is the ability to view logs of a container through the Tutum web UI. When you’re having trouble debugging a deployment, the logs are the first place to check.

However, sometimes you need a more powerful log tool for your webapp. There’s been a request for better log draining support, so I thought I would show how I’ve handled this issue for my app.

What you need

  • A Tutum node that is ready to deploy services
  • A Docker service to play with that logs to STDOUT/STDERR (you can use my sample app below if you don’t have your own).
  • A server to forward syslogs. I use Papertrail, which is really easy to set up.

Docker logging basics

Currently, the Docker logs API has access to output from STDOUT and STDERR. As such, you may need to rethink how your app handles logging. If you want to log to syslog or another utility within your container, this tutorial won’t work for you. However, if you’re loving the logs displayed by the Tutum web UI, this will be perfect for you, as the Tutum UI is displaying logs from the Docker daemon.

If your app doesn’t log to STDOUT/STDERR but you’re still interested in trying this out, you’re in luck. I’ve made a sample Flask app with two endpoints that logs simple messages to STDOUT. You can use this to follow along.

The examples below will assume you’re using my sample Flask app, but it will apply similarly to anything you use.

Using logspout to foward Docker logs

To handle log forwarding, we’ll be using the logspout container from GilderLabs. The logspout container hooks into the Docker daemon and forwards it to a remote syslog.

First, you’ll want to deploy a logspout service on Tutum. At the Tutum Services UI, click on “Create Service”. Under “Public Images”, search the Docker Hub for “logspout”. Select gilderlabs/logspout as your service.

On the Service Configuration page, click the “Advanced Options” button to explore more options. In the Run Command option, enter the address to your remote syslog server. For example, if using Papertrail, you’ll put something like: syslog://logs2.papertrailapp.com:<port>.

Logspout Service Config

You can choose to publish port 8000 on the Service Configuration page to enable streaming of logs and other cool features. This is not required but is a cool additional feature.

After confirming Service Configuration and the Environment Variables page, you will be at the Volumes page. Add a new Volume with a container path of/tmp/docker.sock and a host path of /var/run/docker.sock. This allows the logspout container to listen for Docker logs from the Docker daemon.

Logspout Volumes

Once you’re done, hit Create and deploy. You’re now ready to deploy some apps for logging!

Logging events from containers

Now that your logspout service is deployed, you’re ready to test out some logs.

Back at the Tutum web UI, click ‘Create Service’ again. If you want to use the demo app, search Public Images on the Docker Hub for ‘tutum-log’. You should see ‘alexdebrie/tutum-log’ as an option. Select this image and move to image configuration.

When configuring this image, the only thing you need to do is publish port 8000 on the container to port 80 (or any other port you choose) on your node. This will allow you to access the app via your web browser.

Tutum-logs Service Config

Once your app is deployed, visit your node’s IP address. You should see ‘Hello, Tutum!’ in your browser. Next, go to <YourIPAddress>/time. You should see “Today is <Today’s date>” in your browser. I know you’re blown away by this app, but keep your focus — it’s not the point of this tutorial.

Viewing your logs

Now it’s time to check out your logs from your Docker-ized apps. If you’re using Papertrail, check out your logs page. You should see logs from when you visited your app. If you used the Flask demo app, you should see “You just logged a request from Tutum!” for each time you visited your ‘/’ page, and “Somebody visited at ” for each time you visited your “/time” page.

Papertrail

In addition to these logs for each request, you should see logs related to Docker containers, such as when your Flask app was created.

Additional tips

  • Remember that each container you want to log needs to log to STDOUTor STDERR. This can be hard for certain services. HAProxy, for example, doesn’t support logging to STDOUT. I’ve had success with Nginx, by adding access_log /dev/stdout; in your nginx.conf for logging config.
  • If you have a multi-node setup, you’ll need to deploy a logspout container on each node. To do this, you can select the ‘Every node’ deployment strategy. You can read more on deployment strategies here.
  • Consider filtering your logs to avoid Tutum-specific logs. Tutum sends out certain metrics and other messages related to Weave, a Docker networking tool. I used Papertrail’s log filtering to filter out messages that I don’t need. For example, I entered “weave|influxdb” in my Papertrail filter.
  • Logspout has some really cool additional features. You can inspect log streams using curl or create custom routes, which allows you to do container-specific logging to different syslog servers. If there’s community interest in how these advanced features work, I can write a follow-up post to walk through these.

Questions?

Any questions on logging with Tutum or logspout? Feel free to leave comments, and I’ll try to answer as best as I can!

Tagged with: , ,
Posted in Tutorial
9 comments on “Log Forwarding with Tutum
  1. Great article! I installed the logspout container a few days ago, and it’s working well.

    However, a majority of the logs seem to be weave notifications, and logspout’s own “sending node metrics to influxdb” notifications.

    Is there any way to isolate the logs to remove those messages from the logs being sent?

    • alexdebrie says:

      Thanks, E.T.! I’ve been happy with the results so far as well.

      As far as the unneeded logs, I’ve used log filtering to remove the weave notifications and influxdb metrics. I set a simple filter of “weave|influxdb” and was able to remove almost all of the unnecessary logs. If you use Papertrail, you can read about log filtering at the following link: http://help.papertrailapp.com/kb/how-it-works/log-filtering/.

      If you don’t use Papertrail and don’t have a similar filter feature, you could always try the routes feature of logspout. This allows you to set granular routes for logging, such that all logs from a particular container or group of containers would be sent to one logging location, while logs from other containers could be sent elsewhere. You can read more about routes here: https://github.com/gliderlabs/logspout#create-custom-routes-via-http.

  2. kkouddous says:

    How timely! I just implemented logspout for my cluster. One great addition would be using the container names in the logs vs. the cryptic ids (similar to how the logs show in the tutum ui). I was going to dive into logspout and see if I could easily implement this. Would this be possible?

    • Borja Burgos says:

      Borja from Tutum here, we’re actually changing this in our backend so that containers are provisioned with user-friendly names. You should start seeing friendlier names without having to change anything 🙂 Hope that helps!

      • kkouddous says:

        Great! But I just redeployed all services and I’m still seeing IDs in logs e.g.:

        Mar 28 08:41:24 prod-logspout-3 3d4c6311-539e-422c-a295-f3b49956bd10: Sat Mar 28 12:41:24.943 [initandlisten]

  3. Borja Burgos says:

    kkouddous, it’s coming, it hasn’t been implemented yet.

  4. Can youu tell us more about this? I’d love to find out
    more details.

  5. I ran across https://github.com/tutumcloud/syslogger, which is an “Image based on gliderlabs/logspout optimized for syslog and Tutum”

    It works as a drop-in replacement for gliderlabs/logspout in this guide, and even better, it has a “Deploy to Tutum” button which you can press, then put something like “syslog://logs2.papertrailapp.com:” in the run command.

Leave a Comment

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Categories
%d bloggers like this: