CI/CD the Docker Way

CI-CD@2x

Tutum has recently announced build & testing capabilities and we are very happy not only about what we think is a huge step forward in providing the best CI/CD tools for developers, but also because of the amazing planning and team collaboration that was required from us to deliver such a big feature.

Tutum CI/CD design goals were to provide a flexible, agnostic, automatable and locally replicable solution for build & tests, which mirror the goals for the Docker project. This is why we based our solution for CI/CD on the open source image tutum/builder.

In a nutshell, tutum/builder performs the following steps:

  • Configure docker credentials using $DOCKERCFG.
  • Clone the repository using $GIT_REPO and checkouts the commit in $GIT_TAG.
  • Build the docker image specified by $DOCKERFILE_PATH.
  • Push the image to the image tag specified in $IMAGE_NAME.

How do we get the right values for these variables in order to automate builds in Tutum? The solution is our new Repositories UI that lets you register Docker repositories (from Tutum Private Registry and/or external registries like Docker Hub Registry) with valid credentials (needed to configure $DOCKERCFG) and associate them to Github repositories ($GIT_REPO).

By doing so, Tutum creates a webhook in your Github repository which hits the Tutum API for every commit, providing the value for $GIT_TAG. DOCKERFILE_PATH and IMAGE_NAME are configured by the user for every image tag. More details about the new Repositories UI can be found here.

Now build & testing becomes an orchestration problem for what Tutum has already built a good solution. Tutum will deploy the tutum/builder image using the “emptiest node” deployment strategy for Github webhook calls that match your build settings. If there are nodes tagged with builder, only those nodes will be used for build executions. When the build container stops, Tutum is notified via our events container, which notifies the Tutum API for every docker event (more about this topic here).

Build containers are automatically terminated after executing, but not before collecting the build logs in the build associated Tutum action so you can troubleshoot possible issues. If the image is pushed successfully, you can complete your CI/CD workflow using our autoredeploy flag or our redeploy triggers.

tutum/builder also provides the ability to run tests before pushing your image to your docker repository. As you might already know, Docker containers incur a very small footprint, which makes them ideal to deploy test & dev environments on a single machine. Also, docker-compose allows you to easily define all your dependencies in a docker-compose.yml file, relate them via links, and configure them via environment variables.

We take advantage of these amazing tools by letting you define a docker-compose.test.yml file that tutum/builder will spin up. tutum/builder assumes you have defined a sut (system under test) service on it, which is responsible for running your tests. Consequently, we can validate if your tests are passing by executing:

docker-compose -f docker-compose.test.yml -p app up sut
RET=$(docker wait app_sut_1)
if [ "$RET" != "0" ]; then
        echo " Tests FAILED: $RET"
        exit 1
else
        echo " Tests PASSED"
fi

This also happens to be executable from your local folder to unify the way you execute your tests. Tutum notifies the state of your build & tests using the Github status API, but you can improve the user experience by integrating Tutum native support for notifications with your favorite tools. More details about how to configure testing in Tutum can be found here.

As a final note, I would like to remark on the advantages of using Tutum for CI/CD workflows against existing tools. Tutum makes it incredibly simple to configure automated build & tests from Github push events but keeps everything running in your own infrastructure. Therefore, you have full control of the resources you want to dedicate for CI/CD to improve your performance or reduce your costs, having a predictable build execution environment. It is also trivial to recover from an infrastructure problem by creating a new builder node in a different provider. Last but not least, using your own node allows docker to maximize its caching mechanism when building and pulling docker images (we are working on how to achieve this yet, but it will be available soon).

Tutum is still actively improving our CI/CD user experience. These are the set of features we are working on in the short term:

  • Ability to define environment variables in your Tutum build settings to be injected in the build container.
  • Limit the maximum number of builds that a node can execute in parallel in order to prevent over-saturation of your infrastructure.
  • Autotag: schedule builds of the docker repository tag tagname for every new github tag tagname.
  • GitLab and Bitbucket integration.

We would love to hear your feedback about our approach to CI/CD; how can we make it better for you in the long-term and help integrate our solution into your CI/CD workflow?

Stay tuned for what is coming next!

Tagged with: , , ,
Posted in Features, Tutorial
3 comments on “CI/CD the Docker Way
  1. brycereynolds says:

    Another big advantage is being able to configure multi service test environments. We are planning to use this to help facilitate cross service acceptance testing as well as integration tests. I’d be curious to hear how others are doing this our maybe tutums response/thoughts on using the testing harness this way.

Leave a Comment

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Categories
%d bloggers like this: