How to build a Docker Image that depends on a private gem using CircleCI

Have you ever wanted to write some sharable code that can be used across projects? When writing ruby, gems are great for that… but what if you wanted to keep the code private? Is the gem code a prototype or have you not set up your gem deployment strategy?

At my current company Fitbod, we wanted to do all of the above. We have 2 microservices that depend on some shared code. One option, which I’ve used in the past, is to use git submodules. Another option is to create a gem. We don’t currently have a private gem server set up. We may eventually do that as we start to write more private gems. For the time being, we plan on using a Github private repo to host the shared code.

Here are the steps that we’ve taken to make all of this possible:

Create the gem

Bundler makes it really easy to create new gems. Follow these instructions to create a new gem. Once it has been created, push it to the private repository for your Github Org.

Add the gem into your Gemfile

You may be familiar with bundlers ability to serve up gems from git repos. Take note of the security concerns, especially if you’re dealing with a private repo. Long story short, add this:

gem 'my_gem', :git => 'https://github.com/my_org/my_gem'

Create a Github (Machine) User

Take note of the terms of service for machine users:

A machine account is an Account set up by an individual human who accepts the Terms on behalf of the Account, provides a valid email address, and is responsible for its actions. A machine account is used exclusively for performing automated tasks. Multiple users may direct the actions of a machine account, but the owner of the Account is ultimately responsible for the machine’s actions. You may maintain no more than one free machine account in addition to your free User Account.

This is exactly what we need if we want to have an account that has access to certain repositories without generating and using our own auth tokens. After creating the user, make sure to grant access to the organization and the necessary repositories.

Generate a GITHUB_TOKEN

This will be the token that we will use to access the private repos. To do this, log into the machine user account and follow these instructions.

Write up the CircleCI Config

This can be broken up into two sections:

How to use the gem in tests

If you’re focused on TDD, chances are that the gem that you depend on for the docker image is also needed by your tests. In order to grant access to the private repo, you need to pass the GITHUB_TOKEN along to bundler so that it can authenticate successfully. I’ve seen a couple posts that promote this approach:

This doesn’t really work for us however because it ends up saving the access token in the Gemfile.lock. The alternative is to use bundle config to set the credentials like this:

So, to install the gems in tests we have this config:

and the bundle.sh script looks like this:

How to build the gem into a docker image

Building the Docker image is a little more difficult.

• We need to be able to pass the GITHUB_TOKEN into the docker context so that it can be used when building the image.
• We don’t want the token to be visible with docker history

In order to do this, we add a build arg to our docker file for the GITHUB_TOKEN and use multi-stage builds. Here’s what it looks like:

We have a build script that can be used by all of our microservices which looks like this:

DOCKER_BUILD_ARGS allows us to set any extra arguments that we need when building the image. In this case, we want to set the GITHUB_TOKEN. To do this, we need to dynamically set the env variable in CircleCI like this: