Quite a few continuous integration tools exist, but as soon as you narrow it down to options that properly support Android, it becomes much easier to pick one. Travis and TeamCity are viable options if you care primarily about the codebase compiling and passing tests, but do not need options for storing artifacts, signing them and publishing them. At Big Nerd Ranch, we have adopted Jenkins as our Android continuous delivery environment because it has enough extensibility for plugins to fully implement the entire deployment pipeline properly.

Jenkins can be set up to run in many different configurations ranging from a single local server in a closet running the whole pipeline to a cloud-deployed master that spins up virtual machine slaves to run individual builds. We chose to have a Jenkins master node constantly running on a cloud instance, which has the ability to spin up single-use slave instances for individual builds. This gives us a consistent and completely fresh build environment each run. The slave is created from an Amazon Machine Instance (AMI) on EC2 that can be updated when new Android SDKs and build tools are released. Overall, the cost works out to pennies per build, on top of the base cost of the master instance, which with EC2 isn’t much. The slaves are provisioned as m3.mediums from our AMI, using a bit of userdata configuration to mount the Android SDK and make sure 32-bit compatibility is available.

The details and specifics of how to get EC2 up and running are outside the scope of this post—we’re going to skip ahead to the Jenkins-related bits. Once all the credentials and keys are set up for the Jenkins JClouds plugin, you now have a functioning build server. At this point, the build server can compile and run your test suite and provide a link to a debug APK stored on the master node.

Time to make things interesting.

Once we turn on the Github plugins, the server will be able to mark a pull request as passed or broken and also email the team if master fails to build. We use two Jenkins jobs for this purpose.

One job is configured to merge a pull request (PR) into master and make sure things compile on a PR-specific basis. This job catches issues that would normally be noticed only after the PR had already been merged, and also provides quick feedback on what broke.

The other job is responsible for building and running the full test suite any time master is updated, typically after a PR has been merged. This job double-checks that master is staying in a happy state, but also takes significantly longer than just compiling. It runs the full Lint suite and generates a report, failing the build if the error or warning count increases from previous builds.

So to sum up, one Jenkins job ends up sanity-checking our PRs, and the other checks our master branch.

Just setting up Jenkins to continuously build like we’ve done already yields a useful tool—code quality is kept high, and we can more easily decide when a build is in a shippable state. At this point, creating a release is as simple as checking out a Jenkins-passed commit, bumping the appropriate version codes and numbers, running a release build on a local machine, signing the APK with keys that are inappropriately stored on all the developers’ machines, renaming the file appropriately, assembling release notes, tagging the commit, and then finally uploading the APK to the Google Play Store console.

Simple, right? It’s unfortunate that this amount of work is considered to be an accepted process, but since a Jenkins server has already been configured to build the codebase, we’re just going to automate the whole process of releasing a new build.

Automating Releasing a New Build

The first step is to create a new Jenkins job. This job is distinct from the others in that it will build only when manually triggered (as opposed to being continually integrated by PRs), since this will be our actual release. Uploading APKs to the Play Store requires increasing a version number, but we don’t really want to have to change it each time. Since we have a dedicated job for this release build, we can just use the build number for the job as the version number and (part of) the version name for our application without having to do version-bump commits before each release.

After bumping the version code and building, signing has to be dealt with. The naive approach is to use Gradle’s configuration to set up signing, but it is impossible to do this in a secure manner. Gradle can load the key and passphrase either straight from disk, or it can pull the data in from an environment variable.

Neither of these options is particularly secure. From disk, the key and passphrase would have to be in the slave image, requiring a separate image for every app and allowing anything during the build to compromise the key. Having the passphrase pulled from the environment is only slightly better. However, anyone with the ability to commit and execute a build could compromise the key by accessing their environment during the build. Using a dedicated Jenkins plugin for signing allows us to solve these issues.

Unfortunately, there is not a plugin for Android signing in the official plugin repository. However, we have written a very basic one and published it for you to compile and use. It will soon be available in the Jenkins update manager, but can also be downloaded and compiled manually. With this plugin, all signing happens only on the master node. In addition, the key and passphrase are stored in the Jenkins credential store, which limits access to be write-only from the web interface. This limits exposure to only the sysadmin of the Jenkins master. Signing happens to the artifact post-build; the key is not exposed during the build. Our plugin also has the benefit of creating both an unsigned artifact and a signed and aligned artifact.

The final step of our release job for continuous delivery is to actually upload the generated APK to the Google Play Store. Fortunately, there is a third-party plugin available. Once the plugin is installed, it’s just a matter of granting a Google API Console credential the correct permissions from the Play Store Developer Console. The documentation for the Google Play Android Publisher Jenkins Plugin (what a mouthful!) has good instructions on how to do this.

We upload to the alpha and beta tracks for testing, then promote a build later.

At this point, we have built, signed, and uploaded our APK, but there’s still some post-upload actions that are possible. One possible action can be pushing a tag to Github with the version code, which makes creating a release marker in GitHub one quick click away via the GitHub website. A benefit of having the tagging automated is that bug triage is much easier when you get crash reports from older versions, since those versions are still readily accessible and cataloged.

Long-term Payoffs

While this may seem daunting at first, once all of the work for setting up a Jenkins environment has been done for one project, it’s quick and easy to clone an existing job and just change the project-specific configurations. With a little short-term investment, the payoff in the long run is well worth it.

Thanks to Kurt Nelson for his contributions to this post.