In a previous blog post, I discussed integrating Android Studio, Gradle and Robolectric to perform unit testing as part of Android development. Some inquisitive commenters wanted to know if this setup supported testing multiple product flavors. The answer is the same as with most Robolectric-related topics: “Yes, but…” Since Roboletric 3.0 was released this week, now is a good time to explore what’s needed to set everything up.

If you’re unfamiliar with product flavors, Javier Manzano has written a great explanation of product flavors and how to use them. He cites the documentation’s definition:

A product flavor defines a customized version of the application build by the project. A single project can have different flavors which change the generated application.

If you are familiar with product flavors, you’re also probably familiar with build types. If not, I point again to the documentation:

A build type allows configuration of how an application is packaged for debugging or release purpose. This concept is not meant to be used to create different versions of the same application. This is orthogonal to Product Flavor.

Build types define how our application is packaged, meaning that they are independent of product flavor. The combination of build types and product flavors create build variants. These build variants represent the set of different ways our app is built and how it behaves. With all these different builds, we want to run tests on each variation, and possibly even different tests for different variations.

Starting with an Example

Let’s introduce a quick example to illustrate this. We’ll start with our original example but add “free” and “paid” product flavors with unique application ids. Furthermore, let’s add debug and release build types. This gives us four build variants:

• free debug
• free release
• paid debug
• paid release

To accomplish this, we update our build.gradle file:

apply plugin: 'com.android.application'
android {
    compileSdkVersion 22
    buildToolsVersion "22.0.1"

    defaultConfig {
        applicationId "com.example.joshskeen.myapplication"
        minSdkVersion 16
        targetSdkVersion 22
        versionCode 1
        versionName "1.0"
    }

    buildTypes {
        release {
            minifyEnabled false
            proguardFiles getDefaultProguardFile('proguard-android.txt'), 'proguard-rules.pro'
        }
        debug {
            applicationIdSuffix ".debug"
        }
    }

    productFlavors {
        free {
            applicationId "com.example.joshskeen.myapplication.free"
        }
        paid {
            applicationId "com.example.joshskeen.myapplication.paid"
        }
    }
}

I’ve forked our original example and added these changes. Our project structure now looks like this:

Let’s say that the release flavors are the ones we put on the Play Store, and we give the debug flavors to our QA team so they can see the logs. Let’s also say that our paid version has some fancy feature, and our free version has a button to sign up for the paid version. As you can imagine, we want different tests for each of the variants, so let’s write some Robolectric unit tests.

In our example, we want to ensure that logging is happening on our debug builds. It doesn’t make sense to run this test on our release types. In fact, we don’t want to run this test on release builds because it would (hopefully) fail.

@Test
public void sendingWebRequestCreatesLogStatement() {
    if (BuildConfig.DEBUG) {
        // assert that logs are printed
    }
}

We may also want to test our flavors separately. Inside our tests, we check the product flavor by checking the application id:

@Test
public void clickingUpgradeButtonLaunchesUpgradeActivity() {
    String applicationId = BuildConfig.APPLICATION_ID;
    if (applicationId.equals("com.example.myapplication.free")) {
        // assert that button click does something
    }
}

@Test
public void fancyFeatureDoesSomething() {
    String applicationId = BuildConfig.APPLICATION_ID;
    if (applicationId.equals("com.example.myapplication.paid")) {
        // assert that feature does something
    }
}

Making it Happen

Out of the box, Android Studio wants to help you. When we run our tests, Android Studio will build each variant of our application and run our tests against each variant. This is one of the major benefits of running Gradle tests instead of JUnit tests.

In our case, our test suite is run four times, for our four build variants.

In a perfect world, this would actually happen. In reality, you’ll get a nice error when you try. Using Robolectric’s RobolectricGradleTestRunner causes a problem for these product flavors:

no such label com.example.joshskeen.myapplication.free.debug:string/app_name

android.content.res.Resources$NotFoundException: no such label com.example.joshskeen.myapplication.free.debug:string/app_name
    at org.robolectric.util.ActivityController.getActivityTitle(ActivityController.java:104)

I’ll save you the trouble of diving through the code yourself. The issue is that Robolectric can’t find where Android Studio stashed your generated R.java file. It lives here:

But Robolectric is looking for it in the com.example.joshskeen.myapplication.free.debug directory for the free/debug build variant, and similarly-named directories for other build variants.

Why all the Confusion?

In order to understand the problem, we need to understand the difference between package name and application ID. The Build Tools Team has written a good description. Basically, you can think of the application ID as the outward-facing name of your application for unique identification, and the package name as the internal name of your application for organization of your .java files.

When I initially drafted this post, the then-current release candidate of Robolectric 3.0 confused the two. But they are not alone. Javier Manzano’s post mixes the two. Even the Build Tools documentation I linked to above makes the same mistake:

The manifest of a test application is always the same. However due to flavors being able to customize the package name of an application, it is important that the test manifest matches this package. To do this, test manifests are always generated.

As we’ve seen, the flavors don’t customize the package name; they customize the application id. Understanding the difference is the key to fixing this problem.

The Fix

Luckily for us, Robolectric has caught the problem and issued a fix as part of version 3.0. To use this new feature, you’ll need to annotate each of your test classes to include the packageName configuration value:

@RunWith(RobolectricGradleTestRunner.class)
@Config(constants = BuildConfig.class, sdk = 21, packageName = "com.example.joshskeen.myapplication")
public class MyActivityTest {
    ...
}

This allows RobolectricGradleTestRunner to distinguish between an application ID, which is different from the package name.

If you are using the previous stable version (2.4) of Robolectric and the base RobolectricTestRunner, you can continue to use this TestRunner. Support for specifying application id was added in this commit.

Better Test Structure

Now that we have our tests running successfully for different build variations, I want to go back and address how we’ve structured our tests. In the above test sample code, we had a single set of tests that was run for all build variations but executed different behavior inside the test with ifstatements. This isn’t a very elegant solution: We’ll end up with a bunch of “empty” tests this way.

In our example, the clickingUpgradeButtonLaunchesUpgradeActivity test would automatically pass for the paid version. What is the value in this? Nothing. Furthermore, this will slow down our test suite by running setup() unnecessarily. Luckily, we can separate our test code the same way we separate our product flavor code. By creating test folders that match our product flavor folders, we can run tests specific to product flavors. We can create testPaid and testFree directories, and put only the tests we want run on those flavors inside them. We can remove the if statement from our tests and they become pretty standard:

app/src/testFree/java/com.example.joshskeen.application/MyActivityTest.java

@Test
public void clickingUpgradeButtonLaunchesUpgradeActivity() {
    // assert that button click does something
}

app/src/testPaid/java/com.example.joshskeen.application/MyActivityTest.java

@Test
public void fancyFeatureDoesSomething() {
    // assert that feature does something
}

To test our debug builds, we can also add a testDebug directory, and remove the if statement from these tests as well

app/src/testDebug/java/com.example.joshskeen.application/MyActivityDebugTest.java

@Test
public void sendingWebRequestCreatesLogStatement() {
    // assert that logs are printed
}

It is important to note that when we run free debug or paid debug tests, two test files will be accessed: (free test OR paid test) and debug test. Therefore, we have to rename our debug test file. Here I’ve chosen to call it MyActivityDebugTest.java, but the name choice is up to you. Our entire directory structure now looks like this:

I should note that if you run the test suites for all of your build variants and one of those suites has a failing test, Android Studio will stop without proceeding to the other variants. This makes it a bit hard to figure out the source of the failing test. Is the test failing on just this one variant? Or on others as well? The only way we can know is to manually run the remaining suites, one at a time.

Sample the Flavors

We can now leverage the benefits of building multiple product flavors and build types inside Android Studio without sacrificing the benefits of test-driven Android development, enabling us to continue to deliver high-quality products on this platform.