The heart of Continuous Integration is running tests.
Whenever a test fails, you want to know why ASAP so you can correct it.
Whenever a CI build fails, you want to see that failing test and how it failed.

CircleCI’s Test Summary feature puts this info front-and-center so you can
respond directly to the test failure without anything getting in your way.
The trick is to feed CircleCI your test info the way it expects.

Why set up Test Summary when you already have a build log?

The build log might be fine to start.
You expand the failing step, scroll to the end of the page, and then scroll up till you hit the test failure.

This is not too bad. At first.

But with a big enough project, the build and test logs grow too long to view in-place on the web page.
Then you find yourself downloading the log file first.

Sometimes the failing test isn’t really that near the end of the file.
Then you’re fumbling around trying to find it.

Across a lot of developers on a long project,
this time and friction adds up.

Don’t CircleCI’s docs cover this already?

If you’re building an iOS app, and you copy-paste the
Example Configuration for Using Fastlane on CircleCI,
you should luck into something that works.

But you’ll want to better understand what the Test Summary feature
is looking for if:

• Your Test Summary omits info you want in there, like linter output.
• Test Summary isn’t working for you, and you want to fix it.
• You’re not building an iOS app using Fastlane, and one of the other example configs doesn’t meet your needs.

What does CircleCI need for a Test Summary?

CircleCI’s Collecting Test Metadata doc
calls out one big thing:

• Report tests using JUnit’s XML format.

The store_test_results step reference
calls out another:

• Your test report should be in a subdirectory of another “all the tests” directory.

This subdirectory name is used to identify the test suite.

There’s one more requirement that I haven’t seen documented anywhere, though:

• The JUnit XML test report file must literally have the xml extension.

The rest of the filename doesn’t seem matter for the test summary,
but if you have the wrong path extension,
you won’t see any test summary.

What does that look like on disk?

You’ll wind up with a directory layout like:

/Users/distiller/project/
└── fastlane
    └── test_output
        └── xctest
            └── junit.xml

3 directories, 1 file

This ticks all the boxes:

• XML file: junit.xml
• “Test Suite” directory: xctest
• “All Test Suites” directory: test_output

(Fastlane only produces a single test report,
so the nesting of report-in-folder-in-folder admittedly looks a little silly.)

How do you get Fastlane Scan to write JUnit XML to that path?

Scan provides a lot of config knobs.
You can view a table of the full list and their default values by running
fastlane action scan.

We need to arrange three things:

• The report format: JUnit
• The report filename: junit.xml
• The directory that report file should be written to: fastlane/test_output/xctest

Conveniently enough, Scan has three config settings, one for each of those
needs.

Scan also happens to have three different ways
to set those three options:

• In your Fastfile:
  • Using keyword arguments to scan()
• In your shell:
  • Using option flags
• Anywhere (but probably in your shell):
  • Using environment variables

Keyword Arguments to scan()

In your Fastfile, you can set them using keyword arguments to the
scan method call:

scan(
  # … other arguments …
  output_types: 'junit',
  output_files: 'junit.xml',
  output_directory: './fastlane/test_output/xctest')

Option Flags to fastlane scan

If you’re invoking fastlane directly,
you can set them using CLI options:

fastlane scan 
  --output_directory="./fastlane/test_output/xctest" 
  --output_types="junit" 
  --output_files="junit.xml"

Environment Variables

Because Ruby is a true descendant of Perl,
TMTOWTDI,
so you could also configure Scan using environment variables:

env 
  SCAN_OUTPUT_DIRECTORY=./fastlane/test_output/xctest 
  SCAN_OUTPUT_TYPES=junit 
  SCAN_OUTPUT_FILES=junit.xml 
  fastlane scan

(You could also set those environment variables in the environment stanza in
your CircleCI config. Six one way, half-dozen the other.)

How do you get CircleCI to process the JUnit XML?

Now you have Fastlane Scan writing its test report using the JUnit format into
a *.xml file under a suggestively-named subdirectory.

To get CircleCI to actually process this carefully arranged data,
you’ll need tell the store_test_results step to snarf everything at and under
fastlane/test_output.

That’s right: not just the xctest subdirectory that holds the test report
XML, but the directory.

Add this step to the pipeline that runs scan:

- store_test_results:
    path: "./fastlane/test_output"

What about the rest of Scan’s output?

At some point, you’ll probably want to be able to look at the test report
yourself, as well as the overall build logs.

You can send both of those on up to CircleCI as build artifacts using a couple
store_artifacts steps:

- store_artifacts:
    path: "./fastlane/test_output"
    destination: scan-test-output
- store_artifacts:
    path: ~/Library/Logs/scan
    destination: scan-logs

What about more than just Scan?

You’re not limited to just one artifact or just one test output.
In fact, handling multiple kinds of test output is precisely why there’s the
folder-in-folder nesting.

Say you wanted to have CircleCI call out SwiftLint nits.
You could drop this snippet into your jobs list:

lint:
  docker:
    - image: dantoml/swiftlint:latest

  steps:
    - checkout

    - run:
        name: Run SwiftLint
        command: |
          mkdir -p ./test_output/swiftlint
          swiftlint lint --strict --reporter junit | tee ./test_output/swiftlint/junit.xml

    - store_test_results:
        path: "./test_output"
    - store_artifacts:
        path: "./test_output"

The key links in the chain here are:

• Create an “ALL the tests” directory: ./test_output/
• Create a “test suite” directory: ./test_output/swiftlint/
• Write JUnit output into a .xml file: ./test_output/swiftlint/junit.xml
• Aim store_test_results at that “ALL the tests” directory: path: "./test_output/"

Any output you can massage into meeting those requirements,
you can cadge CircleCI into calling out in your Test Summary.

Conclusion

There you have it:

• Quickly debug CI build failures
• By putting relevant test details in prime real estate
• By feeding a folder two steps up from a carefully located, precisely named test report file
• To CircleCI’s store_test_results build step.