Every complex backend server eventually needs three things: emails, workers and searching. At the Ranch, we like to whip out Solr for handling complex searches. Unfortunately, Solr can make testing frustrating: it needs to be running in the background any time we run integration specs, but starting up Solr is slow. Is there a way to lazily load Solr only for the tests that need it?

Setting up Sunspot

Adding Solr searching to a Rails app is trivial thanks to the Sunspot gem:

# Gemfile
gem 'sunspot_rails'
gem 'sunspot_solr'

$ bundle install
$ bundle exec rails generate sunspot_rails:install
$ bundle exec rake sunspot:solr:start

With Sunspot set up, we can index fields on ActiveRecord models with the searchable helper:

# app/models/user.rb
class User < ActiveRecord::Base
  searchable do
    text :name
    text :location
    time :created_at
  end
end

That was easy! In keeping with good object-oriented design, we should create a new class to handle querying:

# app/models/search.rb
class Search
  attr_reader :query

  def initialize(query)
    @query = query
  end

  def users
    Sunspot.search(User) do
      fulltext query
      order_by :created_at
    end.results
  end
end

Now we can search for users by name and location with a simple:

Search.new('bob').users
=> [#<User id: 2, name: "Bob", location: "Boston">]

No sunshine for you

At the Ranch, we are firm believers in writing quality tests for our code. Unfortunately, adding test coverage to Sunspot searches is a bit tricky. The Sunspot docs note (correctly) that tests involving Solr are integration specs, but consequently discourage tests at all beyond a few Cucumber specs! “If you want to do it anyway,” the docs recommend running Solr inline as a child process.

This route has caused us much grief: managing child processes in Ruby is ugly, and the test code needs to know how to start Solr with the correct environment and schema. Can we do better? Is there a setup that will:

• Allow us to test the Search class unit-style, not at the request level.
• Not slow down any other tests.
• Work well with guard.
• Decouple our test suite from Solr.
• Allow us to run the rspec command most of the time without starting up Solr.
• Allow us to run a Solr daemon in the background for continuous TDD of search based tests.
• Provide an easy way to run the entire test suite in one command.
• Run in CI services such as TravisCI.

That’s a lot of requirements! Can we hit them all?

Adding tests

Let’s start off with tests first. We’ll write a few unit-style specs for the Search class:

# spec/models/search_spec.rb
require 'rails_helper'

describe Search do
  subject(:search) { Search.new(query) }

  let(:alice) { User.create(name: 'Alice', location: 'Anchorage') }
  let(:bob) { User.create(name: 'Bob', location: 'Boston') }
  let(:charlie) { User.create(name: 'Charlie', location: 'Chicago') }
  let(:users) { [alice, bob, charlie] }

  # Force user creation
  let!(:data) { users }

  describe 'searching for users' do
    subject(:results) { search.users }
    context 'when searching by location' do
      let(:query) { 'Chicago' }
      it { is_expected.to match_array [charlie] }
    end
  end
end

Looks great! Only one problem: running rspec spec/models/search_spec.rb won’t load Solr first. That’s no biggy, we’ll manually start it up with bundle exec rake sunspot:solr:start RAILS_ENV=test

Now let’s run the Search class specs:

$ rspec spec/models/search_spec.rb

Finished in 0.37242 seconds (files took 1.29 seconds to load)
1 example, 0 failures

It passes! But it seems a bit unstable when we run the spec again. Sometimes we get:

Failures:

  1) Search searching for users when searching by location should contain exactly #<User id: 3, name: "Charlie", location: "Chicago", created_at: "2016-07-25 15:59:40", updated_at: "2016-07-25 15:59:40">
     Failure/Error: it { is_expected.to match_array [charlie] }

       expected collection contained:  [#<User id: 3, name: "Charlie", location: "Chicago", created_at: "2016-07-25 15:59:40", updated_at: "2016-07-25 15:59:40">]
       actual collection contained:    []
       the missing elements were:      [#<User id: 3, name: "Charlie", location: "Chicago", created_at: "2016-07-25 15:59:40", updated_at: "2016-07-25 15:59:40">]
     # ./spec/models/search_spec.rb:18:in `block (4 levels) in <top (required)>'

Finished in 0.07012 seconds (files took 1.13 seconds to load)
1 example, 1 failure

Failed examples:

rspec ./spec/models/search_spec.rb:18 # Search searching for users when searching by location should contain exactly #<User id: 3, name: "Charlie", location: "Chicago", created_at: "2016-07-25 15:59:40", updated_at: "2016-07-25 15:59:40">

Hmm, sometimes this works, and sometimes it doesn’t. Solr needs time to index new records, so sometimes the new users aren’t indexed before a Solr search is executed. We need to wait until all pending records have been indexed, so let’s flush the index in a before block.

   # Force user creation
   let!(:data) { users }

+  before do
+    Sunspot.commit
+  end

   describe 'searching for users' do
     subject(:results) { search.users }
     context 'when searching by location' do

And with that, we have Sunspot tests passing 100% of the time! But we’ve multiplied our testing woes. First, the rest of our test suite is drastically slower since every database update triggers a Solr request. But search specs make up a tiny fraction of our test coverage: 97% of our code doesn’t need Sunspot!

We also have to launch Solr every time we want to run any ActiveRecord specs. This makes it irritating to run tests continuously with guard and may drive new team members crazy.

Lazy-loading Sunspot

Why can’t we just load Sunspot for the few specs that need it but disable it everywhere else? Let’s do just that! We’ll extend Sunspot with a stub/unstub method to enable/disable Sunspot’s ActiveRecord model hooks.

# spec/support/sunspot.rb
require 'open-uri'

module Sunspot
  def self.stub_session!
    ::Sunspot.session = ::Sunspot::Rails::StubSessionProxy.new(::Sunspot.session)
  end

  def self.unstub_session!
    ::Sunspot.session = ::Sunspot.session.original_session
    wait_for_solr
    ::Sunspot.remove_all!
  end

  def self.wait_for_solr
    return if @started

    print 'Waiting for Solr (run test suite with `bin/test`)'

    until solr_listening?
      print '.'
      sleep 0.1
    end

    @started = true
  end

  def self.solr_listening?
    open(::Sunspot.config.solr.url).read
    true
  rescue OpenURI::HTTPError
    true
  rescue Errno::ECONNREFUSED
    false
  end
end

Then, we’ll load up our extension in the rails_helper.rb. By default, we’ll stub out (disable) Sunspot in specs unless their metadata includes search.

# spec/rails_helper.rb
require 'sunspot/rails/spec_helper'
require_relative 'support/sunspot'

RSpec.configure do |config|
  config.before(:each) do |example|
    ::Sunspot.unstub_session! if example.metadata[:search]
  end

  config.after(:each) do |example|
    ::Sunspot.stub_session! if example.metadata[:search]
  end
end

::Sunspot.stub_session!

If we run rspec now, our specs run blazingly fast again. However, the Search class specs fail since Sunspot has been stubbed out to return an empty array when we ask for results. Not to worry: we can annotate our search specs with the search metadata attribute!

 require 'rails_helper'

-describe Search do
+describe Search, search: true do
   subject(:search) { Search.new(query) }

Boom! Now our test suite runs as fast as possible without Solr and only connects when we come across the Search specs. So if we don’t want to start up Solr to test 97% of our other specs, no problem!

If we do run the whole test suite, the Search specs will kindly wait until Solr has started up, so we can start running the full test suite while Solr is loading (which can take 10–15 seconds). RSpec runs specs in a randomized order, so there’s a good chance Solr will have time to start up before a Search spec is executed.

One last thing: let’s make it easy for colleagues to run the test suite without any tricky details.

bin/test is a bash script that wraps around the rspec command (it even forwards arguments) that starts up Solr before running the test suite, then shuts it down afterwards:

#!/bin/sh
bundle exec rake sunspot:solr:start RAILS_ENV=test
bundle exec rspec "$@"
code=$?
bundle exec rake sunspot:solr:stop RAILS_ENV=test
exit $code

We can even leverage this script in TravisCI!

# .travis.yml
script:
  - bundle exec rake db:setup RAILS_ENV=test
  - bin/test

Now your search specs won’t cloud up the rest of your test suite!

Want the TL;DR version? Here’s a commit timeline of the changes.