The humanities scholar in me rebels when I hear the word “experiment.” To me, an experiment is a big, formal thing with a hypothesis and a control group and reproducibilty. Not really my thing. Yet I love when we conduct experiments at Big Nerd Ranch.

What gives? Well, the things we often call “experiments” as programmers aren’t really experiments at all.

Trials

Back in October 2012, I set out on an experiment that was really a trial. Having been a Vim user for years, I was curious about Emacs, so I decided to do an experiment where I would switch to Emacs full-time. I set an initial three-month time frame for making the switch. To me, this wasn’t really an experiment because there was no hypothesis, there were no conditions for success and it certainly wasn’t repeatable—the next developer who tries it might have a completely different experience with it. And at the end of three months, I was a full-time Emacs user. I still haven’t gone back to Vim and I’m not sure that I will. Changing editors was hard but interesting, and I may do it again. In reality, this “experiment” was a trial of Emacs for me. I wanted to learn it and take it for a spin to see how it felt. So I experimented with it.

Flexible changes

Some experiments are just changes that we are unsure about. At Highgroove Studios, we used initials for everyone in our chat room. With 20-30 employees, it was workable: you learned reasonably quickly who JW and JMW were. But once we merged with Big Nerd Ranch and nearly tripled our size, using initials stopped making sense. So someone proposed that we use full names everywhere, not initials. This wasn’t really an experiment; we were just trying out a new way of doing things. Why call it an experiment, then? Because we knew that it might not work, and we wanted to take a relaxed approach to the change. If you declare that “from now on, everyone uses full names” and update the intranet with the directive and then it doesn’t work out, then you have to go back and un-declare it, then update the intranet again. But if you couch it as an experiment and you see it is no good, you can just go back to business as usual.

Information-gathering exercises

The Ruby team at Big Nerd Ranch is broken into smaller teams of four to eight people. Every week, those smaller teams conduct code reviews of their work. The ways teams opt to do this varies, but it is fairly traditional for each member to review another member at the end of an iteration on Tuesday. My team thought there _might _be a better way for us to do code reviews, so we are trying another approach now.

One of our members champions “continual review”: reviewing, and being reviewed, every single day, rather than batch reviewing them at the end of the iteration. The advantage of continual review is that you have fewer commits to review at a time, and the code-writer gets faster feedback to work with. The problem with continual review is that it adds another thing you have to do every day. To try and approximate the continual review advantages while minimizing its disadvantages, one team member suggested doing code reviews twice a week, on Friday and Tuesday. This gets us a tighter feedback loop but doesn’t make us review every single day. After much discussion, we decided to experiment with this as a team.

This is much more of a traditional experiment: we do know what the other way looks like, and at the end of two or three iterations we will discuss how we liked this, its advantages and disadvantages, and how to go forward. I even think that most of us have a hypothesis (that two times a week will be better than one). But we aren’t committed to that outcome, and really want to see how things turn out.

Experiments and play

These three kinds of experiments are all reasonably narrowly focused process and/or tooling changes designed to make us a better, more productive company and individuals. Why are these “experiments” at all? Honestly, I think it is because part of us associates “experimenting” with “play,” and we want that bit of lightness in these changes. We don’t want our editor to be a drudge, we don’t want the boss to issue demands, we don’t want one team member to tell us what to do. So we play at it. We play with it. We give ourselves room to move, to change, to alter the plans. Experiments are really chances to play at work. They are places to take risks and chances without serious consequences. Experiments. They are a Very Good Thing. And a Very Fun Thing.