Our Integration Tests

Testing “outside-in” is a model that’s coming into favor, in the form of BDD and tools and Cucumber.   High-level tests are easy to understand and decoupled enough from the underlying implementation that they survive significant architectural changes.   We’ve described some our API testing strategy using Cucumber in an earlier post.  Today, I’d like to show some examples of how we use Cucumber for really high level tests.

First, a little background:  we follow a staged rollout model. We keep a production branch that’s always deployable, master that’s deployable but not integration tested, and topic branches for in-process development.   As we stabilize features, we merge them into master, and deploy master into a staging environment.   We run system qualification tests against our staging environment to validate end-to-end coherence.  Once all of our system qualification tests pass, we merge master into production and release.  We drink the devops Kool-Aid, and the ability to automatically qualify our complex system and then automatically deploy it is essential to running a quality service.

Our system qualification tests are written in Cucumber, using the aruba CLI interface gem.  They all follow a similar flow:

  1. Check out a fresh copy of a sample Ruby application repo from github
  2. Find or create a gemset for the app
  3. Install the latest tddium-preview gem into the gemset
  4. Log in to Tddium
  5. Set up a Tddium suite for the repo
  6. Start tests and wait for them to complete
  7. Verify the output
Here’s an example scenario:

Internally, these steps try to be as close as possible to what a real user would do:

There are some gotchas to keep in mind about aruba:
  • It expects commands to finish in 2 seconds.  Running tests in Tddium can take far longer than that, especially in our test environment where we don’t run many tests in parallel.  Conveniently, it’s easy to increase aruba’s timeout.
  • Working with git repos is tricky since aruba by default uses a temp directory in the current directory.  It’s just as easy to tell it to use the system TMPDIR.
To handle these cases, we install a Before hook:

These cucumber-based integration tests are far friendlier than the bash scripts they replace.  There are still some pieces to wrap up, but we’re well on the way to fully automated qualification of major system changes, from the details that are already covered by rspec and capybara tests, all the way to these end-to-end tests.  In fact, we’re not too far away from using our production environment to qualify our staging environment in parallel…

Post a Comment