Speeding Up our Test Suite: From 2.5 hours to 20 mins with Solano Labs

by Drew Blas, Software Engineer, Chargify.com

At Chargify we rely heavily on automated testing to ensure that we always maintain a working app. With so many customers and a heavily utilized API, it’s critical that we maintain complete backwards compatibility and ensure we don’t impact existing customer operations. That’s why our test suite consists of thousands of tests for gateway interactions, workflows, and response formats. Unfortunately, it also made for painfully slow development. By integrating Solano Labs’ TDDium, we were able to reduce our complete test suite run from 2.5 hours to 20 minutes. This incredible improvement helped to promote a radical change in our testing attitudes while keeping our deployment cycle as fast and agile as possible.

What we were doing before Solano Labs’ Tddium

Our previous, homegrown Continuous Integration environment relied on single server to do each build. Unfortunately, because of the intense load from our test suite, we had numerous limitations. We could only run a single build at a time, we often couldn’t see results until it was complete, and debugging test failures was a major pain (that involved sshing into the server to try and extract environment-specific issues). Worst of all, waiting for several hours between builds meant we weren’t able to get quick feedback about broken tests and prevented us from truly making good use of our tests during development.

These factors meant we not only spent a lot of time ‘fixing’ the build, but that we often skipped steps or didn’t test properly because of time constraints. Thanks to TDDium, that process has been greatly streamlined so that we can perform TDD the way it was meant to be.

Faster Testing with Solano Labs

Of course, with a codebase as big as ours, switching to TDDium was not instantaneous. We found a lot of areas in our tests that had to be improved or refactored. Some of these changes were due to the different runtime environments, but most had to do with brittle tests that did not respond well to the randomized distributed testing model. Many of our tests were order dependent or conflicted with other tests and couldn’t be reliably run in parallel. However, TDDium did a great job of giving us the tools needed to re-create and fix these issues. There’s extensive logging available and even an option to pinpoint the exact tests and ordering used in particular run. In the cases where we needed help, the support from Solano Labs was top-notch. They worked side-by-side with us on issues where we needed assistance and saved us even more time. All this helped us to decouple our tests and prepare them for highly-distributed execution. The process was definitely worth it: we wound up with much more robust test suite that runs in a fraction of the time!

Having a test suite that runs quickly means we can get much faster feedback about changes that we make.  Instead of waiting until the next day to see if a simple change ‘breaks the build’, we can instead update, test, package, and deploy at a much more rapid pace.  Reducing the turnaround time on getting code to production is truly a lynchpin in any agile operation!

Easy database support!

Surprisingly, one of the easiest integration points with TDDium is the database support. The connections are mostly automatic and it’s easy to specify multiple databases in specific versions. Our tests rely on Redis, Mongo, & MySQL, and all worked basically out of the box! External service integration (like Campfire & Github) has also helped to make our lives easier and improve our team’s workflow.

Help with Rails Upgrades

Finally, we’re especially thankful to have set up TDDium before we started on our major upgrade from Rails 2.3 to 3.2. It was a huge undertaking and the ever-watchful eye of our tests is a big reason for our success. Constantly running builds and getting rapid feedback about major architectural changes allowed us keep making progress while not affecting our day-to-day development. TDDium gracefully handled the constantly changing configuration of our app with aplomb!

Ultimately, TDDium has given us a wonderful collaborative environment for running our tests AND provided an order of magnitude improvement in build time to keep our development team happily coding away. Thanks!

Post a Comment