Green Is My Favorite Color


As a relative newbie to Solano Labs specifically and parallelized testing in general, I’d like to share the solutions to common on-boarding issues when moving jobs to Solano CI and getting green test passes:

Where did my dependency go?

My train of thought debugging a job recently:

  • I can see the file being required by that Dir.each right there! Why isn’t it being loaded???
  • I’ll explicitly require all the individual files that should be included.
  • A quick ‘find’ to list the files…
  • Oh…now I remember: Dir.each does not guarantee the order on every system.
  • Dir.sort.each to the rescue.
  • Well almost…there are a couple “files = Dir[]; files.each” that need a well placed “.sort”.

William wrote a detailed explanation of why this was causing problems in a specific case three years ago, but it still pops up too regularly.

The environment matters

Sometimes it is just a single environment variable standing in the way of a successful Solano CI build. Temporarily adding a “env && “ at the beginning of a post_setup hook and comparing the results with a developer’s system can help identify any environment variables that should be included.

Another possible point of failure is setting an environment variable (i.e. RAILS_ENV) only in specific Solano hooks. Are you certain that it should be set for only those specific hooks?

Parallel tests accessing the same file system, database, or other resources can lead to locks and/or state uncertainty. Using the Solano CI set environment variables can prevent your tests from conflicting with each other. Solano automatically creates configuration YAML files for many popular services as well as providing matching environment variables, so why not use them? My favorites in isolating workers in regards to the filesystem are $TDDIUM_TID and $TDDIUM_TMPDIR. The presence of $TDDIUM can be used to differentiate between the Solano CI and any other environments.

Test order disorder

If you find seemingly random tests failing, or a specific test failing intermittently across Solano CI jobs, the cause may be related to the test order. Solano’s troubleshooting guide includes steps to help identify the underlying cause. Keep in mind that a test failing may be due to how previously run tests, or any between-test code (like potentially in spec_helper.rb) are leaving the workspace.

To run multiple ruby tests at once you can use `bundle exec ruby -I.:lib -e “ARGV.each{|f| require f}” file1 file2`, replacing “file1 file2” with the files you want to include and ensuring the -I switches are pointing to the correct library script directories. The rspec –pattern switch (aka -P) can take a comma separated list, like:`bundle exec rspec -P “path/to/example1_spec.rb,path/to/example2_spec.rb”`

The docs are your friends

In addition to the troubleshooting guide, the handling failures section of the docs is a very good resource. If you come across a “it’s working on my local system…why not on Solano CI?!?” moment, I would highly recommend taking a look, particularly at the parallelism and intermittent failures section.

Post a Comment