SCM Caching

Background

Part of the motivation for our recent change to queues (read more here) was to allow us to change to the way we handle SCM caching. We were using a git- and hg-native SCM cache cluster that was approaching a scaling limit. To address this, we’re moving to a much more scalable, distributed repo caching system that can handle repos from all different types of SCMs and uses a snapshot-based approach instead. After the switch, the SCM cache cluster will be retired and the workers will directly handle updating the snapshots after a build has run. This will remove some setup overhead at the beginning of a build (win!), and will enable us to add a lot of new features that people have been asking for – for example:  integration with more SCM and code review systems (i.e., Phabricator), pre-merging with master before a GitHub Pull Request build, or running your own code to enumerate tests.

Timeline

We’ll gradually migrate users to the new SCM cache system over the next few months. We’ve already switched over our internal builds, and have begun testing some customer builds.

If you use Custom Queues, you will need to enable the new routing before we can switch your account over. (See blog post above.)

New Features

One of the first new features allowed by the new these changes is test enumeration for all SCMs. Enumeration allows the usage of Ruby-globs to include and exclude test scripts from your repo to run in your build.

Git builds have had test enumeration for a while, but Mercurial has been missing out. For example, you can add the following to your solano.yml file:

That will include all files in the spec folder that end in _spec.rb then it will exclude any files that are in spec/flaky/. If you still want to select tests manually you can add the following instead.

In the upcoming weeks we will provide a status update on the rollout and provide a more technical dive.

Have any questions or comments about this or any other Solano CI features – please don’t hesitate to contact us via email, twitter, or in the comments section below.

Post a Comment