Our recent changes to the way we handle SCM caching (read more here), has allowed us to improve the way Solano CI handles test enumeration. Previously test enumeration happened before any user provided commands were ran. Enumeration was thus limited to finding files that matched ruby globs and combining them with a list of tests provided in a yml file. While this works in a lot of cases its not very flexible– enter Custom Enumeration.
Custom Enumeration allows more control over which tests are run on a build. This feature adds a new hook that will replace the standard enumeration (old way) with a run time generated list of tests. Since the list of tests is generated at run time it is possible to have much more fine grained control over which tests are ran on your builds. The basic requirements of Custom Enumeration are a command to run (in most cases a script) that generates a json list of tests to run. That’s it, you have complete control over the way a build will decide what tests it will run. We have made some examples of custom enumeration scripts available here and the docs are available here.
Here are a few examples of what can be done with Custom Enumeration.
- Recently Edited Files
- Failed on last build
- Integrations with other services (e.g. Jira, Github issues)
Recently Edited Files
This example uses a naive method of guessing which tests were likely affected by recent changes. It looks at what files have been changed since the last passed build and attempts to map those files to matching test cases. This enumeration would be used in a plan as the first step, and the second step would either run the remaining tests or run a full build. This type of enumeration tries to use a small subset of tests to fail sooner on builds that will fail.
Failed on last build
This type of enumeration can be used for multiple purposes.
- Could be used as the default profile and be made to run only tests that failed on the last build, or if all tests passed then run all tests. This could be helpful on development branches when you are working on fixing tests that are failing.
- This type of enumeration could also be used to have a rerun profile as the last step of a plan if you know that there are flaky tests in the build.
In the next post we will look at a specific example of using custom enumeration with parallel commands.