Secure Environment Variables UI


While it has been possible to set secure environment variables from the Solano CLI for some time (see docs here), there has been no way to do it from the app or to see what values are currently set. We have added a new page to Organization settings that will allow Org admins to manage variables that are not set from a config file.


We are also adding a new tab to the report page that will list the environmental variables in order of precedence for a build. This page will display the list of Variables, their scope, and their value stared out. Org Admins will be able to toggle showing the actual values.



Have any questions or comments about this or any other Solano CI features – please don’t hesitate to contact us via email, twitter, or in the comments section below.


1 Comment

Easy Continuous Deployment You Can Trust ft. Apica


Thank you to everyone who joined our webinar! In case you were unable to do so, then worry not. The full presentation, including every slide and resource, is available in this post.

In this presentation, Brian and Troy demonstrated a continuous deployment release process using GitHub, Solano CI, Apica, and AWS CodePipeline. The release process includes smoke, unit, integration and load tests to guarantee issue-free deployments, enabling a safe and easy push-button deploy process.

Brian and Troy also demonstrated how to:

  • Build your software release pipeline by connecting different steps into an automated workflow using AWS CodePipeline
  • Capture accurate and consistent performance data for each of your releases
  • Automatically stop a release when performance or errors don’t meet predefined criteria
  • Auto-parallelize your test runs with a Continuous Integration server using Solano CI

Check out the full presentation video here

Lastly, we would like to thank Apica for co-hosting this webinar, and AWS for their support. We <3 our partners!

Any questions/feedback? Let us know! We would love to have you present at our next webinar, stay tuned for details!

Leave a comment

Continuous Integration at BigCommerce


This guest post was written by Deepa Padmanabhan, Lead Quality Engineer at BigCommerce, a large e-commerce platform enabling hundreds of businesses and processing over $9 Billion in sales. This is their experience with using Solano CI, as well as Solano Services to custom-tailor solutions to address their specific needs, and continue scaling seamlessly to deliver value to the customers that rely on them. Keep up with everything else BigCommerce engineering is working on by visiting their blog

BigCommerce is an e-commerce platform that powers hundreds of online stores across continents. Our hosted shopping cart service processes billions of dollars in total sales from multiple merchants worldwide. Customers rely on us to operate a professional website that generates online revenue for their business. Over hundreds of new features are added to the platform that enables clients to ‘Sell More’ successfully. It is imperative that such a powerful service be highly available, reliable and robust. Quality is critical at BigCommerce and we try relentlessly to deliver high quality products day in and day out for the customers who rely on us.

Our engineering philosophy advocates quality at every level starting from design to product release. With over 100 engineers working on our codebase and delivering product continuously we have a product lifecycle that requires us to follow rigorous testing process. Automating the tests contributes to a significant part of our quality process to ensure high quality and frequent deployment to customers. Every engineer takes immense pride in automating tests as new features are developed and automation team ensures validating every change using these tests. Here is the deployment flow we follow at BigCommerce to release high quality code to customers


Continuous Deployment Flow

continuous deployment flow

Automated tests have grown significantly and the need for regularly testing the quality of the builds and to catch issues instantly has led us to invest in a continuous integration tool. At BigCommerce we rely on Solano Labs for continuous integration and we are happy we went this route. We explored many other options including maintaining our own CI machines but Solano CI outweighed all of them for a number of reasons. There are several features that Solano CI provides that has greatly improved the engineering team’s efficiency.


Highlighting some of the notable features

Build Parallelization:

With developers merging code from different time zones, we had a large number of test builds queued up constantly. UI test suites took more than 5 hours to run causing significant delays in production release, less frequent test runs and accumulating breaking checkins. Solano CI’s build parallelization made this a breeze, where multiple test suites can be run simultaneously. Each build is further parallelized to run tests within a suite across multiple workers. This has significantly reduced the total time needed to run a suite. Our basic suite runs in less than 4 minutes and UI suite in 30 minutes thus significantly improving test feedback time. This has enabled us to implement feature branch monitoring thereby catching breaking changes even before they are merged.


Customization with yml:

Several customizations to fit our need have been made using the solano.yml config file. Ability to create stores per batch has made tests less flappy and avoids concurrency issues due to parallel test runs.  An internal implementation of test reruns, executes failed tests on the same Solano CI build and updates the final Solano CI output to reflect the rerun results. This has helped us in generating more robust results by eliminating flappers and avoiding manual validation of flappers. Solano Labs also went the extra mile to modify the Solano phpunit and suggested changes to custom test runner.



Grouping tests by environment and/or feature area is another important need for BigCommerce. We group tests by type (UI, API, HTTP), features (cart, checkout) and priority (release blocker, long running tests). Solano Build Profiles feature exactly translates to this need. It allows us to pass profile specific environment variables thereby targeting small test sets and executes only those. Solano API also provides a way to set environment variables per session on the fly thereby changing each test group and running different set of tests depending on the feature branch we are testing. This helps developers to narrow down and test only the product areas that are affected by their change and not wait for all the unrelated tests to run.


Solano API:

Solano Webhooks has helped us automatically kick off and monitor build status without manual intervention. Our deployment workflow has been highly customized to deploy every merge and kick off tests on Solano CI automatically. Integrating Solano CI with our internal deployment tool has helped us to automatically push to production on successful test runs or alert various channels calling for attention on failed runs. Using Solano API we have built an internal result monitoring tool that provides more granular analytics on test results.


Build Prioritization:

A new feature was rolled out just to address our immediate need where production release test builds were waiting in queue for long during busy days. This caused deployment delays and frustration among release team and customers. Profile prioritization enabled moving important test builds up the queue and to be picked up immediately. Production test runs are instant and don’t wait for long anymore. This has significantly improved release cycle time.


Customer Support:

Their unparalleled customer support makes it easier for us to address issues immediately. Session specific environment setup and build prioritization are some of the features they rolled out quickly that helped in accelerating our test execution time. Solano Labs has played a major role in helping us setup and rollout continuous integration within a short amount of time. Their continuous support has helped us go a long way in delivering high quality reliable product in a timely fashion. At the moment, every single merge to mainstream is monitored for quality and every single production deploy is guarded by regression suites run on Solano CI.


Contact us to learn how Solano CI and Solano Services can help address your organization’s unique needs too. To learn more about BigCommerce and their e-commerce platform’s latest, follow this link






Leave a comment

Custom Enumeration


Our recent changes to the way we handle SCM caching (read more here), has allowed us to improve the way Solano CI handles test enumeration. Previously test enumeration happened before any user provided commands were ran. Enumeration was thus limited to finding files that matched ruby globs and combining them with a list of tests provided in a yml file. While this works in a lot of cases its not very flexible– enter Custom Enumeration.

Custom Enumeration allows more control over which tests are run on a build. This feature adds a new hook that will replace the standard enumeration (old way) with a run time generated list of tests. Since the list of tests is generated at run time it is possible to have much more fine grained control over which tests are ran on your builds. The basic requirements of Custom Enumeration are a command to run (in most cases a script) that generates a json list of tests to run. That’s it, you have complete control over the way a build will decide what tests it will run. We have made some examples of custom enumeration scripts available here and the docs are available here.


Here are a few examples of what can be done with Custom Enumeration.

  • Recently Edited Files
  • Failed on last build
  • Integrations with other services (e.g. Jira, Github issues)
Recently Edited Files

This example uses a naive method of guessing which tests were likely affected by recent changes. It looks at what files have been changed since the last passed build and attempts to map those files to matching test cases. This enumeration would be used in a plan as the first step, and the second step would either run the remaining tests or run a full build. This type of enumeration tries to use a small subset of tests to fail sooner on builds that will fail.

Failed on last build

This type of enumeration can be used for multiple purposes.

  1. Could be used as the default profile and be made to run only tests that failed on the last build, or if all tests passed then run all tests. This could be helpful on development branches when you are working on fixing tests that are failing.
  2. This type of enumeration could also be used to have a rerun profile as the last step of a plan if you know that there are flaky tests in the build.

In the next post we will look at a specific example of using custom enumeration with parallel commands.

Have any questions or comments about this or any other Solano CI features – please don’t hesitate to contact us via email, twitter, or in the comments section below.

Leave a comment

Welcome to Solano: Concurrency and Parallelism

Welcome to Welcome to Solano! In this blog series, we’ll examine a few key concepts and features of Solano Labs’ flagship product, Solano CI. The goal of this series is to help orient new users while learning to use the system, and serve as a refresher for current users; we hope you find them handy. Please let us know what you think in the comments or if you have any questions send us a note at


Solano CI is a powerful and complex system; it is, by design, highly flexible in its configuration and potential scale so that it can support the large variety of use cases that CI/CD systems are often relied upon to perform. However, at its core, Solano CI’s primary function is to help its users produce higher quality software.

One way it does this is by returning the results of automated tests quickly. To do this, Solano CI supports the ability to perform multiple actions at the same time and thereby reduce turnaround time by simply doing more at once. There are two simple ways that Solano CI does this: (1) running multiple build “sessions” at once, or “Concurrency;” and (2) automatically subdividing the tasks performed within a session and distributing that work to “workers” to perform them at the same time, or “Parallelism.”

The number of concurrent sessions a user can perform at once is set by that user’s account plan. Each session occupies what we call a “build slot,” and users can only run as many sessions at once as their plan has build slots. When all build slots are occupied by actively running sessions, subsequent sessions that are started are queued until a running session finishes and a slot becomes available. [Editor’s Note: We’ll discuss Queues in more detail in a subsequent post; we’ll provide a link to that post here once it’s posted.]

Our free trial and metered hourly plans, e.g. the Large plan, by default all have two build slots. At the Pro and Enterprise plan levels, the number is customizable.

As a session goes through the build setup process, Solano CI spins up a number of parallel workers to actually perform tasks, usually tests, the user wants to execute. These parallel workers are Docker-based containers, and can run either all together on a single VM or on an orchestrated set of VMs. When the workers are up, the system will scan the repo’s filesystem for test files based on the user’s configuration settings, collect that set of files, and then subdivide it into discrete batches that are distributed to the workers for execution. This distribution of the batches over time over the duration of the test execution phase of the session is our automatic parallelism.

One important conceptual note: the number of workers used per session is set as part of the plan configuration, like the number of build slots, and are the same for all of a plan’s build slots, regardless of which repo they build, and are not free-ranging. This is often confusing for many new users, as they often think of workers as just a set of units in a resource pool that can be arbitrarily configured on a per-repo basis, and so if a smaller repo is built they want to redirect the superfluous workers towards other larger repos’ builds; this isn’t possible in Solano CI. A helpful analogy for constructing a mental model is that of lanes in a bowling alley: just as a bowling lane uses a predetermined number of pins and all other lanes use that same number, so to build slots have a number of workers they use by default as set by the user’s plan.

The number of workers per build slot is highly diverse by plan, even within the set of metered plans. A list of the workers by metered plan is available on our Product page; the Pro and Enterprise plans’ worker configurations are customizable.

Leave a comment

Testing at Tobi


The following is a guest post by Josh Brown, Senior DevOps Engineer at Tobi. Tobi is an online fashion label, a unique combination of fashion, technology, and retail requiring a world-class engineering process in order to maintain and grow such a robust e-Commerce platform. Solano CI helped Tobi achieve these goals by reducing their testing time to the time it takes to get a cup of coffee— coffee time! 

At Tobi, we serve millions of customers from around the world with an e-commerce platform that we built from scratch. Not only that, but all of our internal applications, like inventory tracking and product lifecycle management, are all built in-house. The entirety of our company’s infrastructure is built by us and so we’re acutely attuned to and invested in its performance and reliability. This need for a great and reliable platform and experience by both ourselves and our customers has led us to automated testing and Continuous Integration. Testing and CI are crucial to us, and here I’ll outline how we’re testing and with what systems we currently use, and why.

Our codebase is over 9 years old. Many of our engineers, then, interact with code that may not have changed in some time, let alone actually working directly on that code itself! So they rely on our unit, functional, and integration tests to confidently make changes to those portions of the code that have been frozen for years. These give us the confidence to build on that legacy with the assurance that it serves as a stable foundation.

Since we have over 400 employees using our in-house applications daily, we want engineers to write as many tests as possible so that our employees can execute optimally.  Our team is encouraged to write a lot automated tests and know that our CI system can handle as many tests as the codebase requires to remain stable. CI is a core part of our workflow and all engineers at Tobi utilize it for our multiple, daily releases.

The infrastructure we use to run our build and test process is based around Solano CI. We came to Solano CI in early 2016 from CircleCI, where our builds were taking about 50 minutes (a significant improvement over the 1.5 hours they took before on Jenkins). We liked both Circle and Solano because they were SaaS offerings (so we didn’t have to manage the underlying system). Both rely on a YAML configuration file to configure the build environment, select which tests to run, and manage parallelism.

The biggest reason we switched to Solano CI was, in a phrase, “coffee time.” Solano’s team encouraged me to think about what my team could do if we could get our build and test results back in the time it takes to get a cup of coffee, and intrigued, I challenged them to show me that Solano CI could make that happen. The system’s automated scheduling and parallelism were impressive, and between my conceptual understanding of setting up through a YAML config file from CircleCI and Solano’s docs I got most of the way by myself. But the biggest win, and the reason we decided to buy, was their team’s support: I took my own work to them and they offered recommendations for more efficient ways to do things like set up the database and precompile assets that, ultimately, got us to coffee time. That expertise was really valuable, and their Support team has continually shown that the Solano team are experts of CI and its best practices.

As Tobi’s operations have grown our processes needed to as well, and this set up is the current iteration of our continued efforts to create a strong testing infrastructure. We can confidently produce a resilient and dependable e-commerce platform for our customers and supporting backend infrastructure for our colleagues. And with the testing backstop secure we can focus on what matters most – providing the best service experience and bringing our customers amazing fashion.


You can learn more about Tobi and what they are up to by clicking here

Leave a comment

Announcing Build Pipelines

We’re very excited to announce the release of Solano Build Pipelines. Build Pipelines are a way for you to chain together multiple Solano CI sessions into a seamless Continuous Deployment pipeline.


Solano Build Pipeline Animated


Each pipeline step represents a separate Solano CI session, so each runs with its own set of Solano’s parallel workers. This means that you can utilize the power and flexibility of Solano CI’s auto-parallelization to get faster build results, thereby accelerating your deploy times. Writing a deploy step can be as simple as executing a custom shell script; or you can utilize one of our many deployment integrations. Pipelines will only continue execution if every step preceding a particular step successfully passes.

Along with the release of Build Pipelines, you’re now able to run Solano Build Profiles in parallel! This means that Build Matrices can also run in parallel by default. To enable this functionality users can change their plan key to pipeline, or can contact and we can turn on the functionality for the plan key.

We’re pumped to see how you’re planning on using Build Pipelines. Tweet us @SolanoLabs (and me @b_kendz) with your crazy pipeline configurations – see if you can come up with something useful that’s crazier than this pipeline:

a crazy Solano Build Pipeline

If you want to learn more about Build Pipelines, check out our documentation, or contact us directly at

Leave a comment

Custom Enumeration (command mode)

Parallel command mode can be used to run command mode tests in parallel. Solano CI will create and distribute batches of tests across the workers assigned to your build. For each batch, it will invoke the provided shell command with the test class files it should execute.

This example is available here solanolabs/guzzle, this example uses PHPUnit. Here is what a configuration file for the project might look like.

The project is set to run runs 3 profiles each of which select a different set of tests by running basic_enumeration.php. basic_enumeration.php is ran on each build and decides which tests to run from the profile by applying the following patterns to the files found:

Here are some screen captures of the final builds.

Screen Shot 2016-07-01 at 5.05.37 PM

Screen Shot 2016-07-01 at 5.07.57 PM

Screen Shot 2016-07-01 at 5.06.19 PM

Screen Shot 2016-07-01 at 5.06.44 PM

There are more examples of custom enumeration scripts avalible at solanolabs/solano_plugins/enumeration and the docs are avalible here.

Have any questions or comments about this or any other Solano CI features – please don’t hesitate to contact us via email, twitter, or in the comments section below.

Leave a comment

SCM Caching


Part of the motivation for our recent change to queues (read more here) was to allow us to change to the way we handle SCM caching. We were using a git- and hg-native SCM cache cluster that was approaching a scaling limit. To address this, we’re moving to a much more scalable, distributed repo caching system that can handle repos from all different types of SCMs and uses a snapshot-based approach instead. After the switch, the SCM cache cluster will be retired and the workers will directly handle updating the snapshots after a build has run. This will remove some setup overhead at the beginning of a build (win!), and will enable us to add a lot of new features that people have been asking for – for example:  integration with more SCM and code review systems (i.e., Phabricator), pre-merging with master before a GitHub Pull Request build, or running your own code to enumerate tests.


We’ll gradually migrate users to the new SCM cache system over the next few months. We’ve already switched over our internal builds, and have begun testing some customer builds.

If you use Custom Queues, you will need to enable the new routing before we can switch your account over. (See blog post above.)

New Features

One of the first new features allowed by the new these changes is test enumeration for all SCMs. Enumeration allows the usage of Ruby-globs to include and exclude test scripts from your repo to run in your build.

Git builds have had test enumeration for a while, but Mercurial has been missing out. For example, you can add the following to your solano.yml file:

That will include all files in the spec folder that end in _spec.rb then it will exclude any files that are in spec/flaky/. If you still want to select tests manually you can add the following instead.

In the upcoming weeks we will provide a status update on the rollout and provide a more technical dive.

Have any questions or comments about this or any other Solano CI features – please don’t hesitate to contact us via email, twitter, or in the comments section below.

Leave a comment

Queues are on the Move

Update July 22nd 2016 – Queue Routing has been moved to the UI, routing using solano.yml is no longer supported


A little over a year ago we released a feature called Custom Queues which exposed some of our internal routing logic to better select the compute resource that your build runs on. If you wish to use Docker containers you can route your builds to the “docker” queue to take advantage of Solano CI’s built-in Docker environment (read more). If you have a large test suite that needs to run using beefy machines with multiple GPUs, we’ll set up a queue for you to route your vector processing jobs through. 

Moving configuration items from the source-controlled solano.yml file isn’t a decision we take lightly. We’re big fans of moving away from the decade-old model of having your build’s configuration settings stored within the build system in a cryptic XML format. We are working on changes to the method we use to process configuration files very early in the session lifecycle. Configuration values that we absolutely must know before a worker starts now need to be set using the UI. We have some exciting enhancements coming soon that we’re super jazzed to share with you all, and moving to a UI-based queue routing configuration is the first step in enabling these enhancements.


For users who don’t currently use Custom Queues, this change won’t have any impact on your builds. For those of you that are using Custom Queues, we’ve pre-migrated the queue settings from your most recently run session to help you in the transition. All you need to do is verify that the auto-generated Queue Routers are correct, and click a button to enable the new routing scheme. We will be deprecating the config file approach to queue routing in July 2016 – we will be in contact with you prior to the cutoff to make sure everyone is migrated and ready for the change.

How to migrate

Step 1.

Visit your Solano CI Dashboard page, and click the gear icon for the repo you wish to configure. This will take you to the Repo Settings page.

repo setting page link location

Step 2.

Click “Queue Routing” in the left sidebar to bring up the Queue Routing settings page. Using the form at the bottom of the page, add branch/profile combinations and indicate which queue you want each combination to route to. You can use ‘*’ in the branch field to match all branches. If you’re not using profiles or don’t wish to change routing rules based on profile you can simply leave the field blank.Queue Routing settings page

Step 3.

Click “Save” to add the new route to your list of active Queue Routers. The routes are evaluated from top to bottom, the first rule that matches the branch & profile of the session that is being routed will be the one that determines the queue to use.

Step 4.

Once you validate that your Queue Routers are configured correctly, you can turn on Queue Routing for this branch by toggling the slider at the top of the page so that “Enabled” is highlighted in green. Congratulations, you’re fully migrated! Now would be a good time to remove any references to queue routing in your solano.yml to avoid any future confusion.

enable queue routing

Like I mentioned earlier, we’ve automatically populated Queue Routers for repos that have queues defined in their solano.yml configuration file. All you should need to do is verify that the rules are correct, and click “Enable” to turn on the new Queue Routing. If you don’t see your queues auto-populated, please send us a link to a Session that has a queue defined in its solano.yml, and we’ll auto-generate the routers for you.

Stay tuned for some exciting news about upcoming changes that will have a big positive impact on your build performance.

Have any questions or comments about this or any other Solano CI features – please don’t hesitate to contact us via email, twitter, or in the comments section below.

Leave a comment