This guest post was written by Deepa Padmanabhan, Lead Quality Engineer at BigCommerce, a large e-commerce platform enabling hundreds of businesses and processing over $9 Billion in sales. This is their experience with using Solano CI, as well as Solano Services to custom-tailor solutions to address their specific needs, and continue scaling seamlessly to deliver value to the customers that rely on them. Keep up with everything else BigCommerce engineering is working on by visiting their blog.
BigCommerce is an e-commerce platform that powers hundreds of online stores across continents. Our hosted shopping cart service processes billions of dollars in total sales from multiple merchants worldwide. Customers rely on us to operate a professional website that generates online revenue for their business. Over hundreds of new features are added to the platform that enables clients to ‘Sell More’ successfully. It is imperative that such a powerful service be highly available, reliable and robust. Quality is critical at BigCommerce and we try relentlessly to deliver high quality products day in and day out for the customers who rely on us.
Our engineering philosophy advocates quality at every level starting from design to product release. With over 100 engineers working on our codebase and delivering product continuously we have a product lifecycle that requires us to follow rigorous testing process. Automating the tests contributes to a significant part of our quality process to ensure high quality and frequent deployment to customers. Every engineer takes immense pride in automating tests as new features are developed and automation team ensures validating every change using these tests. Here is the deployment flow we follow at BigCommerce to release high quality code to customers
Continuous Deployment Flow
Automated tests have grown significantly and the need for regularly testing the quality of the builds and to catch issues instantly has led us to invest in a continuous integration tool. At BigCommerce we rely on Solano Labs for continuous integration and we are happy we went this route. We explored many other options including maintaining our own CI machines but Solano CI outweighed all of them for a number of reasons. There are several features that Solano CI provides that has greatly improved the engineering team’s efficiency.
Highlighting some of the notable features
With developers merging code from different time zones, we had a large number of test builds queued up constantly. UI test suites took more than 5 hours to run causing significant delays in production release, less frequent test runs and accumulating breaking checkins. Solano CI’s build parallelization made this a breeze, where multiple test suites can be run simultaneously. Each build is further parallelized to run tests within a suite across multiple workers. This has significantly reduced the total time needed to run a suite. Our basic suite runs in less than 4 minutes and UI suite in 30 minutes thus significantly improving test feedback time. This has enabled us to implement feature branch monitoring thereby catching breaking changes even before they are merged.
Customization with yml:
Several customizations to fit our need have been made using the solano.yml config file. Ability to create stores per batch has made tests less flappy and avoids concurrency issues due to parallel test runs. An internal implementation of test reruns, executes failed tests on the same Solano CI build and updates the final Solano CI output to reflect the rerun results. This has helped us in generating more robust results by eliminating flappers and avoiding manual validation of flappers. Solano Labs also went the extra mile to modify the Solano phpunit and suggested changes to custom test runner.
Grouping tests by environment and/or feature area is another important need for BigCommerce. We group tests by type (UI, API, HTTP), features (cart, checkout) and priority (release blocker, long running tests). Solano Build Profiles feature exactly translates to this need. It allows us to pass profile specific environment variables thereby targeting small test sets and executes only those. Solano API also provides a way to set environment variables per session on the fly thereby changing each test group and running different set of tests depending on the feature branch we are testing. This helps developers to narrow down and test only the product areas that are affected by their change and not wait for all the unrelated tests to run.
Solano Webhooks has helped us automatically kick off and monitor build status without manual intervention. Our deployment workflow has been highly customized to deploy every merge and kick off tests on Solano CI automatically. Integrating Solano CI with our internal deployment tool has helped us to automatically push to production on successful test runs or alert various channels calling for attention on failed runs. Using Solano API we have built an internal result monitoring tool that provides more granular analytics on test results.
A new feature was rolled out just to address our immediate need where production release test builds were waiting in queue for long during busy days. This caused deployment delays and frustration among release team and customers. Profile prioritization enabled moving important test builds up the queue and to be picked up immediately. Production test runs are instant and don’t wait for long anymore. This has significantly improved release cycle time.
Their unparalleled customer support makes it easier for us to address issues immediately. Session specific environment setup and build prioritization are some of the features they rolled out quickly that helped in accelerating our test execution time. Solano Labs has played a major role in helping us setup and rollout continuous integration within a short amount of time. Their continuous support has helped us go a long way in delivering high quality reliable product in a timely fashion. At the moment, every single merge to mainstream is monitored for quality and every single production deploy is guarded by regression suites run on Solano CI.