Custom Enumeration (command mode)

Parallel command mode can be used to run command mode tests in parallel. Solano CI will create and distribute batches of tests across the workers assigned to your build. For each batch, it will invoke the provided shell command with the test class files it should execute.

This example is available here solanolabs/guzzle, this example uses PHPUnit. Here is what a configuration file for the project might look like.

The project is set to run runs 3 profiles each of which select a different set of tests by running basic_enumeration.php. basic_enumeration.php is ran on each build and decides which tests to run from the profile by applying the following patterns to the files found:

Here are some screen captures of the final builds.

Screen Shot 2016-07-01 at 5.05.37 PM

Screen Shot 2016-07-01 at 5.07.57 PM

Screen Shot 2016-07-01 at 5.06.19 PM

Screen Shot 2016-07-01 at 5.06.44 PM

There are more examples of custom enumeration scripts avalible at solanolabs/solano_plugins/enumeration and the docs are avalible here.

Have any questions or comments about this or any other Solano CI features – please don’t hesitate to contact us via email, twitter, or in the comments section below.

Leave a comment

SCM Caching

Background

Part of the motivation for our recent change to queues (read more here) was to allow us to change to the way we handle SCM caching. We were using a git- and hg-native SCM cache cluster that was approaching a scaling limit. To address this, we’re moving to a much more scalable, distributed repo caching system that can handle repos from all different types of SCMs and uses a snapshot-based approach instead. After the switch, the SCM cache cluster will be retired and the workers will directly handle updating the snapshots after a build has run. This will remove some setup overhead at the beginning of a build (win!), and will enable us to add a lot of new features that people have been asking for – for example:  integration with more SCM and code review systems (i.e., Phabricator), pre-merging with master before a GitHub Pull Request build, or running your own code to enumerate tests.

Timeline

We’ll gradually migrate users to the new SCM cache system over the next few months. We’ve already switched over our internal builds, and have begun testing some customer builds.

If you use Custom Queues, you will need to enable the new routing before we can switch your account over. (See blog post above.)

New Features

One of the first new features allowed by the new these changes is test enumeration for all SCMs. Enumeration allows the usage of Ruby-globs to include and exclude test scripts from your repo to run in your build.

Git builds have had test enumeration for a while, but Mercurial has been missing out. For example, you can add the following to your solano.yml file:

That will include all files in the spec folder that end in _spec.rb then it will exclude any files that are in spec/flaky/. If you still want to select tests manually you can add the following instead.

In the upcoming weeks we will provide a status update on the rollout and provide a more technical dive.

Have any questions or comments about this or any other Solano CI features – please don’t hesitate to contact us via email, twitter, or in the comments section below.

Leave a comment

Queues are on the Move

Update July 22nd 2016 – Queue Routing has been moved to the UI, routing using solano.yml is no longer supported

Background

A little over a year ago we released a feature called Custom Queues which exposed some of our internal routing logic to better select the compute resource that your build runs on. If you wish to use Docker containers you can route your builds to the “docker” queue to take advantage of Solano CI’s built-in Docker environment (read more). If you have a large test suite that needs to run using beefy machines with multiple GPUs, we’ll set up a queue for you to route your vector processing jobs through. 

Moving configuration items from the source-controlled solano.yml file isn’t a decision we take lightly. We’re big fans of moving away from the decade-old model of having your build’s configuration settings stored within the build system in a cryptic XML format. We are working on changes to the method we use to process configuration files very early in the session lifecycle. Configuration values that we absolutely must know before a worker starts now need to be set using the UI. We have some exciting enhancements coming soon that we’re super jazzed to share with you all, and moving to a UI-based queue routing configuration is the first step in enabling these enhancements.

Impact

For users who don’t currently use Custom Queues, this change won’t have any impact on your builds. For those of you that are using Custom Queues, we’ve pre-migrated the queue settings from your most recently run session to help you in the transition. All you need to do is verify that the auto-generated Queue Routers are correct, and click a button to enable the new routing scheme. We will be deprecating the config file approach to queue routing in July 2016 – we will be in contact with you prior to the cutoff to make sure everyone is migrated and ready for the change.

How to migrate

Step 1.

Visit your Solano CI Dashboard page, and click the gear icon for the repo you wish to configure. This will take you to the Repo Settings page.

repo setting page link location

Step 2.

Click “Queue Routing” in the left sidebar to bring up the Queue Routing settings page. Using the form at the bottom of the page, add branch/profile combinations and indicate which queue you want each combination to route to. You can use ‘*’ in the branch field to match all branches. If you’re not using profiles or don’t wish to change routing rules based on profile you can simply leave the field blank.Queue Routing settings page

Step 3.

Click “Save” to add the new route to your list of active Queue Routers. The routes are evaluated from top to bottom, the first rule that matches the branch & profile of the session that is being routed will be the one that determines the queue to use.

Step 4.

Once you validate that your Queue Routers are configured correctly, you can turn on Queue Routing for this branch by toggling the slider at the top of the page so that “Enabled” is highlighted in green. Congratulations, you’re fully migrated! Now would be a good time to remove any references to queue routing in your solano.yml to avoid any future confusion.

enable queue routing

Like I mentioned earlier, we’ve automatically populated Queue Routers for repos that have queues defined in their solano.yml configuration file. All you should need to do is verify that the rules are correct, and click “Enable” to turn on the new Queue Routing. If you don’t see your queues auto-populated, please send us a link to a Session that has a queue defined in its solano.yml, and we’ll auto-generate the routers for you.

Stay tuned for some exciting news about upcoming changes that will have a big positive impact on your build performance.

Have any questions or comments about this or any other Solano CI features – please don’t hesitate to contact us via email, twitter, or in the comments section below.

Leave a comment

Webinar Recap: Easy Continuous Deployment You Can Trust

Thank you to everyone who joined us for our webinar last week, produced in collaboration with Sauce Labs, titled “Easy Continuous Deployment You Can Trust”, featuring Solano Labs Founding Engineer, Brian Kendzior, and Sauce Labs Solutions Architect, Neil Manvar.

In this presentation, Brian and Neil demonstrated a continuous deployment release process that used GitHub, Solano CI, Sauce Labs, and AWS CodePipeline. The release process included smoke, unit, integration and browser tests to guarantee issue-free deployments, enabling a safe and easy push-button deploy process.

Brian and Neil also demonstrated how to:

  • Build your software release pipeline by connecting different steps into an automated workflow using AWS CodePipeline
  • Run your automated web tests on any browser and operating system combination using Sauce Labs
  • Auto-parallelize your test runs with a Continuous Integration server using Solano CI

Check out the video here:

Slides are available here:

NOTE: These slides do not include the demo – please watch the video above for the demo.

Handy Resource Links:

Got questions, comments, or requests for what we should do for our next webinar?  Feel free to use the comment section below, email us at support@solanolabs.com, or send us a tweet @SolanoLabs!

Leave a comment

Report Page Interface Enhancements

We are very pleased to present the first design update of 2016 to the Solano CI product: a new Report Page layout that is better organized, introduces new interface components, and is much easier to use.

The Challenge

The “Report Page”—as we call it here at Solano Labs—is the primary view into any given session on Solano CI. It shows every state of a session, from its initial creation, through to a complete collection of test results, build lifecycle timings, and build artifacts.

It has also served as a way for many users to navigate through their workflow, and view other pieces of information that relate to the state or configuration of the organization and repo. Over time, in an effort to provide as much information and control as possible over a growing number of factors, the interface had become less intuitive for long-time users and not as easily ’learnable’ for new users. This causes friction and inefficiency; it is, in a sense, user experience debt.

With so much data and navigation to present in one place, the challenge is this: how do we improve the usability of such a complex view? How do we introduce incremental enhancements while maintaining the best possible usability overall?

First, we listened to you, and we sat with you and watched you use the application; we challenged our own assumptions. We learned a lot.

Second, we reviewed both new and long-time users’ primary expectations of the view to ensure different team roles will be well served by the interface.

And third, we mapped out a plan to arrive at an overall redesign via a careful sequence of incremental improvements to specific areas of the interface.

Then we got to work on the first design update.

New Layout

The first thing you’ll notice about the new layout is that we have removed the sidebar. This is a major change in the interface, and it was made based on both our own use of the Report Page, as well as in response to user feedback. Many users found the placement of status and navigation related to other sessions distracted from the current session’s status and results. By focusing the layout on the current session status, and moving other status and navigation elements to more intuitive locations, we created a new view that is easier to understand at a glance, and affords more comfortable navigation.

Tabbed Sections for Supporting Information

In addition to simplifying the top of the Report Page so you could more easily focus on session status, we also wanted to simplify how you view and navigate the supporting information, such as setup logging, test results, important messages—everything below the main session status display. For complex builds especially, viewing these different sections resulted in lots of scrolling, and intra-page navigation that was often confusing to new users.

Through observation and research it became clear that you were interested in specific kinds of information at specific times, and being able to choose what information was visible by type or purpose would be very helpful. So we divided the supporting information into separate tabbed sections shown immediately below the primary session status. [see 1 below] Now, rather than needing to scroll the length of the view past info you may not need in order to find what you’re looking for, there is a clearly labelled tab such as ‘Build Artifacts’.

The tabbed sections also provide a flexible yet well-structured way for us to present supporting information that would have cluttered the view unnecessarily: the Session History that was in the sidebar is now shown in a tab; in the future important new features on our roadmap will also be found under a new tab.

report-page-update-new-tabbed-sections

Active Session Menu

Another new addition to the Report Page is the Active Session menu. [see 2 below] Having committed to removing the sidebar, we set out to optimize the usability around certain tasks. As mentioned earlier, various components on the Report Page became central to many user’s workflow, and we needed to make sure we didn’t break those flows while solving other things. We found that the most used element in the sidebar is the list ’Current Sessions’, and therefore it would be more useful if it was available on all views in the app. A simple solution emerged quickly: place a control in the header of the page that allows you to quickly see a drop-down that shows all active sessions from not only the Report page, but also the Dashboard, or any other section. The added benefit of this new component is that it provides a convenient way to quickly see just the number of active sessions without needing to click on the drop-down—and see at a glance from one session when the most recent session has started.

Session Navigation

While the Active Session menu allows you to quickly access the most recent build activity in your organization, we also wanted to provide a simple way to navigate to the previous and—when it exists—next builds on the same branch. We added text links in the heading of the Session component for paginated navigation so you can quickly move up and down the timeline of sessions. [see 3 above]

In the future we plan on providing richer controls for navigating the app, so this was an ideal time to step incrementally in that direction by moving the branch name up to the top of the page. Now you’ll see organization, repositories, and branches all in one line. [see 4 above] This will soon become convenient “breadcrumb” navigation into all three areas.

Support Drop-down

Consistent with the changes made to the Active Sessions component, we also moved the Notify Support form. We felt that having a control accessible near the Action menu for opening up a support form made the most sense—often the need for support arises once a user has taken actions such as rebuilding, or opening up a debug console. [see 5 below] In addition, for users on laptop screens the details of the report are now easily visible while entering the details of your support requests.

 

report-page-update-new-support-component

Visual Refresh, Part One

In addition to the new layout and navigation elements, we are also introducing a simplified visual style that is more clear and readable. While we won’t be embracing a trendy ‘flat design’, I’m a firm believer that too much design interferes with usability; by simplifying the visual styling of each component, a complex interface becomes much more clear, and as a result provides a much better experience. We’ll be transitioning to this new style in all areas of the app over the coming months.

One of the promises of Solano CI is to save you time; the more the interface gets out of your way and allows you get your work done quickly and efficiently—less scanning the page for the information you’re after, less searching for a control—the more design has helped deliver on that promise.

The Other CI

For all of us collaborating on design at Solano Labs, CI also stands for Continuous Improvement. This design update is the result of a continuous cycle of learning, ideating, and building: learning what you need to work more confidently and efficiently; ideating ways to meet your needs; building a reliable and usable product for you. As a designer, the most important part of this cycle is the learning: there is nothing that motivates me more than learning about what is and isn’t working for you when you’re logged in to ci.solanolabs.com. So log in, take a look at the latest updates, and please tell us what you think.

Send us any questions or comments via Twitter @SolanoLabs, email at support@solanolabs.com, or the comments section below.

4 Comments

Displaying Reports Earlier

We’re changing how you can view Solano CI’s processing of events that trigger builds, for example like the webhooks sent from GitHub. An event can be a webhook, a push from the command line, a “build now”, or a scheduled build.

We’ve always tracked incoming events (e.g. pushes) separately from viewable Sessions (i.e. Builds), and there hasn’t been a good presentation of events that haven’t had associated Sessions.

 Since Solano CI will (by default) collapse pushes on a branch together into a new build if there’s an existing build running (“CI Squash”), we use this distinction between event and Session to minimize the number of running Sessions, in order to keep your build queues free of extra work.  Also, incoming webhooks are often filtered out or dropped based on repo settings, and it made sense to hide the dropped events from view. This has had the negative UX impact that it may have taken many minutes for a push to be displayed as a visible report, even though it was received successfully and is being properly processed.  We provide controls in your Repo Settings for CI Squash”, and the related behavior of not re-running a new build if there’s already a build on the same commit (“Dedup”).

This change allows us to share more information about these events sooner.

We’re now making visible and linkable reports for the event(s) that will turn into the next build on a branch.  These will show up as “initializing”, and will remain like this while there’s already a running build.  Once the running build on a branch completes, the initializing build will progress to “queued” as normal.  A build that shows up as “initializing” won’t take up a build slot.

Initially we are adding three new report pages:

  1. An updated page for builds in “initializing” state
  2. A page for builds that have been squashed,
  3. A page for duplicate builds that were skipped because of a pre-existing build on the same commit.

Build Initializing

First, let’s talk about the updated “initializing” page. This page will be available as soon as an event is received. As not much has happened yet in this build, it will just contain some basic information such as creation time, type (ci, cli, scheduled), commit id (if available), and event initiator. Once a build actually starts running it will return to the normal report page view.

Screen Shot 2016-01-06 at 4.17.18 PM

Squashed Builds

Next up is “squashed” builds. Until now there has been no display for squashed builds. They now have their own report page that contain a link to the build that ran, and a link to the repo settings page where squash is set. Quick primer on squashed builds: “Solano CI will build only the most recent un-built push on the current branch, skipping intermediate pushes.” Here is an example of what a squashed build now looks like:

Screen Shot 2016-01-06 at 2.56.44 PM

The build number in the title (i.e.#1096) is a link to the build that actually ran. It leads to the normal report page.

Screen Shot 2016-01-06 at 3.02.00 PM

Duplicate Builds

Finally Duplicate builds: Solano CI will not run a build for a webhook (by default) or scheduled build (if you set the option) whose latest commit has already run on the same branch within the past few days. Additionally it is possible to enable this feature repo-wide, to not run a new build if the same commit has been run on any branch in the repo. The Duplicate build page will contain a link to the build that ran, and the latest commit sha, in addition if it was a scheduled build it will contain the Schedule Info. Scheduled builds can be set to always run by selecting “always run” in the schedule builds options.

Screen Shot 2016-01-16 at 12.21.27 PM

Send us any questions or comments via Twitter @SolanoLabs, email support at support@solanolabs.com, or use the comment section below.

We hope you like this change!

Leave a comment

Solano CI Integrates with Amazon EC2 Container Registry at Launch

Today, we are proud to announce our integration of Solano CI with Amazon EC2 Container Registry (ECR).  With Amazon ECR and Solano CI, you can now reliably build, test, and deploy your Docker workflow without operating your own container repositories or scaling your infrastructure.  Amazon ECR is a fully-managed, secure, and highly-available Docker container registry that makes it easy for developers to store, manage, and deploy Docker container images.  In addition to the simplified workflow, you can configure policies to manage permissions and control access to your images using AWS Identity and Access Management (IAM).  There is no easier way to gain fine-grain permission control over your Docker images today.

We decided to create an example repository to demonstrate integrating Amazon ECR and Amazon EC2 Container Service (ECS) with Solano CI. Each commit to the repository will trigger a build, and when all of the tests pass, the new Docker image will be pushed to Amazon ECR and Amazon ECS will start a new container. The step-by-step process we used:

Prepare your local system

  1. Since Amazon ECR is a new service, we had to install a new version of the AWS Command Line Interface to enable the aws ecr command.
  2. Set the environment variables for the aws command:

Setup Amazon ECR and ECS

For Amazon ECR enabled accounts, the Amazon ECS first run wizard allows creating the private Docker image repository in a few easy steps:

Set the repository name:

Aws ECR Step 1

Upon creation of the Amazon ECR registry, run the provided commands. The aws ecr get-login command will output a docker login command to run, but this can be done in one step by running it in a subshell: $(aws ecr get-login --region $AWS_DEFAULT_REGION)

AWS ECR Step 2

After setting up Amazon ECR, the wizard continues with the standard Amazon ECS setup steps.

Amazon ECS Step 3

On the task definition step, clicking the Advanced button allows us to set our own entry point and ensure the command field is empty.

Amazon ECR Step 3 Advanced

Configure the service:

Amazon ECS Step 4

Configure the cluster:

Amazon ECS Step 5

Review the settings and Launch!

Amazon ECS Step 6

Setup Solano CI

After AWS has launched the container, we set Solano CI to automatically build and test on each git push. When all of the tests pass, the new Docker image is pushed to Amazon ECR and Amazon ECS launches a new container from the image.

Inform Solano CI of the repository:

Now that the repository has been registered with Solano CI, we went into the repositories settings by clicking the gear icon for the repository on the Solano CI dashboard:

Repository on Solano CI Dashboard

On the CI Setup tab of Solano CI’s repository settings page we used the values to link the GitHub repository. The values under #2 are set in GitHub’s Tddium service hook. The value under #3 is Solano CI’s repository public key and is added as a GitHub deploy key.

Github configuration

To set the auth credentials necessary for connecting to AWS, the solano config:add command is the recommended way to set sensitive environment variables:

Success!

Now the example repository will trigger a build and deploy to both Amazon ECR and ECS upon passing builds. In particular there are a couple of files that we recommend taking a look at:

solano.yml: This file informs Solano CI to use a Docker enabled queue and specifies the setup hooks, tests, and other settings.

scripts/solano-pre_setup.sh: Installs aws-cli and jq (for JSON processing), logs into Amazon ECR, builds a Docker image, and installs dependencies.

scripts/solano-worker_setup.sh: Starts the docker container and prepares the test database.

scripts/solano-deploy.sh: Verifies the tests pass, pushes the new image to Amazon ECR, registers a new task definition with Amazon ECS, and updates the Amazon ECS service.

Give it a try and send us any questions or comments via Twitter @SolanoLabs, email support at support@solanolabs.com or use the comment section below.

With Amazon ECR and ECS alongside Solano CI: May green builds be with you!

Yoda Broken Build

Leave a comment

Solano CI integrates with AWS CodePipeline

Solano Labs is proud to announce our recent integration with AWS CodePipeline. AWS CodePipeline users can now quickly and easily set up a build and test workflow without manually provisioning an on-premises Jenkins cluster.

Screen Shot 2015-11-16 at 6.07.35 PM
CodePipeline provides users with a standardized and extensible build & deploy pipeline system. Being available as a pipeline step means that Solano CI can be used as a gatekeeper in your fully-automated deploy system. CodePipeline makes it simple to string together multiple build steps into a full-featured and standardized execution pipeline. For example: users can trigger a pipeline execution by uploading their source code to a private AWS S3 bucket, run a Solano CI build on that source to validate that unit and integration tests are passing, deploy the application to production using CodeDeploy, and finish things up with a load test using BlazeMeter.

Here’s a quick video run-through of how easily you can get set up using AWS CodePipeline with Solano CI.

Give it a try, and please reach out if you have questions!

Documentation and Step-by-Step instructions on getting set up with Solano CI andAWS CodePipeline

Find out more about AWS CodePipeline

Solano Labs Press Release

Leave a comment

Docker Container Testing…in Parallel

Like many in DevOps, I’ve sipped the Docker Kool-Aid and like how it tastes. Being able to standup any well-balanced combination of servers pretty much at will to handle load is a beautiful thing. I almost feel embarrassed that it was just a handful of years ago that I had to explain to a previous coworker “if we need to buy another web server, we are going to need another application server and possibly another database server…and at that point we might as well throw in a couple more web servers”. One could say virtual machines, Amazon AWS, and Docker have allowed me to break through the wall of considering servers in permanent integer amounts.

One of my favorite parts of working at Solano Labs is the exposure to the ever-changing landscape of DevOps technologies. We take questions like “do you support technology XYZ?” almost as personal challenges. After a couple Docker-support questions, I decided to whip up a relatively simple example repository demonstrating building Docker images/containers on Solano CI. (After a “do you support deploying to Amazon ECS?” inquiry, I quickly updated the repo to demonstrate that capability.)

Since the repository is only serving a single web page, I initially included a very minimal test suite (basically if the page rendered, it passed) in the container itself. Even though it is just an example repository, there were a couple of issues with this approach:

  1. It really only demonstrated building Docker containers, and not accessing them as part of a larger, more comprehensive test suite.
  2. By including the test suite in the container, the tests could only run in serial, and thus not living up to our FAQ: “Solano CI automatically and intelligently parallelizes automated software tests to deliver results 10 to 80 times faster than existing solutions.”
  3. Considering continuous integration and testing pay the bills here, there was the chance that co-workers would launch a barrage of flying rockets at me due to the tiny test suite. (I’d like to thank a Pivotal conference booth for introducing the rockets to us.)

I’ve addressed the first two issues by updating the example repository. It now builds the Docker container during the pre_setup hook, allowing the subsequent setup hooks and each of the parallel test workers to access the container. To take advantage of this, a more comprehensive test suite now runs PHPUnit tests in parallel.

Since there is only so much testing you can do against such simple functionality, I may need to invest in a foam shield of some sort to address the last issue.

Leave a comment

Sauce Labs and Solano CI: Browser and Mobile Testing Made Simple and Secure

Ensuring that your app works across all the devices, operating systems, and browsers that your users can access it is a daunting task. Market share figures from Net Applications will tell you that Internet Explorer still reigns supreme, while StatCounter will tell you that Chrome won the browser war globally a long time ago. The sure shot way to ensure that your end-user experience is sound is by testing across the major platforms used by your users to access your app. Fortunately, our friends at Sauce Labs offers an industry-leading platform in solving this problem, and automating the Sauce Labs workflow in Solano CI is easy and secure.

saucelabs_automated_test_configuratiorSauce Labs – What do they do?
Sauce Labs lets users run Selenium, Appium and JavaScript unit tests across 650+ browser and OS platforms at scale without setting up or maintaining dedicated testing infrastructure. Many of our customers run an initial round of Selenium tests with a couple of browsers on Solano CI, but Sauce Labs makes available all the different versions of browsers and OS platforms that you’ll ever need. In addition to mobile emulators and simulators, they’ve recently added real mobile devices to their offering.

Test Securely via Sauce Connect

Sauce Connect is a secure tunneling app which allows you to execute tests securely when testing behind firewalls via a secure connection between Sauce Labs’ client cloud and Solano CI. Sauce Connect allows you to test staged apps behind a firewall while maintaining control of proxy and policies. To use Sauce Connect in Solano CI, there are only two things you need to do:

  1. Set the SAUCE_USERNAME and SAUCE_ACCESS_KEY environment variables for your builds. The solano config:add command should be used to set sensitive environment variables.
  2. Ensure you have the latest version of Sauce Connect installed and start it as a background process for your workers. This can be done by merging the following setup hooks in your solano.yml configuration file. Our karma-sauce-example repository is a working demonstration of using Sauce Connect in this manner.

One Step Closer to Continuous Delivery

Now that you have successfully setup your continuous integration workflow to test your changes across all the things, the next step toward a continuous delivery workflow would be to automate your deployment. Solano CI offers a multitude of options for deployment, from running your deploy script to deep integrations with cloud infrastructure partners like Amazon Web Services, Heroku, and Engine Yard. Check out our docs on Deployment and Post-Build for more info. As always, ping us at support@solanolabs.com if you have any questions or comments!

Leave a comment