Lessons learned moving jobs from Jenkins to Solano CI

I guess the first day of a new job was as good as any other to hear the words “we need you to port over a bunch of Jenkins jobs to Solano CI”. Considering I had never used Solano CI before, it seemed like a great learning experience. What I did not expect was that most of the lessons learned to be more applicable to Jenkins than Solano.

What are all of these Jenkins jobs actually doing?

Solano CI jobs are based on individual git repositories and use a single configuration file stored within the repository. Seems very convenient. A commit-appropriate job configuration is always available in the repo history. JobConfigHistory provides a similar, if not as precise, capability for Jenkins, and should definitely make anyone’s short list of must-have Jenkins plugins.

I initially considered the repository-focused nature of Solano jobs to be a potential shortcoming, since there can be any arbitrary number of Jenkins jobs, doing pretty much any arbitrary thing, for each repository and its branches. After mapping out each of the assigned Jenkins jobs that used each specific repository and branch, I came to appreciate Solano’s one repository/branch, one job nature, particularly since each repo branch can have a different configuration, if necessary. “If necessary” was being overly misused on Jenkins, with multiple, initially identical-except-for-repo-branch jobs that were not always kept up to date. Using parameterized build options with the branch as a parameter can help eliminate this job divergence on Jenkins.

Per repository, the Jenkins jobs generally broke down into: 1) preparation – fetch and/or build artifacts, 2) testing – unit, integration, code coverage, etc., and 3) deploy, which could neatly be slotted into Solano’s 1) pre_setup and/or worker_setup hooks, 2) tests, and 3) post_worker and/or post_build hooks respectively. For the few instances where the testing jobs had to be run sequentially, Solano provides that capability with its “serialize” configuration and build profiles and plans. In an ideal world, these inter-test dependencies would have been eliminated, so one Jenkins job failing would not effect other downstream jobs (and Solano’s parallelized testing structure could be used to full advantage).

Speaking of downstream jobs, I wish Jenkins job creators/maintainers were consistent. Sometimes jobs are grouped by name prefix, sometimes a “group” could be defined by build triggers, and sometimes its one of several plugins that provide the “grouping”. Two weeks after I thought I was done with all of the Jenkins jobs regarding a specific git repo, someone pointed out that some other, long forgotten, consistently failing Jenkins job used the same repo. I didn’t really need anymore convincing at that point, but having a repository-specific configuration listing all of the associated tasks makes a lot of sense. That, and the next time I start a project like this, the first thing I’m going to do is download every Jenkins job config.xml file, and parse out each repository and branch.

Getting to know your Jenkins nodes

I’m sure most Jenkins nodes (no, I will not call them “slaves”) started out as consistent, relatively blank slates, likely built from a disk image. Then one Jenkins job needed a specific service, application, library, etc., which was promptly installed on the node. The node name, description, and/or label were changed appropriately. Then another dependency was required, and another. Names, descriptions, and labels were hopefully changed appropriately. After a long time, maybe some nodes became unique snowflakes with only a few compatible Jenkins jobs. It happens. To some extent, it is just a resource distribution problem at that point. (If you have a “java-6” label that refers to nodes with descriptions claiming only java-7-openjdk is installed, you also have a confusion problem…or at least I had that problem.)

A bigger problem is when all those dependencies are not documented. And when a new identical node is required. And particularly when that node’s capabilities need to be installed on a fresh image.

On the other hand, trying to run the steps in a Jenkins job on a fresh image (or in my case a Solano CI image which is freshly created for each job) is a great way to discover any undocumented dependencies. The cycle of: 1) find a dependency error, 2) ensure the job mitigates the problem, 3) goto 1 can get a little frustrating, but in the end you have a job that is truly portable and not node-dependent.

I think my favorite dependency error was a git submodule pointing to the version-2 tag of a repository, but version-2 contained only a readme.md file. The Jenkins job was working only because the node’s workspace still contained all of the stale version-1 files. I found another Jenkins unique snowflake node!

Your test results…faster

From a development perspective, a primary benefit of continuous integration is letting developers know as soon as possible if their code changes break something. This is where Solano CI truly shines as it will automatically spread tests across parallel workers. Having the testing results an order of magnitude faster helps, as recent code changes are still fresh in a developer’s mind.

While Jenkins’ one-job, one-worker workflow can not match Solano’s speed, an incremental improvement might help. Instead of using build triggers (or a similar mechanism) to run the jobs sequentially, run them concurrently on different workers. While this is more of a “throw hardware at the problem” solution, getting the results back in two hours (the longest running job) instead of eight (all the jobs chained together) can be the difference between a developer thinking “I was just working on that and know how to fix it” and “Oh…that one thing from yesterday…”

Considering my “satisfaction” with Jenkins’ job configuration, I’m reluctant to even write the following…but if your Jenkins administrator is a masochist, or you just don’t like him/her/them: you can fake the parallelization that Solano CI provides (to some extent) by splitting Jenkins jobs as much as possible, in order for them to run as smaller individual jobs on additional parallel workers.

If you considered that last paragraph as a viable solution for more than a couple seconds, help yourself and your organization: take a look at Solano CI and try the auto-parallelized builds.

Post a Comment