Category Archives: Miscellaneous

OpenSSL Heartbeat Vulnerability (aka: Heartbleed)

As most of our users are by now no doubt aware, on April 7th a serious vulnerability was announced in recent versions of OpenSSL.  Dubbed Heartbleed, CVE-2014-0160  allows a remote attacker to read potentially sensitive data on the server. This vulnerability has had a widespread impact on many providers.  We take security and the trust […]

Leave a comment

Per-Repo GitHub Status Configuration

The Solano CI integration with GitHub uses OAuth for authentication.  Today we have rolled out the ability to set the credentials used to post GitHub status on a per-repository basis.  To configure an alternate set of credentials for a repository, go to the GitHub Status menu item on the repo configuration page  (click on the […]

Leave a comment

3/14 Service Interruption

Around 7am PT, one of our app server nodes (and alas, also our primary redis server) started exhibiting average network (ping) latency of several tens of ms — spiking to >100ms — to our DB master and other nodes in the cluster. We have removed the app server from use, and failed over to replicas […]

Leave a comment

Recent Service Interruptions

There have been several recent service interruptions that have delivered an experience of using TDDium that’s below our high standards. We here at Solano Labs sincerely apologize for these issues. We’d like to take a few minutes to explain the incidents and describe our short- and long-term mitigation strategies. 2/24 Domains Unresolvable Sometime before […]

Leave a comment

Solano Labs Sponsors New Community:

We are extremely happy to announce the launching of an online community blog based on the interest we have received from our first three Automated Testing Meetup Groups.  The blog is called “Where people who write and test software come to talk about automation”  We are very excited to grow and support this community! […]

Leave a comment

SimpleCov and Ruby 2.1

Solano CI uses the exit status from commands to determine whether a test passes or fails.  The behavior follows in a venerable Unix tradition whereby the exit status of zero indicates success and a non-zero exit status indicates failure. On occasion we’ve seen bugs in test frameworks that can cause false positives, or worse false […]

Leave a comment

Github Authentication Updates Released

We’re happy to announce that the changes we’ve been planning to our GitHub authentication integration are live in our production environment! As we described in an earlier post, we’ve changed our OAuth model to allow users to select the privilege level they give Tddium to communicate with GitHub.  Now, when you link a GitHub account, […]

Leave a comment

Service Outage Oct 24, 2013 – Maintenance Today

At approximately 2:14pm PT on Oct 24, 2013, Tddium’s  DB master server experienced a CPU usage spike that cascaded into to a server stoppage.  No data was lost. Examining data (thanks New Relic!) and logs, our conclusion is that though average usage hovers around 20-30%, our DB master has burst CPU usage close to 100%. […]

Leave a comment

Speeding Up our Test Suite: From 2.5 hours to 20 mins with Solano Labs

by Drew Blas, Software Engineer, At Chargify we rely heavily on automated testing to ensure that we always maintain a working app. With so many customers and a heavily utilized API, it’s critical that we maintain complete backwards compatibility and ensure we don’t impact existing customer operations. That’s why our test suite consists of […]

Leave a comment

Service Outage Remedies and Post Mortem

We’re planning a short downtime tonight, 4/30/2013 at 10pm PT and a longer maintenance window over the weekend (5/4 from 5-8pm PT) to address some of the root causes of our recent downtime. We’ll update this blog post with a much more detailed post-mortem, but the 6 word summary:  an AWS instance went to lunch. […]