Author Archives: jaymoorthi

3/14 Service Interruption

Around 7am PT, one of our app server nodes (and alas, also our primary redis server) started exhibiting average network (ping) latency of several tens of ms — spiking to >100ms — to our DB master and other nodes in the cluster. We have removed the app server from use, and failed over to replicas […]

Leave a comment

Recent Service Interruptions

There have been several recent service interruptions that have delivered an experience of using TDDium that’s below our high standards. We here at Solano Labs sincerely apologize for these issues. We’d like to take a few minutes to explain the incidents and describe our short- and long-term mitigation strategies. 2/24 Domains Unresolvable Sometime before […]

Leave a comment

Github Authentication Updates Released

We’re happy to announce that the changes we’ve been planning to our GitHub authentication integration are live in our production environment! As we described in an earlier post, we’ve changed our OAuth model to allow users to select the privilege level they give Tddium to communicate with GitHub.  Now, when you link a GitHub account, […]

Leave a comment

GitHub API Authentication Updates

At Solano Labs, we believe that a seamless integration between our service and our customers’ tools provides the best user experience. Many of our customers today use GitHub and have connected a GitHub account with their Tddium account using OAuth. We take the security of our customers’ code very seriously, and we’re making some important […]

1 Comment

Service Outage Oct 24, 2013 – Maintenance Today

At approximately 2:14pm PT on Oct 24, 2013, Tddium’s  DB master server experienced a CPU usage spike that cascaded into to a server stoppage.  No data was lost. Examining data (thanks New Relic!) and logs, our conclusion is that though average usage hovers around 20-30%, our DB master has burst CPU usage close to 100%. […]

Leave a comment

New CLI Feature: ‘tddium rerun’

Now you can re-run a focused set of tests based on what failed in a Tddium Session! We’ve released a new version of our CLI, tddium-1.13.0, that introduces two commands to make our CLI workflow smoother:

The ‘rerun’ command will start a new session on Tddium’s cloud with only the failed tests from the original […]

Leave a comment

New Feature: Organizations

We’ve just released a major update to user and billing management, along with streamlining changes to the main “dashboard” view. Our concept of “organizations” should be familiar to users of other popular products, and its introduction should be seamless for all of our existing users. Instead of belonging to a single “account”, a user can […]

Leave a comment

Service Outage Remedies and Post Mortem

We’re planning a short downtime tonight, 4/30/2013 at 10pm PT and a longer maintenance window over the weekend (5/4 from 5-8pm PT) to address some of the root causes of our recent downtime. We’ll update this blog post with a much more detailed post-mortem, but the 6 word summary:  an AWS instance went to lunch. […]


Tddium Service Outage: 12/19/2012

On Wednesday 2012 Dec 19, the main Tddium web service experienced an outage for about 4 hours when our primary database crashed.   We had been preparing a warm-standby DB replica for production deployment in January. We were able to use it to recover completely.  We are now live with a high-availability architecture in our primary […]

Leave a comment

Tddium And New Relic

Today, we’re happy to announce a few new developments with our partners at New Relic! First, we’re proud to say that Tddium customers can get New Relic Standard for free! Simply click on this link or sign up on Then, once you’ve got your New Relic account setup, you can link your deploys straight […]