The boring work of keeping software running

The boring work of keeping software running

Flash-back to 2008.

Most of the Moraware team was hanging out in Florida at the beach in the summer of 2008. Actually, we were having a company meeting in the ocean, with the occasional boogie board and skim-board thrown in, too.

The topic at hand: deciding whether to switch our business model from traditional installed software to software as a service (usually called SaaS).

Ted & I had talked about it for years, but neither of us had really been convinced that it was the right thing. There were advantages – we’re actually good at keeping computers running, doing updates would be simpler, our customer’s data would be safer, we’d get recurring revenue, and we thought it’d make the software buying process easier.

But, it was a huge responsibility. If we decided to do SaaS, Moraware would be on the hook to make sure everything was running smoothly 24/7. And, as the owners of Moraware, that meant we’d ultimately be on call every day, night, and weekend. So, we kept putting off the decision.

But then, the housing crash happened…

Suddenly, our new prospects were begging us for low monthly payments. Even though we had originally provided some payment plans for our countertop fabricator software, payments were over the course of months. The longest we were willing to do was a year. While the housing market was crashing, folks asked for 2-year, 3-year, and even longer payment terms.

It seemed like the perfect opportunity to dip our toes into the water of SaaS.

So, in October of 2008, we started down the path. Within a few months, the majority of our new customers were signing up for the SaaS version of our software. And, within about another year, we had fully moved to that model.

Life was looking good – revenues from the SaaS version were modest, but they were on the right path. And, because we actually made a business decision to run servers, we could invest more in getting good hardware in secure data-center with multiple top-tier internet connections and a hardware-monitoring and replacement service-level agreement.

Oh Shit!

January 11, 2010. On a typical day in 2010, we’d get around 50 support calls and emails. That day, we got a couple hundred… over the course of 52 minutes.

We had a hardware failure at the data center, and “luckily” it happened when all of us were around and awake. We were able to diagnose and replace the bad gateway quickly, and had it all rebooted and running within the hour. That piece of hardware was one of the few things in the data center that didn’t have a backup, and once we recovered from the trauma, we fixed our architecture to make sure that exact problem would never happen again.

But it was our nightmare scenario come to life – we were responsible for hundreds of countertop businesses’ critical business data – and we had screwed it up. Even though our downtime wasn’t dramatic – 99.99% uptime over years, it still felt awful.

What are they building in there?

Apologies to Tom Waits fans, but we spend a lot of time building things that you’ll probably never notice, because they’re ensuring that the software you use every day is reliable, fast, and (pretty much) bug-free.

Depending on the projects that we’ve going on, usually between a quarter and a half of our development effort goes into just keeping things running.

For example, a few years ago, we started monitoring for bugs in the software much more proactively. Before that, we’d always worked on fixing bugs when customers called about them… but when we first turned on the monitoring, we discovered there were hundreds of bugs that nobody had ever called about. Even now, one user somewhere will run into a weird case that nobody has tried before… and find a bug that’s been lurking in the software for years.

The other thing we’ve spent a lot of time on lately is upgrading our server infrastructure. Just recently, we upgraded our architecture to run live in multiple data centers. We had always backed things up in multiple places, but we were still losing sleep imagining disaster scenarios (What if a nuclear bomb hits Michigan? We’ll have over a day of downtime!).

Even though impossible to get to 100% guaranteed up-time, we keep striving to get closer to that goal. And, with this newest change, it really feels like we made another huge leap in that direction.

The boring stuff

I’m going to skip the technical details of what we’ve done lately. By the time you read this, there will already be new updates and changes that I don’t know about.

But, we frequently get the question “What’s new?” from our existing customers. If it’s not directly helping the pain you’re feeling today, the details of our answer probably don’t matter much to you. But the real answer is “The most valuable thing“.

Want to know more? At Moraware, we make software for countertop fabricators. CounterGo is countertop drawing, layout, and estimating software. JobTracker is scheduling software that helps you eliminate the time you waste looking for job folders. RemnantSwap is a free place to buy and sell granite remnants with fabricators near you.

Leave a Reply

Your email address will not be published. Required fields are marked *