I was on a board call for a company today where we talked about “acceptable downtime” for their web-based service. This company has commercial customers that depend on its software to run their businesses and the software in question is delivered “as a service.” I’ve got a number of companies using this approach (vs. a straight software license approach) and I have a lot of experience with this issue dating back to investments in the 1990’s in Critical Path, Raindance, and Service Metrics.
While it’s easy to talk about “5 nines” (99.999% uptime), there are plenty of people who think this metric doesn’t make sense, especially when you are building an emerging company and have a difficult time predicting user adoption (if you are growing > 20% per month you understand what I mean.) While most companies have SLA’s, these also don’t really take into consideration the actual dynamics associated with uptime for a mission critical app.
For example, when I was on the board of Raindance, the CEO (Paul Berberian) described Raindance’s system as similar to an incredibly finely tuned and awesomely powerful fighter plane (on that even Jack Bauer would be proud to fly in.) There was no question that it was by far the best service delivery platform for reservation less audio conferencing in the late 1990’s. However, in Paul’s words, “when the plane has problems, it simply explodes in the air.” Basically, there was no possible way to create a situation where you can guarantee that there will not be a catastrophic system failure and – in this situation – while there was plenty of fall over capacity, it’s too expensive to create 100% redundancy so it will take some time (usually in Raindance’s case an hour or two, although it once took two days) to get the system back up. The capex investment in Raindance’s core infrastructure (at the time) was around $40m – we simply couldn’t afford another $40m for a fully redundant system, even if we could configure something so it was – in fact – fully redundant.
There have been many high profile services that have had catastrophic multi-day outages. eBay had a number of multi-day outages in 1999; Critical Path had a two day outage in 1999 (I remember it not so fondly because I was without email for two days in the middle of an IPO I was involved in); Amazon had some issues as recently as last years holiday season; my website was down for an hour last night because of a firewall configuration issue; the list goes on.
Interestingly, as an online service (consumer or enterprise) becomes more popular, the importance of it being up and operational all the time increases. While this is a logical idea, it’s a feedback loop that creates some pain at some point for a young, but rapidly growing company. As the importance of the service increases, expectations increase and – when there is the inevitable failure (whether it’s for a minute, an hour, or a day) – more people notice (since you have more users).
After watching this play out many times, I think every company gets a couple of free passes. However, once you’ve used them up, the tides turn and users become much more impatient with you, even if your overall performance increases. Ironically, I can’t seem to find any correlation concerning price – the behavior that I’ve witnessed seems to be comparable between free services, services that you make money from (e.g. eBay), and services that you pay for.
My advice on the board call today was that – based on our rate of growth (rapid but not completely out of control yet) – we should get ahead of the issue and invest in a much more redundant infrastructure today. We haven’t yet used up our first free pass (we’ve had several small downtime incidents, but nothing that wasn’t quickly recovered from), but we had a scare recently (fortunately it was in the middle of the night on a weekend and – given that we are a commercial service – didn’t affect many users). The debate that ensued balanced cost and redundancy (do we spent $10k, $100k, $500k, or $1m) and we concluded that spending roughly up to 50% of our current capex cost was a reasonable ceiling that should give us plenty of redundant capacity in case of a major outage. Of course, the network architecture and fall over plan is probably as important (or more important) then the equipment.
I’m searching for a way to describe “acceptable downtime” for an early stage company on a steep adoption curve. I’m still looking (and I’m sure I’ll feel pain during my search – both as a user and an investor), but there must be a better way than simply saying “5 nines.”
Hysterical – John Travolta looks damn good as a bunny. Uma Thurman – not so good. Hat tip to David Beisel.
Killing Rain by Barry Eisler was dynamite. I continue to make one step forward and two steps backward through my shelves of unread books (Amazon delivers more than I consume.) After every few books I resort to pure mental floss – Eisler’s series on assassin John Rain is spectacular summer reading.
When Killing Rain showed up via Dan the UPS Man, Amy grabbed it and wouldn’t give it up until she finished it. It’s the fourth book in the series – if you haven’t read the others, you must start with Rain Fall, the first in the series. The flyleaf review from Entertainment Weekly – usually shoddy stuff – is very accurate this time:
“If Quentin Tarantino ever got to take a crack at the James Bond franchise, chances are the resulting film would resemble one of Eisler’s novels about John Rain. [Rain] is the stuff great characters are made of.”
Great stuff. Dad / Ed – don’t both buying copies – they are on their way to you.
Spam is one of the flagship members of the Internet Axis of Evil – it sucks worse than War of the Worlds. I’ve been supporting the war against spam through my investments in Postini and Return Path and I plan to continue to do whatever I can to help eradicate this scourge from Planet Internet.
While it would be nice if spam just disappeared, it’s not going to anytime soon. So – in addition to attacking spam, it’s time to really address the “legitimate email issue.” Return Path has been after this for a while and just released a new version of their email delivery monitoring tools. Today – TRUSTe (a non-profit dedicated to online privacy issues) launched an “Email Privacy Seal Program” – members of this program can confidently say “We Don’t Spam.”
As I’ve gotten deeper in to the spam issue, it’s clear that it’s needs to be addressed from both sides. The obvious side – prevent the bad shit – is what the anti-spam companies like Postini do. The less obvious side – allow the good stuff through (which also includes “tell consumers who is good and who is bad”) is becoming more central to the war. Return Path and TRUSTe are doing good things around this.
About every three months I hear someone say “RSS will eliminate email (implying that the spam issue will go away).” This is a ridiculous construct – Matt Blumberg the CEO of Return Path has several good posts on this so I’ll refer to him rather than repeat what he has to say. But – Matt gets it – he knows email, knows RSS, knows online marketing, and – well – generally has a clue. If you care about this issue, you should pay attention to what he’s up to and what he’s thinking.
Of course – spam has moved well beyond email at this point. My email spam issue completely under control because of the magic of Postini. However, I get between 50 and 100 comment and trackback spams on a good day (and several hundred on a bad day). My tools for this suck – SixApart is promising new happiness in MT 3.2, but for the time being I’m struggling along with the MT-Blacklist plug-in. Michael Parekh tells an entertaining story of the ineptness of my blog when he tried to comment.
While the world would be better if jerks didn’t feel compelled to write software that posted crap like:
URL: https://ceramics-tiles.blogspot.com
Title: מטבחים
Weblog: מטבחים
Excerpt:
Tell us what you like about the מטבחים events and what you think would make them even better. You are the key to making your <a href=’https://ceramics-tiles.blogspot.com’>מטבחים</a>…
to my blog, having lived through the last 10 years of the email spam wars, I accept that we’ll have continued fun with all sorts of new variants. Ironically, I recognize that this helps power the “technology perpetual machine” – I guess that’s just something we have to live with.
So, earlier tonight, I turned to Amy with a twinkle in my eye and watched as she stared intently at something that looked suspiciously like Typepad on her monitor. The following conversation ensued:
Brad: “Hey, Atom 1.0 is out.”
Amy: “Adam Who?”
Brad: “Atom 1.0 – the new feed format.”
Amy: “I thought that was RSS.”
Brad: “Yeah, yeah, there’s this other format called Atom that … (long unnecessary technical explanation followed).”
Amy: “Uh huhnnnhh”
Brad: “Stop playing with your blog – this is important.”
Amy: “So – what does it mean – what do I have to do differently?”
Brad: “Heheh – nothing – because you use FeedBurner and they automagically will format your feed for Atom 1.0 if it makes sense.”
Amy: “Brad – you are such a nerd.” (As she continued typing on her computer).
I was a little surprised she didn’t say “What was wrong with the last version of Atom?” – this from a woman who regularly says “Damnit – DOS was good enough – why doesn’t Shift-F7 Y exit Word?”
BMC Software – a company that has been around for 25 years and has a major software footprint – announced today that they are using Rally Software’s lifecycle management software and Agile consulting services. This follows Rally’s announcements last week around Release 5 of their Agile software lifecycle management product.
While BMC and Rally just announced this deal, BMC has been using Rally since the end of last year and the rollout and implementation has been superb. The BMC announcement is a big deal – it’s the first of several major software companies to have adopted Rally’s software tools.
About 20 days after the end of each quarter, Ernst & Young/VentureOne releases a survey on venture capital funding. At about the same time, PWC releases their similar survey (MoneyTree). A rash of articles are written over the weekend and usually hit first thing Monday in the business section of newspapers around the country (and – shockingly – in blogs) covering funding nationally, by city, and by market segment.
On Friday, I got a call from Ross Wehner, a Denver Post staff writer who I like. He was calling to get my thoughts on the “numbers for the quarter.” After ascertaining which numbers Ross was talking about (it was a beautiful Friday afternoon in Homer, Alaska – I was not thinking about the E&Y / VentureOne survey), the conversation went something like:
Brad: “Ross – who gives a fuck?”
Ross: “C’mon Brad, you know I can’t print that. What do you think?”
Brad: “The numbers are irrelevant – plus they are probably wrong and misleading.”
Ross: “Well – there were 24 deals in the first half of the year up from 19 in 2004 and the amount invested was $291m up from $192m.”
Brad: “Yeah, but wasn’t Webroot in there?”
Ross: “Yeah …”
Brad: “So – take it out because – while it’s a venture deal, it’s an anomaly and skews the numbers – so 23 deals and $183m. That seems like a statistically unimpressive difference.”
We had a good chuckle (Ross is smart – he gets it) but he still had an article to write. Seriously, does anyone really care about this stuff? Why are E&Y and PWC spending time on this crap instead of doing good audits and getting S-3’s effective for public companies?
I’m going to keep tonight’s term sheet post short and sweet, especially since I’m still reeling from the horrifyingly bad War of the World movie I just saw. Jason and I are almost done with the term sheet series (yeah, we know we keep promising that) but – like the Spielberg tragicomedy I just watched, it’s not over until it’s over (sorry – that cliche just snuck its way in – I was helpless against it – the aliens made me do it.)
Occasionally a term sheet will have – buried near the back – a short clause concerning “founders activities.” It usually looks something like:
“Founders Activities: Each of the Founders shall devote 100% of his professional time to the Company. Any other professional activities will require the approval of the Board of Directors.”
This should be no surprise to a founder that your friendly neighborhood VC wants you to be spending 100% (actually 120%) of your time and attention on your company. If this paragraph sneaks its way into the term sheet, the VC has either been recently burned or is suspicious and / or concerned that one or more of the founders may be working on something besides the company in question.
Of course, this is a classic no-win situation for a founder. If you are actually working on something else at the same time and don’t disclose it, you are violating the terms of the agreement (and – breaching trust before you get started – not a good thing as it’ll eventually come to light.) If you do disclose it – or push back on this clause (hence signaling that you are working on something else), you’ll reinforce the concern that the VC has. So – tread carefully here. Our recommendation – unless of course you are working on something else – is simply to agree to this (why wouldn’t you, unless you don’t believe in the thing you are asking the VC to fund?)
In situations where I’ve worked with a founder that already has other obligations or commitments, I’ve always appreciated him being up front with me early in the process. I’ve usually been able to work these situations out in a way that causes everyone to be happy and – in the cases where I can’t get there – I’m glad that the issue came up early so that I didn’t waste my time or the enterpreneurs’ time.
While there are situations where VCs get comfortable with entrepreneurs working on multiple companies simultaneously (usually with very experienced entrepreneurs or in situations where the VC and the entrepreneur have worked together in the past), they are the exception, not the norm.
I just wasted two hours of my life. At least I got to sit next to Amy and eat popcorn.
As I’ve said in the past, we get two movies a week in Homer. Friday morning comes with great anticipation as we drive past the theater marque (on the corner of Main Street and Pioneer – how fitting) to see what we get to enjoy next. This week is War of the Worlds and Bewitched and I had low hopes for both of them. Since we don’t have a TV here, this is all we are going to get this week (no – I haven’t broken down to watching movies on my laptop yet.)
We gave War of the Worlds a shot tonight. It was tragically awful. Yeah – the special effects were good. But H.G. Wells is screaming his head off wherever he is (or would be if you believe in an afterlife.) The acting was marginal. The plot was completely predictable. The Spielberg cute-heroic-screaming child did exactly what you’d expect. The sullen teenager did exactly what you’d expect. All the humans behaved poorly under pressure and the military was completely ineffective against the aliens. After 115 minutes of stupidity, the aliens and their machines fell over dead, humanity was saved, and the Morgan Freedman / God character got to make a stultifying speech.
And – I’m still not entirely sure what actually happened. Ugh – that was terrible. Sadly, Bewitched may be the better movie of this week.