Category: Technology
As a user, how often have you thought “I wish this web service was faster.” As a CEO, how often have you said “just make it faster.” Or, more simply, “why is this damn thing so slow?”
This is a not a new question. I’ve been thinking about this since I first started writing code (APL) when I was 12 (ahem – 33 years ago) on a computer in the basement of a Frito-Lay data center in Dallas.
This morning, as part of my daily information routine, I came across a brilliant article by Carlos Bueno, an engineer at Facebook, titled “The Full Stack, Part 1.” In it, he starts by defining a “full-stack programmer”:
“A “full-stack programmer” is a generalist, someone who can create a non-trivial application by themselves. People who develop broad skills also tend to develop a good mental model of how different layers of a system behave. This turns out to be especially valuable for performance & optimization work.”
He then dissects a simple SQL query (DELETE FROM some_table WHERE id = 1234;) and gives several quick reasons why performance could vary widely when this query is executed.
It reminded me of a client situation from my first company, Feld Technologies. We were working on a logistics project with a management consulting firm for one of the largest retail companies in the world. The folks from the management consulting firm did all the design and analysis; we wrote the code to work with the massive databases that supported this. This was in the early 1990’s and we were working with Oracle on the PC (not a pretty thing, but required by this project for some reason.) The database was coming from a mainframe and by PC-standards was enormous (although it would probably be considered tiny today.)
At this point Feld Technologies was about ten people and, while I still wrote some code, I wasn’t doing anything on this particular project other than helping at the management consulting level (e.g. I’d dress up in a suit and go with the management consultants to the client and participate in meetings.) One of our software engineers wrote all the code. He did a nice job of synthesizing the requirements, wrestling Oracle for the PC to the ground (on a Novell network), and getting all the PL/SQL stuff working.
We had one big problem. It took 24 hours to run a single analysis. Now, there was no real time requirement for this project – we might have gotten away with it if it took eight hours as we could just run them over night. But it didn’t work for the management consultants or the client to hear “ok – we just pressed go – call us at this time tomorrow and we’ll tell you what happened.” This was especially painful once we gave the system to the end client whose internal analyst would run the system, wait 24 hours, tell us the analysis didn’t look right, and bitch loudly to his boss who was a senior VP at the retailer and paid our bills.
I recall having a very stressful month. After a week of this (where we probably got two analyses done because of the time it took to iterate on the changes requested by the client for the app) I decided to spend some time with our engineer who was working on it. I didn’t know anything about Oracle as I’d never done anything with it as a developer, but I understood relational databases extremely well from my previous work with Btrieve and Dataflex. And, looking back, I met the definition of a full-stack programmer all the way down to the hardware level (at the time I was the guy in our company that fixed the file servers when they crashed with our friendly neighborhood parity error or Netware device driver fail to load errors.)
Over the course of a few days, we managed to cut the run time down to under ten minutes. My partner Dave Jilk, also a full-stack programmer (and a much better one than me), helped immensely as he completely grokked relational database theory. When all was said and done, a faster hard drive, more memory, a few indexes that were missing, restructuring of several of the SELECT statements buried deep in the application, and a minor restructure of the database was all that was required to boost the performance by 100x.
When I reflect on all of this, I realize how important it is to have a few full-stack programmers on the team. Sometimes it’s the CTO, sometimes it the VP of Engineering, sometimes it’s just someone in the guts of the engineering organization. When I think of the companies I’ve worked with recently that are dealing with massive scale and have to be obsessed with performance, such as Zynga, Gist, Cloud Engines, and SendGrid I can identify the person early in the life of the company that played the key role. And, when I think of companies that did magic stuff like Postini and FeedBurner at massive scale, I know exactly who that full system programmer was.
If you are a CEO of a startup, do you know who the full-stack programmer on your team is?
Fred Wilson emailed me a link to Dennis Crowley’s post I’m running the NYC Marathon tomorrow! Fred knows my obsession with human instrumentation, marathons, and social media. And if you recognize Dennis’ name, that’s because he’s the founder / CEO of Foursquare.
As I write this from my house in Eldorado Springs, Colorado, I can see that Dennis is at mile 4.64 of the NYC Marathon via RunKeeper. He just checked in at mile 5 on Foursquare. And yes, Twitter and Facebook are active also.
While some people may not like this future, I love it. Yeah, it’s kind of a pain to carry a bulky iPhone around on a marathon, but there are armbands for that and – in a decade – it’ll just be a thing you inject into your arm under the skin. But for now, guys like Dennis are helping us create the future.
Oh – and he’s running a marathon. He’s now at mile 5.64. Way to go Dennis!
I’ve written in the past about my obsession with measuring things. While my manual measurements via Daytum include miles run, books read, flights taken, and cities slept in, I’ve become much more focused in the past year on what I’ve been calling “human instrumentation.” This resulted recently in Foundry Group leading a $9 million financing in a San Francisco company called Fitbit.
If you want to see the type of data I’m tracking, take a look at my Fitbit profile. For now, I’m focused on the data that Fitbit tracks automatically for me, primarily derived from the step and sleep data. But from my profile page you can see a variety of other data which I can currently enter manually (I’ve entered a few examples) even though I use other sources to track them (for example, my weight using my Withings scale.)
I now have a house full of personal measurement devices and an iPhone full of apps to track various things. A few are still active; many have long been relegated to the “closet of dead, useless, obsolete, or uninteresting technology.” During this journey over the past year, I feel like I tried everything and finally found a company – in Fitbit – that has a team and product vision that lines up with my own.
A year ago when I first encountered the company, they were just launching their product. I was an early user and liked it a lot, but hadn’t clearly formed my perspective on what the right combination of software and hardware was. As I played around with more and more products, I started to realize that the Fitbit product vision as I understood it was right where I thought things were going. The combination of hardware, software, and web data integration are the key, and the Fitbit founders (James Park and Eric Freidman) totally have this nailed. That made it easy when we explored investing again to pull the trigger quickly.
One of the things my partners and I love about products like the Fitbit are the combination of hardware, software, and a web service that lets the product continually improve without having to upgrade the hardware. Fitbit is a great example of this which I expect you’ll see over the next quarter if you buy one today.
I firmly believe that in 20 years we’ll simply swallow something that will fully instrument us. Until then, we still have to clip a small plastic thing to our belt or keep it in our pocket. But that’s ok since it now knows how to talk to my computer, which is connected to the web, which is getting smarter every millisecond.
About a month ago I wrote a post titled Trying Gmail For A Week. I haven’t thought about Outlook, Entourage, or Mac Mail for a month and I don’t think I’m ever going back. It took about a week to rewire my brain for how conversations worked and what the keyboard shortcuts were, but not that I’m there it’s just awesome.
A few weeks ago Fred Wilson wrote a post titled Inbox Zero. In it he mentioned two Gmail services he found indispensable – Priority Inbox (from Google) and Unsubscribe.com (from James Siminoff who created Phonetag, another great service.) I agree with Fred on both of these, but have discovered a few extra things that are killer. I’ll list them below and for balance talk about a few shortcomings.
Priority Inbox: I’ve seen numerous tweets and blogs about how Priority Inbox doesn’t really do much. These are wrong / misinformed reactions. The trick to Priority Inbox, like many other things, is to actually use it for a few weeks. Part of using it is training it by quickly marking things up to “important” (by clicking +) or down to “everything else” (by clicking -). A small configuration change can make Starred emails (for quick follow up) a different category. I found that it only took about three days of this before I saw benefit and now (a month later) Priority Inbox gets it right 99 out of 100 times. I get over 500 emails a day – there is a long list of them that fall in “Everything Else”. I used to have to check / clear email obsessively throughout the day to stay at Inbox Zero. With Priority Inbox I’m finding solid email stretches a couple of times during the day are more than enough for me to stay on top of everything.
Unsubscribe.com: Like many people, I’m stuck in the endless “unsubscribe from email lists” infinite loop. I get vigilant for a few days and do the annoying unsubscribe drill one by one and knock a few off the list, but within a few weeks I’ve got even more. I’ve never seemed to be able to eliminate all the stuff I don’t want, especially around an election when it all escalates like crazy. With Unsubscribe.com, I simply click the Unsubscribe button in Gmail and the service gets rid of it. Don’t bother with the trial – trust me and just pay $19 for the service for a year if endless mailing list email that you don’t want is a problem for you.
Google Voice: I’ve had a Google Voice for a long time but I never fully switched over to it. The Google Voice integration with Gmail has tipped me over. I’ve been dreaming about getting rid of my desktop phone for a while – I now find myself almost exclusively doing every call from my computer except when I’m not online (where I have to use my cell phone.) More importantly, video chat and text chat is completely integrated within Gmail so from one screen I have email, my phone (inbound and outbound calling) Skype-equivalent video chat, and text chat. While I still use Skype extensively (I’m bradfeld) I find I’m using it much less as I end up using brad.feld@gmail.com instead.
Gist: I’m an investor in Gist and use it for my unified contact manager. Google Contacts is ok, but has a long way to go. But Gist integration with Gmail at a data level is superb. I’m still using Gmail’s consumer service so the integration is primarily at a data level, but I’m now playing around with a full switch over to Google Apps and the Gist + Google Apps integration (via the Google Apps Marketplace) just rocks. In addition, there’s a new browser-based Gist add-on coming out shortly (hint hint) that will provide direct integration into the consumer version of Gmail.
GooTasks: Since I am an Inbox Zero guy, I don’t keep anything (including paper), but I do have a short task lists of things like blog posts I’m going to write. I went through an Evernote phase recently but it’s overkill for me. Google Tasks is perfect, but I didn’t have an obvious way to sync with my iPhone. Now I do.
There are a handful of annoying things. The biggest one is that I have multiple accounts on Google (brad.feld@gmail.com as well as brad@feld.com) and they aren’t tightly integrated across all services. The other is the weak / inconsistent iPhone integration which keeps pushing me toward using an Android phone full time (I’m now carrying both an Android phone and an iPhone.) My dad’s recent story on the Samsung Fascinate has me seriously considering a full time switch over to Android.
My “while I’m working” migration from a full Windows / Outlook / Exchange / Office world to an almost completely non-Microsoft world has been fascinating. I’m in Seattle next week including a 24 hour stretch at Microsoft for some stuff – maybe it’ll come up and be an interesting discussion that my friends at Microsoft can learn from. In the mean time, I think the next big switch will be an organization one completely over to a Google Apps infrastructure.
We’ve shifted into a new zone in the world of software patent stupidity. A few weeks ago, Oracle sued Google over a series of Java-related patents they got when they acquired Sun. Last week, Paul Allen sued 11 major software companies, including Google, over four patents that were granted to his now defunct Interval Research think tank.
Much of the early commentary has already been said. And, from what I’ve read, it’s not very generous to either Oracle or Paul Allen. One of the best lines is from James Gosling, the authors of RE38.104 (Method and apparatus for resolving data references in generated code) in his post The shit finally hits the fan. There has been plenty of speculation about the motivation of the Oracle patents and the speculation as to Paul Allen’s motivation is just beginning. Regardless, there are lots of lawyers in the mix advising their clients and devising strategies around these patents.
My own opinion will be no surprise to regular readers of this blog. I think this behavior is an absurd abuse of the patent system. I think it’s a massive tax on innovation. I think it’s an insult to anyone who is a real innovator.
As I was reading through some of the Paul Allen commentary this morning, it occurred to me that this might finally be a tipping point. Last week, Microsoft asked the supreme court to hear their appeal of the I4i patent suit. I hope Google steps up and really takes a stand here given that they are on the receiving end of both the Oracle and Allen suits.
Maybe we’ve reached a tipping point. If you want a little more perspective, go read the book review of Lewis Hyde’s Common as Air. Or better yet, read Common as Air.
Every major software or web company I’ve ever been involved in has had a catastrophic outage of some sort. I view it as a rite of passage – when this happens when your company is young and no one notices, it gives you a chance to get better. But eventually you’ll have one when you are big enough for people to notice. How you handle it and what you learn from speaks volumes about your future.
Last week, two companies that we are investors in had shitty experiences. SendGrid‘s was short – it only lasted a few hours – and was quickly diagnosed. BigDoor‘s was longer and took several days to repair and get things back to a stable state. Both companies handled their problems with grace and transparency – announcing that all was back to normal with a blog post describing in detail what happened.
- BigDoor: Recovery Retrospect
- SendGrid: Postmortem: Aug. 25 2010 Delivery Delays
While you never ever want something like this to happen, it’s inevitable. I’m very proud of how both BigDoor and SendGrid handled their respective outages and know that they’ve each learned a lot – both in how to communicate about what happened as well as insuring that this particular type of outage won’t happen again.
In both cases, they ended up with 100% system recovery. In addition, each company took responsibility for the problem and didn’t shift the blame to a particular person. I’m especially impressed how my friends at BigDoor processed this as the root cause of the problem was caused by a new employee. They explain this in detail in their post and end with the following:
“Yes, this employee is still with us, and here’s why: when exceptions like this occur, what’s important is how we react to the crisis, accountability, and how hard we drive to quickly resolve things in the best way possible for our customers. I’m incredibly impressed with how this individual reacted throughout, and my theory is that they’ll become one of our legendary stars in years to come.”
I still remember the first time I was ever involved in a catastrophic data loss. I was 17 and working at Petcom, my first real programming job. It was late on a Friday night and I got a call from a Petcom customer. I was the only person around so I answered the phone. The person was panicked – their hard drive had lost all of its data (it was an Apple III ProFile hard drive – probably 5 MB). The person was the accounting manager and they were trying to run some process but couldn’t get anything to work. I remember discerning that it seemed like the hard drive was fine but she had deleted all of her data. Fortunately, Petcom was obsessive about backups and made all of their clients buy a tape drive – in this case, one from Tallgrass (I vaguely remember that they were in Overland Park, KS – I can’t figure out why I remember that.)
After determining the tape drive software was working and was available, I started walking the person through restoring her data. She was talking out loud as she brought up the tape drive menu and starting clicking on keys before I had a chance to say anything at which point she pressed the key to format the tape that was in the drive. I sat in shock for a second and asked her if she had another backup tape. She told me that she didn’t – this was the only one she ever used. I asked her what it said on the screen. She said something like “formatting tape.” I asked again if there was another backup tape. Nope. I told her that I thought she had just overwritten her only backup. Now, in addition to having deleted all of her data, she had wiped out her backup. We spent a little more time trying to figure this out, at which point she started crying. I doubt she realized she was talking to a 17 year old. She eventually calmed down but neither of knew what to do next. Eventually the call ended and I went into the bathroom and threw up.
I eventually got in touch with the owner of Petcom (Chris) at his house who told me to go home and not to worry about it, they’d figure it out over the weekend. I can’t remember the resolution, but I think Chris had a backup for the client from the previous month so they only lost a month or so worth of data. But that evening made an incredible impression on me. Yes, I finished the evening with at least one illegal drink (since the drinking age at the time in Texas was 18.)
It’s 28 years later and computers still crash, backups are still not 100% failsafe, and the stress of massive system failure still causes people to go in the bathroom and throw up. It’s just part of how this works. So, before you end up in pain, I encourage you to think hard about your existing backup, failover, and disaster recovery approaches. And, when the unexpected, not anticipated, not accounted for thing happens, make sure you communicate continually and clearly what is going on, no matter how painful it might be.
Last week SimpleGeo and their partner Stamen Design jointly released a project they have been working on together called Polymaps. It’s absolutely beautiful and a stunning example of what you can do with the SimpleGeo API. They’ve released the Polymaps source code on GitHub so any developer can quickly see how the API is used, play around with real production code, and modify the base examples for their own use.
When I first started program, it was 1979. I started on an Apple II – I learned BASIC, Pascal, and 6502 Assembler. I studied every page and example in the Apple II Reference Manual (the “Red Book”). Whenever I got source code for any application at a user group meeting, I stared at it, played with it, and tried to understand what it was doing.
When I started programming on an IBM PC in 1983, I did exactly the same thing. I spent a lot of time with Btrieve and there were endless source code examples to build on. I had a few friends that were also using BASIC + the IBM BASIC Compiler + Btrieve so we shared code (by handing each other floppy disks). We built libraries that did specific things and as each of us improved them, we shared them back with each other.
In my first company, we were heavy users of Clarion. While Clarion was compiled, it still came with a solid library of example code, although we quickly built our own libraries that we used throughout the company as we grew. When I started investing in companies that were building Web apps in 1994, it was once again all HTML / source code and examples everywhere. My friends at NetGenesis (mostly Raj Bhargava and Eric Richard) wrote one of the first Web programming books – Build a Web Site: The Programmer’s Guide to Creating, Building and Maintaining a Web Presence – I vaguely remember NetGenesis getting paid something like $25,000 (which was a ton of money to them at the time) to write it.
In the last few months, the phrase “data as a service” has started to be popular. I’m not totally sure I understand what people mean by it and I’ve been involved in several larger discussions about it and even noticed an article today in the New York Times titled “Data on Demand Is an Opportunity.” I’ve invested in several companies that seem to fit within this categorization, including SimpleGeo, Gnip, and BigDoor, but we don’t really think about them as “data as a service” companies (SimpleGeo and Gnip are in our Glue theme; BigDoor is in our Distribution theme).
When I reflect on all of this, it seems painfully obvious to me (and maybe to you also) that the best way to popularize “data as a service” is to start with an API (which creates the revenue model dynamic) and build a bunch of open source examples on top of it. Your goal should be to make it as simple as possible for a developer to immediately start using your API in ways relevant to them. By open sourcing the starting point, you both save an enormous amount of time and give the developers a much more interactive way to learn rather than forcing them to start from scratch and figure out how the API works.
I like how SimpleGeo has done this and realize that this can apply to a bunch of companies we are both investing in and looking at. I’m not sure that it has anything to do with the construct of “data as a service” (which I expect will quickly turn into DaaS) but it does follow from the long legacy of how people have learned from each other around the creation of software, especially around new platforms.
While we are using SFLA (silly four letter acronyms – we’ve got PaaS, and IaaS, along with our old friend SaaS), any ideas what ZaaS is going to stand for?
Ever since I switched to the Mac, I’ve had N (where N is a suitably large number) tell me that I should switch to Gmail from Exchange. I finally decided to try it for a week and see if it works for me. Given my Mac experience – where I had to commit and really use it, I’ve decided to do the same on Gmail.
For now, I’m just going to use Gmail (instead of Google Apps) because I don’t want to go through the hell of switching the feld.com domain since I’ve got a bunch of other people (e.g. my family members) on it in a variety of configurations. That’ll limit me a little as I won’t be able to use the Apps Marketplace, but the benefit is I’ll be able to mess around with a variety of other Gmail stuff.
If you’ve got Gmail addons, hints, tips, and trick, leave them for me here. At the end of next week, I’ll either be switching to Gmail or heading back to Mac Mail against my Exchange server.
Ah – the joy of a meme. Today’s meme is “The Web Is Dead.” Whatever. My favorite article about this in the past 24 hours is The Tragic Death of Practically Everything – this is basically what I would have written if I’d had time today.
This latest round apparently started with the new Wired cover story “The Web is Dead.” Yeah, I read it. My reaction to it was “whatever.” Are books dead? Is email dead? Are memes dead?
Whatever.