Tuesday, April 24, 2007

The secret to making things easy: avoid hard problems

That may seem obvious, but in my experience most engineers prefer to focus on the hard problems. Working on hard problems is impressive to other engineers, but it's not a great way to build successful products. In fact, this is one of several reasons why YouTube beat Google Video: Google spent a lot of time solving technically challenging problems, while YouTube built a product that people actually used (using PHP and MySQL, I think, which is not at all technically impressive).

For me, the most effective method of getting things done quickly is to cheat (technically), take a lot of shortcuts, and find an easier way around the problem (and before anyone jumps in with some comment about security or bank transactions, there are obviously a few exceptions). You only need to think ahead enough to avoid painting yourself into a corner, or have a plausible plan for escaping the corner. There's always an easier way -- work lazier, not harder. Note that this doesn't preclude doing things that SEEM difficult -- easy solutions to important problems that LOOK really hard are the best.

I was reminded of this while replying to the comments on news.yc in response to my post on disks and databases. Whenever anyone mentions the possibility of not using a conventional database, a lot of people will immediately reply that databases solve a lot of very difficult problems, and that you shouldn't put a lot of work into reinventing the wheel. These people are, of course, correct.

The thing is, a lot of those difficult problems are irrelevant for 99% of products. For example, "real" databases can handle transactions that are too large to fit in memory. That was probably a really important feature in 1980. Today, you can buy a computer with 32GB of memory for around $5000. How many GB transactions do you suppose Twitter performs? My guess is zero -- I suspect that their average transaction size is closer to 0.0000002 GB (messages are limited to 140 characters).

I want to be perfectly clear about one thing though: I'm not advising you to ditch your database! If your database is plenty fast, then the easiest thing to do is probably "nothing", and that's what I advise you do. If, however, your db is getting slow or overloaded, then you need to do two things:
  1. Understand the problem
  2. Fix the problem
The correct solution to your problem will depend on your situation. For example, if you have some data that's very important but doesn't change very often (username and password), and some data that gets updated continually but doesn't have to be correct (last active time or hit counters), then a simple solution would be to leave the important data in your database and move the less important data into something really simple but less reliable.

Want an example of "simple but less reliable"? Here's one (in one or two easy steps):
  1. All updates go in to memcached, but not the database
  2. (optional) A background process occasionally copies entries from memcached to the db. Without this, the values will be completely lost when memcached restarts.


John said...
This comment has been removed by the author.
keizo said...

"in my experience most engineers prefer to focus on the hard problems."

I love that. I am taking some senior design courses for my aero/mechanical engineering degree and this has happened constantly. Smart people love to design complicated things, but often all that's needed is something simple.

Thanks for writing, I have been enjoying everything so far.

Shanti Braford said...

Great post, Paul.

Just a nit: I believe YouTube was written in Python. (I remember reading a post by Guido a while back saying that...)

ps. would be interested to hear your thoughts sometime on Mailroom (sproutit.com) if you've ever taken a look. I did most of the backend development for it. =)

Paul Buchheit said...


They probably used a variety of languages, especially now that they've grown so big. Their job posting for "Senior web developer" asks for PHP experience though.

As for Mailroom, it looks pretty nice. It might be helpful to have a video or something to make it easier to see exactly what it does though.

Revin F Floyd said...

Dude, I couldn't disagree with you less! However right you are.

I'm that guy! I'm the guy that sees what seem to be the most difficult problems and fool-heatedly, take them on with daring spirit and little or no since of self-preservation or office politics (like the guy who shoots everything down with that security crap before you've even presented a solution.

I've achieved much fan-fare, earning many a propeller beanie solving important problems with the simplest solutions. Those are the best!

I try not to avoid any problem, especially the "hard" ones. In fact, in my experience, the hardest problems have a single glaringly obvious solution: ditch the process that creates it.

Unnecessary data-entry somewhere in the system is a typical culprit in this regard. Someone is entering data or extracting, editing and re-loading data in some manual process because they need to work with it in some other format, and it grows into a "HARD" problem to fix because over time the chain of inefficiencies has grown uncomprehensibly complex. When anyone sees you're going to eliminate that problem, they see it as an attack on them, because the work is unnecessary, then they may also be unneccessary and eliminated along with the problem...

I could be wrong... I've often made the mistake of taking on hard problems and somehow became responsible for the problem's existence in the first place. Once you commit to a hard problem, you become responsible for it. That sucks!

Sad but true.