Paul Buchheit

I am nothing

2011-08-20T16:17:00.001-07:00

On a scale of one to ten, how good of a cog are you? How well do you function in your assigned role? How much of a man or woman are you? How do you rate yourself as a son or daughter, father or mother, wife or husband, heterosexual or homosexual, liberal or conservative, black or white, winner or loser, shark or sheep, introvert or extrovert, Christian, Muslim, atheist? How smart are you? How rational? How emotional? Do people like you? Are you getting ahead, or falling behind?

How do you know? Are you keeping an eye on the others in your category, comparing to see how you measure up to your peers? Is it more important for a man to be tall, or to have good hair?

This is, of course, the path of insanity, and not the good kind of insanity.

What will you do if you're too tough to be a good woman, too sensitive to be a good man, too selfish to be a good husband, too lazy to be a good employee, too shy to be a good friend, too caring to be rational, too fat to be pretty, too effeminate to be straight, too introverted to be a good leader, too smart to be kind, too young to be taken seriously, too old to make a difference, or too far behind to even get in the race?

These are all false standards and false dichotomies, but they are so common and so ingrained that we sometimes believe in them without even realizing it. And this leads to a mountain of insecurities, because nobody measures up to these crazy standards (and nobody should). But even if we don't believe in these things, it still matters what other people think, right? What will the neighbors think? Or how about our co-workers, or the people at church? And so everyone works to hide their insecurities, and they look around at their peers for comparison, and maybe they feel bad because everyone else seems to have it easy, to have it all figured out. The truth is, nobody can see the truth anymore. They are all working to hide the truth, because the truth is that they are afraid of who or what they really are. So they all put on a show, and they pretend to be a good whatever. Or maybe they rebel, and make a point of being a bad whatever, but then they are still under the control of that false standard, and they are still not being themselves.

That is all so exhausting.

I am nothing. It's simple. If I were smart, I might be afraid of looking stupid. If I were successful, I might be afraid of failure. If I were a man, I might be afraid of being weak. If I were a Christian, I might be afraid of losing faith. If I were an atheist, I might be afraid of believing. If I were rational, I might be afraid of my emotions. If I were introverted, I might be afraid of meeting new people. If I were respectable, I might be afraid of looking foolish. If I were an expert, I might be afraid of being wrong.

But I am nothing, and so I am finally free to be myself.

This isn't license to stagnate. Change is inevitable. Change is part of who we are, but if we aren't changing for the better, then we are just slowly decaying.

By returning to zero expectations, by accepting that I am nothing, it is easier to see the truth. Fear, jealousy, insecurity, unfairness, embarrassment -- these feelings cloud our ability to see what is. The truth is often threatening, and once our defenses are up, it's difficult to be completely honest with anyone, even ourselves. But when I am nothing, when I have no image or identity or ego to protect, I can begin to see and accept things as they really are. That is the beginning of positive change, because we can not change what we do not accept and do not understand. But with understanding, we can finally see the difference between fixing problems, and hiding them, the difference between genuine improvement, and faking it. We discover that many of our weaknesses are actually strengths once we learn how to use them, and that our greatest gifts are often buried beneath our greatest insecurities.

Letting go of your identity can be difficult and takes time, possibly forever, but as with any change, moving in the right direction is all that really matters (which is why you shouldn't compare yourself with others -- you didn't start in the same place or with the same challenges). Fortunately, we have a variety of emotions that can help us: pride, anger, fear, jealousy, insecurity, unfairness, embarrassment, bitterness, etc. These are sometimes portrayed as bad emotions, but there's no such thing as a bad emotion, just bad responses to emotions. (For example, torturing children is a very harmful response to fears about your own sexuality) If we instead use these emotions as a cue to remember, "I am nothing", let go of our identity, and examine why we are feeling the emotion (typically because something has threatened our identity) then these emotions are actually beneficial. They point us towards the buried truth.

True self improvement requires becoming a better version of our selves, not a lesser version of someone else. But without self acceptance and understanding, how can we even know what that looks like or whether we're headed in the right direction? It would be like putting the final touches on the Mona Lisa while picturing some celebrity you saw on the cover of People magazine -- the result would be a mess. Until we let go of our mental images of who we are or who we should be, our vision remains clouded by expectation. But when we let go of everything, open ourselves to any truth, and see the world without fear or judgement, then we are finally able to begin the process of peeling off the shell of false identity that prevents our true self from growing and shining in to the world. And it starts with nothing.

The two paths to success

2011-02-02T18:31:00.001-08:00

The recent WSJ article on the supposedly Chinese style of parenting has generated a lot of interesting discussion. The most amusing commentary comes from The Last Psychiatrist, who also points out that Amy Chua, the "Chinese" mother, was actually born in America. There were also claims that the WSJ misrepresented her views, which may or may not be true, but is ultimately irrelevant since it's the ideas that are being debated.

Here's the part of the article that interests me:

What Chinese parents understand is that nothing is fun until you're good at it. To get good at anything you have to work, and children on their own never want to work, which is why it is crucial to override their preferences. This often requires fortitude on the part of the parents because the child will resist; things are always hardest at the beginning, which is where Western parents tend to give up. But if done properly, the Chinese strategy produces a virtuous circle. Tenacious practice, practice, practice is crucial for excellence; rote repetition is underrated in America. Once a child starts to excel at something -- whether it's math, piano, pitching or ballet -- he or she gets praise, admiration and satisfaction. This builds confidence and makes the once not-fun activity fun. This in turn makes it easier for the parent to get the child to work even more.

She has two main points here: 1) Learning is not fun and 2) It's important to make kids dependent on praise and admiration.

One of the problems I've faced throughout life is that I'm kind of lazy, or maybe I lack will power or discipline or something. Either way, it's very difficult for me to do anything that I don't feel like doing. If I try to force it, my energy disappears, and I hate life. Furthermore, not only were my parents not Chinese, but they had five kids, so there wasn't time for Amy Chua's style of parenting. I kind of had to figure it out on my own.

My strategy can be reduced to two rules: 1) Find a way to make it fun and 2) If that fails, find a way to do something else.

For example, I didn't really like school, and I especially hated homework, so I turned it into a game to see how little time and effort I could possibly waste on it while still getting "good enough" grades. I barely made it into the top 10% of my public high school class, so I probably wouldn't be accepted into whatever college Amy Chua has picked out for her kids, but I find that I really don't care. Instead, I went to a "good enough" college, slept through most of my classes, then got a "good enough" job after graduation. Meanwhile, I taught myself how to program and build all kinds of things, which to me was much more fun than school.

I'm not going to claim that my approach is superior to Amy Chua's, or that it will work for everyone, but I do think it provides an interesting contrast.

Amy Chua's approach is based on extrinsic motivation. Children must do exactly what they are told to do, and they must not be happy unless an external authority gives them praise and admiration. They must learn that their own internal motivations and judgement are worthless and not to be trusted. They are successful when an external authority, such as an Ivy League school, tells them that they are successful.

The extrinsic path to success is to focus on being the person you are told to be, and put all of your energy and drive into fitting that mold.

The approach I stumbled into is based on intrinsic motivation. To the greatest extent possible, do whatever is most fun, interesting, and personally rewarding (and not evil). External constraints, such as the need to go to school or make money are simply obstacles to be hacked. Be skeptical of external authorities, as they are often manipulating you for their own benefit, or for the benefit of the institutions they represent (often unknowingly, as they were already captured by the same systems which are attempted to ensnare you). Your identity comes from within -- external recognition such as degrees and awards are only of tactical importance -- don't allow them to define who you are.

The intrinsic path to success is to focus on being the person that you are, and put all of your energy and drive into being the best possible version of yourself.

Of course this leads to the question, "What is success?" Someone who spent his life working 80 hour weeks, living in hotels, and fighting his way up the corporate ladder to become VP of toilet paper marketing would probably consider himself more successful than a sandwich shop owner who spends his nights and weekends playing with his kids and working on hobby projects, but maybe the sandwich shop owner would be happier and healthier. Ultimately, it is up to each person to decide what success means to them, but I think it's important that everyone be mindful of the decision they are making.

It's often said that people become entrepreneurs because they can't handle a regular job. Perhaps these people are simply too "defective" to fit into any mold, or maybe they lack the extrinsic motivation necessary to care about bosses, performance reviews, and other things which are so important for success in the corporate environment. However, what they do have is the creativity and natural sense of direction necessary to run their own business. I doubt that this is a coincidence.

As explained in this TED talk by Dan Pink, extrinsic motivation is a great way to get people to do boring and repetitive tasks, but it actually harms performance on more creative tasks. Creativity is a surprisingly fragile thing. It comes from deep inside, and external concerns (most especially, "What will people think?") seem to scare it away. But that's a topic for another time.

Of course, intrinsic vs extrinsic motivation isn't a completely black and white distinction, and we probably can't survive entirely on one or the other (I aim for 90% intrinsic). I also doubt it's possible to simply "switch" from extrinsic to intrinsic motivation. It's a skill like anything else. It takes time to find your internal voice, learn when to trust it, and stop fearing outside opinion (closely related: ego-fear).

Amy Chua's book was about parenting. Her style is based on extrinsic motivation. How do we raise successful, intrinsically motivated children? I'm sure someone will leave book recommendations in the comments -- Alfie Kohn comes to mind. However, I suspect that one of the most important factors is how we live our own lives. If we demonstrate that work and creativity can actually be fun and enjoyable, that at least sets an example. If we first solve the puzzle for ourselves, we have a better chance of helping others to find their answer.

Angel investing, my first three years

2011-01-03T16:43:00.001-08:00

I started investing in startups with the assumption that I don't know what I'm doing (which is always true), but that the only way to actually learn anything is through experience. Therefore, my goal was to invest in a variety of startups, learn from the experience, help some startups, and hopefully not lose too much money while doing so. I don't have any single criterion for investing. Sometimes, the idea seems good, sometimes the people seem good, and other times I'm just curious to see what will happen. No matter what though, I want to be helpful and learn something interesting from the experience.

I've definitely learned a lot, but the recent Heroku acquisition (for a reported $212 million) made me wonder if I've also reached the point of "not losing too much money", so I did the math. From 2006 (when I started investing) through 2008, I invested about $1.21 million in 32 different companies. About half of those companies were either acquired or are dead/mostly-dead. The other half are still alive and seem to be doing well. The current acquisitions total about $1.34 million, only about a 10% gain, which isn't great, but at least I'm not losing too much. Fortunately, the "still alive" category includes a number of very promising investments, such as Meraki, Weebly, and Wufoo, so I expect the final return will be much better than 10% (which is all gravy anyway, since my primary goal was to learn more and be helpful).

Of the current acquisitions, only two have yielded a greater than 10x return: Heroku and Mint. Unfortunately they were also two of the smaller investments, proving that I don't know what I'm doing, or at least showing that I need to make a point of investing more money into the best companies (Mint was oversubscribed, but I don't remember why I didn't put more into Heroku -- edit: apparently it was also oversubscribed).

It's worth noting that the highest return was from a Y Combinator company (Heroku, winter 2008). I've been investing in the YC companies almost from the start (Wufoo was winter 2006), and they keep getting better. YC is attracting the best founders, and Heroku is just the tip of the iceberg. The more great YC companies there are, the more reasons there are for other smart founders to join YC, and I find myself asking non-YC companies why they aren't yet in YC. This trend definitely contributed to my decision to join YC as a partner.

There were also a couple of medium-sized acquisitions (AppJet/Etherpad and 280North) and several smaller but still nice (2x-3x) exits such as Auctomatic, Parakey, and Zenter. Sometimes people complain about these deals, but as much as I'd like to get a nice 10,000x exit, I'm certainly not going to complain when someone doubles my money!

A few companies (such as ScanScout) were acquired by other private companies, so I include those in the "still alive and doing well" category, since it was not an exit from the investor perspective (no liquidity). Only two companies are officially dead, but there are at least four "zombies" that still exist, but in a minimal form and are almost certainly worthless.

The two bits of advice that I always give to people looking to get started with angel investing are: 1) Assume you'll lose your money and 2) Plan on investing in a large number of companies. (my third bit of advice is to attend YC demo day) I think my experience so far validates this advice. It's important that investors have the right motivations (helping out startups and learning how to be a better investor) and the right expectations. Anyone doing it with the idea that they can simply find the next Google, invest a bunch of money, and then get super-rich is going to be very disappointed. That said, finding the next Google and buying a 1% stake is my current billion dollar plan :)

Four reasons Google is still Awesome

2010-12-21T15:44:00.000-08:00

My recent predictions about which Google products will succeed (and which won't) are causing people to think that I'm anti-Google, which makes me sad since Google is probably still the best company of its size, and I really enjoyed my time there.

Unfortunately, positive stories are never as popular as negative ones, but regardless, it's worth highlighting some of the things that continue to differentiate Google as one of the best companies in tech.

1 - They take big risks. People often point to projects such as Wave as evidence that Google has "lost its magic" or something. To me, it's evidence that they are still willing to take risks on new ideas and new ways of doing things (Wave was run as a completely autonomous project in Australia). If everything you do works, then you're not taking many risks and probably aren't innovating either. Obviously, if everything you do fails, that's not good either, but there's a sweet-spot somewhere in the middle. Google has enough big successes, such as Chrome and Android, to show that they are somewhere near that sweet-spot.

2 - They are willing to build new technology seemingly unrelated to the core business. When we started work on Gmail, many people said it was a distraction and that Google should focus on web search. Now nobody questions email, but they wonder why Google is developing self-driving cars. From a market perspective, this looks like a lack of focus, but that's a rather narrow way of viewing things. From a broader perspective, it can be seen as a focus on using technology to improve the world. Did people criticize Edison or Tesla for inventing too many different things?

From an employee perspective these non-core projects are also an opportunity for greater autonomy. Part of what made the Gmail project so fun was that we had a lot of independence and could pursue ideas that other people inside Google thought were "the wrong way to do it". Most other tech companies do not offer that kind of freedom.

3 - They compete in positive ways. Many companies compete in ways that actually destroy value, such as using patent lawsuits to slow down or kill competitors. Google's weapon of choice is more often open source and open standards. There's no question that projects such as Android and Chrome have strategic value and work to weaken Microsoft and others, but they also happen to be good for the world. Google has managed to keep their interests surprisingly well aligned with the interests of their users.

4 - They don't seem to mind honest criticism. I'm currently reading a draft of a forthcoming Google book, and was amused to find that it includes an email that I sent back in 2000 trashing our then most recent product launch. It's painful for me to not tell people what I think, so for the most part I try to find people who don't mind hearing the truth (or my take on it, rather). Much of my interaction with startups consists of me telling them everything that I don't like about their product (and then they thank me!). I've worked for a lot of different companies, and Google was the only one where me speaking my mind never seemed to cause a problem. I'm not claiming that I'm always right, because obviously I'm not, but systems (or individuals) that don't welcome negative feedback are doomed. Cultures that don't laugh at themselves are cults.

Talking about Google is always a little tricky for me given my background, but they continue to be a fascinating company and a great source interesting lessons, so I'm going to keep trying. Hopefully I don't come off as a hater or a fan boy, but simply an honest observer.

The Cloud OS

2010-12-18T14:35:00.001-08:00

My recent remark about the future of ChromeOS generated a surprisingly passionate response. Some said that my prediction was obvious and boring, but others disagreed, arguing instead that I am ugly and "don't get it". I won't disagree with either side, but I also noticed that my prediction was sometimes inaccurately characterized as me "hating" ChromeOS, Google, or The Cloud, all of which is false. Since there seems to be so much interest in this topic, and because people keep emailing me about it, I should probably explain my actual thoughts a little better.

First, what is a "cloud OS" and why should I want one? Actually, I don't even know if anyone calls it a "cloud OS", but I couldn't find a better generic term for something like ChromeOS. The basic idea is that apps and data all live on the Internet, which is has been renamed "The Cloud" since that sounds cooler, and your laptop or whatever is basically just a window into that cloud. If your laptop is stolen or catches fire or something, it's not a big deal, because you can just buy another one and nothing has been lost (except your money). Many people characterize this approach as using a "dumb terminal", but that's wrong. Your local computer can still do all kinds of smart computation and data manipulation -- it's just no longer the single point of failure.

To me, the defining characteristic of cloud based apps is "information without location". For example, in the bad old days, you would install a copy Outlook or other email software on your PC, it would download all of your email to your computer, and then the email would live on that computer until Outlook corrupted its PST file and everything was lost. If you accidentally left your computer at home, or it was stolen, then you simply couldn't get to your email. Information behaved much like a physical object -- it was always in one place. That's an unnecessary and annoying limitation. By moving my email into "the cloud", I can escape the limitations of physical location and am able to reach it from any number of computers, phones, televisions, or whatever else connects to the Internet. For performance and coverage reasons, those devices will usually cache some of my email, but the canonical version always lives online. The Gmail client on Android phones provides a great example of this. It stores copies of recent messages so that I can access them even when there is no Internet access, and also saves any recent changes (such as new messages or changes to read state), but as soon as possible it sends those changes to the Gmail servers so that they can be reflected everywhere else (such as my home computer). To the greatest extent possible, the information all "lives" in the cloud, and all other copies are simply caches which may be discarded at any time. (BTW, Apple is lame for not allowing a native Gmail app on the iPhone -- email is the one place where Android really outshines the iPhone for me)

Continuing with the Gmail example, it's not just your data that resides in the cloud -- the entire application lives there. This is the part that causes people to erroneously describe cloud based apps as a "return to dumb terminals". Just because an application "lives" in the cloud doesn't mean that your local computer isn't still doing work. When you use Gmail from your web browser, it downloads large chunks of Javascript code to run on your computer doing things such as rendering your inbox, handling keyboard and mouse events, pre-fetching messages, etc. The advantage of having this code run on your computer is that it can respond to your actions within a few milliseconds instead of the hundreds of milliseconds it could take to reach Google's servers (thanks to the relatively low speed of light). Which parts of the application run on your computer and which run on Google's computers? Ultimately, it does not matter, and can change over time (and in fact the split is different for different interfaces -- the basic html interface does not need any Javascript) As an end-user, you simply use the app, and let Google worry about making it all work, keeping it up to date, etc.

Because we're now treating the executable code and system configuration as data that lives in the cloud and is only cached locally, it also makes sense to do away with the old notions of installing and administering applications on your computer. And of course we also need a security and sandboxing system that prevents the code from breaking your computer (as is so common in the Windows world). In the web/Javascript world, this happened somewhat automatically because web apps evolved from simple web pages, and obviously you don't have to install or uninstall web pages -- your browser simply fetches what it needs to display, optionally caches parts of it for improved performance, and discards resources that it no longer needs (since it can always re-fetch them later on).

Cloud-based apps don't necessarily have to be written in Javascript and run in your web browser however. iPhone and Android apps behave in much the same way. Although they can be "installed" or "uninstalled", from a user perspective, that process isn't substantially different from adding or removing a bookmark, and in fact many of those apps are little more than a thin wrapper around an embedded web browser. A combination of technical and review policies prevent those apps from doing anything dangerous to your computer (unlike a Windows app, which could install a new device driver, replace a core system library, install a root-kit, etc).

One way of understanding this new architecture is to view the entire Internet as a single computer. This computer is a massively distributed system with billions of processors, billions of displays, exabytes of storage, and it's spread across the entire planet. Your phone or laptop is just one part of this global computer, and its primarily purpose is to provide a convenient interface. The actual computation and data storage is distributed in surprisingly complex and dynamic ways, but that complexity is mostly hidden from the end user. For example, interacting with my FriendFeed page involves the coordination of thousands of individual processors and disks owned by a dozen different entities, including you, Facebook, Amazon, Google, your ISP, and many intermediate ISPs. The same is true of the services provided by thousands of other web apps.

This global super-computer enables us to do things that would have been impossible not long ago, such as instantly search billions of documents, access our email and other info from almost anywhere, easily share ideas with thousands or millions of people, collaboratively edit documents with people spread around the world, leak embarrassing diplomatic cables, etc. It also makes it easy to launch new services and applications with almost zero money, which has created a new generation of low-budget startups and expanded the world of high-tech entrepreneurship to many more people.

Inevitable, some curmudgeonly types will say this is all bad, and indeed it is not without some downsides and complications, but overall I believe the development of this global super-computer is one of the most important technological advances in history.

And what about ChromeOS? If my laptop is just one part of a much larger computer, what is the ideal design for my local node? It should be relatively cheap and reliable, secure (no viruses or anything), zero-administration (I don't want to be a sys-admin), easy to use, and fast. I believe this is roughly the design target of ChromeOS. They are building laptops that run the Chrome web browser and approximately nothing else.

I actually like the idea of ChromeOS, so why did I predict its demise? The answer is that we already have millions of devices that almost meet the same ideal, and they are running iOS and Android. In the 1.5 years since ChromeOS was announced, Apple launched the iPad, which quickly became one of the fastest selling new devices ever. Google will necessarily respond by building Android tablets, which means Android will be running on larger, more powerful devices. All of the benefits of ChromeOS (security, instant-on, etc) should apply to Android as well, and I expect that any new Chrome features (mostly under the umbrella of HTML5, but perhaps Native Client as well) will also be added to the Android browser, since platforms succeed by being as large as possible. Once Android has all the benefits of ChromeOS, the most obvious difference will be that ChromeOS lacks the thousands of native apps which are popular on Android. Android apps are closer to web apps than Windows apps in terms of security and manageability, so eliminating them doesn't seem like much of an advantage for ChromeOS.

The other obvious difference between ChromeOS and Android is that ChromeOS assumes a mouse/track-pad while Android currently assumes a touch interface (many Android devices already have a keyboard). If my prediction is wrong and both OSs stick around, this will probably be the reason. However, I doubt that's enough of a difference to justify maintaining two separate OSs, and ultimately everything may end up with a touch screen anyway. Perhaps the tablet / laptop convertible will make a comeback.

Put another way, the ChromeOS laptops are awkwardly positioned between the established Mac/Windows laptop market, which isn't going away anytime soon, and the rapidly growing Tablet market, and it has approximately zero users. That's not a great place for a new platform to get traction and establish itself. But if it does, I will be happy for it. And even if it doesn't, it may still be a worthwhile experiment.

The quest for freedom and safety: Why I donated $100,000 to YesOn19

2010-10-29T14:34:00.000-07:00

The common theme uniting most of my efforts is the desire to be free. One of the reasons I'm so interested in startups is that they give people the freedom to create, independent of the institutional limitations found in large companies. This is why unexpected new ideas and techniques (such as new languages or development practices) often appear first in startups. Of course the startups don't always succeed, but at least I'm free to pursue my own ideas, even if others don't believe in them. And when a startup is successful, it can provide a great deal of financial freedom to the people who built it.

Internal freedom is also very important, though often less obvious. If we are always held back by our own fears and self-limiting beliefs, then we aren't really free. That is why my previous post on serendipity is largely about escaping ego-fear and other negative limitations.

Benjamin Franklin once wrote, "Those who would give up essential liberty to purchase a little temporary safety, deserve neither liberty nor safety."

Philosophically, I agree with that. However, if we don't feel safe, if we can't go out in public without fearing for our lives and the lives of our family, then we aren't really free. Since becoming a parent, I've come to understand why parents often seem especially fearful. Our children are so precious to us, and we must keep them safe. I can understand the impulse to simply make more rules, to build taller walls, and to lockup anyone who seems scary.

I think the real point of Benjamin Franklin's quote is that when we destroy freedom, we are ultimately destroying safety as well. This is most apparent when we examine the disastrous effects of drug prohibition.

Not only is prohibition an attack on our basic right to control our own bodies and minds (a philosophical point which most people probably don't care about), but prohibition also provides a multi-billion dollar subsidy to violent criminal organizations that threaten our physical safety and security, something everyone cares about.

The drug cartels have already overrun much of Mexico, and that violence will inevitably spill over in to the United States if we continue subsidising them with one of the world's most lucrative monopolies.

The alternative path is to begin restoring individual freedom and responsibility, defund the drug cartels, and instead shift those dollars towards roads, parks, public safety, and other beneficial causes. This is the solution offered by California's proposition 19, the "Regulate, Control and Tax Cannabis Act of 2010".

Some politicians have argued that proposition 19 is "flawed". To me, this seems like a weak defense of the status quo from politicians afraid to stand out on a controversial issue. Of course it's not perfect -- no law is perfect. However, the current system of drug prohibition is much, much worse. "Perfect" is not one of the options offered on Tuesday's ballot. We can either choose "much better" (Yes on 19), or "keep the current, disastrously bad, system" (No on 19).

If proposition 19 passes, the immediate effect may not be that significant due to federal challenges and such. However, I believe the long-term effects will be enormous. Prohibition is a disaster. Many politicians will admit to this fact, however most of them have been too timid to actually do anything about it, to lead the country towards safer, saner policies. In this case, the voters must lead, and the politicians will follow. Even if 19 does not pass (Nate Silver of FiveThirtyEight gives it "even odds" of passing), it will still mark an important shift in the debate over drugs, especially if YOU vote for it.

And that's why I decided to donate $100,000 towards the Yes on 19 campaign. It's tempting to wait for the "perfect" solution to the drug issue, but meanwhile millions of lives are being destroyed by the current system. That's evil.

Here are a few more thoughts on the issue:

Nicholas Kristof in the NY Times
The success of drug decriminalization in Portugal (from the Kristof article)
George Soros in the Wall Street Journal
San Jose Police Chief Joseph McNamara

If you would like to be part of the solution, please share this or other articles, and encourage your friends and family to show up at the polls on Tuesday and vote "YES on 19" (assuming they live in California). Also, donate to YesOn19 -- it's not too late (ads are still being purchased).

Serendipity finds you

2010-10-24T10:41:00.001-07:00

Here is an interesting comment from Hacker News, on a story about someone turning down an early Google offer:

Similar thing happened to me in 1999. I realized Google was way cooler than alta vista and better at finding unknown things rather than Yahoo's directory. Truly the future, I thought. I sent in a resume to do some kind of work not development related; data center & sys admin stuff. They called me twice but I convinced myself that they would not have hired me anyways so I never called back.

Whether or not ignoring Google's calls was the right decision for him, his reason for not taking the call (fear of rejection) isn't great.

I don't have many positive memories from high school, but the one that has stayed with me more than any other comes from the first day of my 11th grade English class. My teacher (I believe his name was "Mr. May") shared a brief anecdote from the prior evening. He was driving home in the rain, and noticed two people on bicycles along the side of the road. He stopped to ask if they needed any help, and ended up driving them back to his house, where they dried off and had dinner with him and his wife. During dinner, the couple shared the stories from their ongoing bike ride across the country.

It's not a very dramatic story, but I loved the serendipitous nature of it, both on the part of the couple having adventures biking across the country, and my teacher who saw people along the road and invited them into his home. None of it was planned -- they simply allowed it to happen. It was inspirational to me because it felt like the right way to live, the fun way to live. I don't think that's how most people operate though.

My own story of how I ended up at Google in 1999 is rather boring. I was interested both in startups, and Linux (which was still somewhat fringe at the time), so I sent my resume to a few companies that I had seen mentioned on Slashdot (a rather lazy job search, in hindsight). Fortunately, most of them never even responded, and only one actually offered me a job, Google. I was skeptical of their business and didn't expect it to last long, but it seemed like it could be fun and educational, so I accepted.

Obviously that's an example of rather extreme luck, but I've noticed that most of the good things that happen to me follow that general pattern, and aren't part of any "plan". The story of how I met my wife is remarkably similar. Shortly after moving to California, I signed up for match.com, read a bunch of profiles, emailed three of them, and only one responded. I was very much not looking for someone to marry, but that's what happened anyway. As they say, "Life is what happens to you while you're busy making other plans."

My plans rarely work (unless they are boringly simple), but serendipity has been good to me, so over time I've tried to make the most of that. My theory of serendipity is still evolving, but from what I've seen, it's better to think in terms of "allowing" serendipity rather than "seeking" it or "creating" it. Opportunity is all around us, but we have beliefs and habits that block it.

The two biggest blocks to serendipity seem to be ego-fear and "other plans".

I'm using the term "ego-fear" to describe fears that go beyond rational concern. For example, you wouldn't run out into the middle of a freeway thanks to a healthy fear of getting run over by a car -- that's not ego-fear. However, the fear that often keeps people from public speaking, talking to strangers, interviewing for jobs, etc is typically driven by fear of embarrassment, humiliation, rejection, criticism, etc -- that's ego-fear. Sometimes it can be difficult to separate the two types of fear because ego-fear will rationalize itself as healthy fear, e.g. "I don't want to talk to that stranger because they could attack me, or waste my time."

The HN commenter quoted above who never accepted Google's calls because, in his words, "they would not have hired me anyways", seems to be experiencing quite a bit of ego-fear, fear of rejection and humiliation. That fear is probably blocking a lot of great opportunities.

It's tempting to try and think your way out of ego-fear, but I suspect that only makes the problem worse by generating a more complex tangle of rationalizations for the fear. Fear is defeated by confrontation -- avoidance only makes it stronger.

"You gain strength, courage, and confidence by every experience in which you really stop to look fear in the face. You must do the thing which you think you cannot do." - Eleanor Roosevelt

The program for eliminating ego-fear and unblocking serendipity is very simple: seek ego-fear. Hunt it down and soak in it. Steal its energy. This is, by definition, scary. That's good.

The other big serendipity block seems to be "the plan". Serendipity and luck are by their very nature unpredictable, and therefore not part of any good plan. When something unexpected happens, things are no longer "going according to plan", and there is a tendency to view the unexpected event either as a distraction, or as a frustrating obstacle to success.

The difference between a life full of frustrating obstacles, and a life full of serendipity, is largely a matter of interpretation. It can be difficult, but the most beneficial response to unexpected events is a sense of gratitude. Even seemingly adverse events can lead to something great. Accept what is given. (see Yes Man for a cute caricature of this mindset)

"Plans are worthless. Planning is essential." - Dwight D. Eisenhower

Planning in itself is not a bad thing, but picking a single plan and obsessively sticking to it doesn't allow for much serendipity. The world is very complicated, and we humans are very stupid, so it's good to be flexible and open minded about things. Instead of having one plan, have one thousand plans, and revise them as necessary.

The desire to have "a plan" can also cause "paralysis of analysis" -- we put all of our energy into formulating the perfect plan, and consequently never actually do anything. The more effective approach is to simply pick a plan with the knowledge that it's flawed, set the plan in action, and then adapt, revise, or switch plans as the world unfolds.

I suspect the desire to have a definite plan is also partially rooted in fear. Uncertainty can be scary, and having a plan helps create the illusion of predictability in a very unpredictable world. However, if we actually manage to reduce risk and unpredictability, then we are also reducing serendipity. This is one reason why large organizations often have trouble producing innovation -- they want it to be planned and scheduled, but that just kills it.

The whole notion that plans are something that we should "stick to" makes them distracting enough that I prefer to call them "ideas" or "rough sketches" instead. Personally, I try to avoid having plans for my life, but I have many ideas. Which ones actually happen will be a surprise to me. It's more fun that way.

What to do with your millions

2010-05-27T16:54:00.001-07:00

There's another "I just got a bunch of money, what do I do now?" type post on Hacker News today, and much of the advice is from people who clearly don't know, though the current top comment is actually very good. Since this is a relatively common issue (ha ha) in the startup world, I think it's worth sharing a little of what I've learned from observing others who have this "problem" (yeah, cry me a river, right?). This is somewhat dangerous since money is a very delicate topic for many people, so if you have any strong feelings, please skip this post.

Although today's poster only asked, "What do I do with my money?", there's a second, related question that's also very important, "What do I do with my life?" In both cases, I think the right answer is, "start slow, and avoid making any big decisions now", though as always, there are exceptions.

The money question is the easier of the two to answer: First, don't lose the money!

Many people will naively tell you to immediately hire a financial advisor. What those people don't understand is that the only skill a financial advisor needs in order to be successful is the ability to sell you things. Their actual financial skills are almost irrelevant. Unfortunately, this means that you will need to learn something about money management, and that will take time. Fortunately, you have plenty of time. Read what Warren Buffett has to say about financial helpers. Spend a few years getting recommendations and talking to various advisors before deciding (intermittently, not full-time, of course). Avoid hiring this guy. Meanwhile, put your money in a very safe fixed-income investment, such as short-term CDs. You can circumvent the FDIC insurance limit by having the money spread accross multiple banks (think of it as "RAID for money") -- see CDARS for more info. Don't rush to invest it in the stock market -- that's risky and you could easily lose half of your money in a matter of months. Avoid long-term or illiquid investments, though it's fine to put a few percent into random things such as startups, but understand that you'll probably lose that money, so consider it an educational expense.

Edit: Many people have incorrectly interpreted my advice as, "don't hire a financial advisor". My actual advice is, "don't rush to hire a financial advisor -- just keep things very simple and take time to decide what is right for you." I personally have a bunch of advisors.

Longer term, you'll probably want to diversify into other types of investments. Unfortunately, there's no simple formula for how to do this, and the right answer will depend on your own financial particulars, emotional composition (how does losing money make you feel?), etc. Again though, the most important thing to understand is that you don't need to decide this now. If anyone pressures you to do anything right now (especially financial advisors), tell them that you are not presently interested in their services, only be less polite about it :)

Now for the hard question: What to do with your life?

First, it's important to understand that once you have the basics, happiness comes primarily from healthy social connections and a sense of purpose. If you quit your job and move to a new city where you don't know anyone or have a clear purpose, there's a good chance that you'll end up depressed or even suicidal. So unless your current life is very broken, don't do that. Take it slow.

Many people with jobs have a fantasy about all the amazing things they would do if they didn't need to work. In reality, if they had the drive and commitment to do actually do those things, they wouldn't let a job get in the way. Unfortunately, if given a lot of money, they are much more likely to end up addicted to crack, or even worse, World of Warcraft. (edit, since people are getting offended: there are, as always, exceptions, but the point is that actually doing stuff is about a million times harder than just dreaming about it, which is why 99% of people wouldn't actually do it even if money weren't an issue) If you've been institutionalized your entire life (school, work, etc), it can be very difficult to adjust to life on "the outside".

Again, don't make any drastic changes unless you really need to. Spend time building up new activities, interests, and social connections, especially ones that will give your life a sense of purpose.

Money can also mess with your identity in bad ways. It's important to remember that we're all made of the same shit -- some people are just a little luckier than others. The nice thing about money is that it gives you more freedom, but it can also be a prison if it takes over your identity, makes you fearful, or causes you to cut off connections with the people around you. True freedom comes from the inside anyway -- we're all still slaves to the larger system. (while searching for a story to illustrate this point, I ended up on Epictetus's Wikipedia page -- he seems to have had it about right, so I'll go with that, though The Matrix is also entertaining)

Sometimes, good fortune can also make people feel guilty. But if you find yourself in this situation, you were probably already very lucky (reasonably healthy, intelligent, well educated, etc) -- there's no reason to feel bad about getting slightly more lucky. Your luck is a gift. It's ungrateful to not make the most out of it (and also help others become a little more lucky).

Explore the opportunity. Do something remarkable. Go for a walk in the park. Appreciate the trees.

Today is my wife's birthday...

2010-04-03T14:41:00.000-07:00

For her birthday, she wants as many people as possible to donate to her cause, raising $55,555 for the Intensive Care Nursery (ICN) at UCSF Children's Hospital, where our daughter spent the first several months of her life. Proceeds will go towards funding hospital neonatal supplies and monitoring equipment, family-oriented support programs, and neurodevelopmental programs.

To donate, go to causes.com/ucsfpreemies. You can also contribute by: 1) Donating directly using UCSF's 'Make a Gift' page. 2) Making a contribution offline, by sending a check. Please write your check out to 'UCSF Foundation', indicate on the memo line 'UCSF Preemies', and mail to: UCSF P.O. Box 45339 San Francisco, CA 94145-0339 3) For those of you who are contributing through your Donor Advised Fund, please reference the foundation's EIN/tax ID#: 94-2829914

If your product is Great, it doesn't need to be Good.

2010-02-09T00:53:00.000-08:00

By now, everyone is tired of hearing about the iPad, but the negative responses are so perfectly misguided that it would be wrong to waste this opportunity. Even better, we can look back at the 2001 iPod launch and see the exact same mistakes. But this isn't about the iPad or the iPod -- it's about product design.

The most famous iPod review was from Slashdot, which simply declared, "No wireless. Less space than a nomad. Lame." The iPad reviews are similar in that they focus on the "missing" features. Those missing features are typically available in a variety of unsuccessful competing products, which leads people to erroneously conclude that a successful product would necessarily have even more features!

I believe this "more features = better" mindset is at the root of the misjudgment, and is also the reason why so many otherwise smart people are bad at product design (e.g. most open source projects). If a MacBook with OSX and no keyboard were really the right product, then Microsoft would have already succeeded with their tablet computer years ago. Copying the mistakes of a failed product isn't a great formula for success.

What's the right approach to new products? Pick three key attributes or features, get those things very, very right, and then forget about everything else. Those three attributes define the fundamental essence and value of the product -- the rest is noise. For example, the original iPod was: 1) small enough to fit in your pocket, 2) had enough storage to hold many hours of music and 3) easy to sync with your Mac (most hardware companies can't make software, so I bet the others got this wrong). That's it -- no wireless, no ability to edit playlists on the device, no support for Ogg -- nothing but the essentials, well executed.

We took a similar approach when launching Gmail. It was fast, stored all of your email (back when 4MB quotas were the norm), and had an innovative interface based on conversations and search. The secondary and tertiary features were minimal or absent. There was no "rich text" composer. The original address book was implemented in two days and did almost nothing (the engineer doing the work originally wanted to spend five days on it, but I talked him down to two since I never use that feature anyway). Of course those other features can be added or improved later on (and Gmail has certainly improved a lot since launch), but if the basic product isn't compelling, adding more features won't save it.

By focusing on only a few core features in the first version, you are forced to find the true essence and value of the product. If your product needs "everything" in order to be good, then it's probably not very innovative (though it might be a nice upgrade to an existing product). Put another way, if your product is great, it doesn't need to be good.

So where does this leave the iPad, with it's lack of process managers, file managers, window managers, and all the other "missing" junk? I'm not sure, but one thing I've noticed is that I spend more time browsing the web from my iPhone than from my laptop. I'm not entirely sure why, but part of it is the simplicity. My iPhone is ready to use in under 1/2 second, while my laptop always takes at least a few seconds to wake up, and then there's a bunch of stuff going on that distracts me. The iPhone is a simple appliance that I use without a second thought, but my laptop feels like a complex machine that causes me to pause and consider if it's worth the effort right now. The downside of the iPhone is that it's small and slow (though the smallness is certainly a feature as well). That alone guarantees that I'll buy one to leave sitting next to the couch, but I'm kind of atypical.

Ultimately, the real value of this device will be in the new things that people do once they have a fast, simple, and sharable internet window sitting around. At home we'll casually browse the web, share photos (in person), and play board games (Bret's idea -- very compelling). At the office, maybe we'll finally have an easy way of chatting with remote people while discussing a presentation or document (e.g. audio iChat with a shared display). Of course these things are theoretically possible with laptops, but it always ends up being so clumsy and complicated that we don't bother (or give up after trying once).

Making the iPad successful is Apple's problem though, not yours. If you're creating a new product, what are the three (or fewer) key features that will make it so great that you can cut or half-ass everything else? Are you focusing at least 80% of your effort on getting those three things right?

Disclaimer: This advice probably only applies to consumer products (ones where the purchaser is also the user -- this includes some business products). For markets that have purchasing processes with long lists of feature requirements, you should probably just crank out as many features as possible and not waste time on simplicity or usability.

Censorship flamewar

2010-01-26T09:33:00.001-08:00

This post is inflammatory and unfair. It argues an extreme position that I don't agree with, but nevertheless find amusing. When writing angry responses, please direct your hate at the straw-man, not at me :)

Agree/Disagree:

When a powerful group forces information to be removed from the internet, that is censorship. Some acts of censorship are more acceptable than others depending on what information is being censored and why.

For example, Disney can force people to take certain information off of the internet because they have exclusive rights, and the free availability of that information threatens to undercut their profits, which would undercut their power to make new movies and also new laws to protect their interests (such as retroactively extending copyright, or increasing penalties for violations). In this case, censorship is good because if Disney lost that power, their profits could disappear entirely and then the world might run out of Princess movies. Perhaps someone else would start making Princess movies, but making Princess movies is difficult, and without the ability to censor the internet, they too might fail.

A second example is the Communist Party of China. They can force people to take certain information off of the internet because they have exclusive rights, and the free availability of that information threatens to undercut their power and profits, and without that power they could lose control of China. In this case, censorship is good because if the Communist party lost that power, their control could disappear entirely and they would no longer be able to preserve the peace, stability, and growth of China. Perhaps someone else would start governing China (after a quick revolution), but governing China is difficult, and without the ability to censor the internet, they too might fail.

Debate.

Books and stuff: three years of Amazon addiction

2010-01-04T20:44:00.000-08:00

I'm a slow reader, but that doesn't stop me from buying a lot of books. Inspired by a Blippy review, I thought it might be fun to publish my Amazon order history.

My threshold for buying books is very low, so don't interpret these purchases as endorsements. Also, I haven't read most of them yet, so if you spot anything especially good, please let me know.

10 predictions for the world of January 1, 2020

2010-01-02T15:53:00.001-08:00

After writing my predictions for the past 10 years, I decided it might be fun to write a few predictions for the next 10 years as well. This is a little more dangerous though, since I now lack the benefit of hindsight and at least one of them will probably turn out to be as dumb as "Palin/Gore win the 2014 presidential election".

That said, here are my (random and probably over-optimistic) predictions for the world as of Jan 1, 2020, assuming civilization doesn't collapse first:

All data lives "in the cloud" and can be accessed from anywhere. Most computers are essentially stateless caches which can be replaced without any data-loss or need for reconfiguration or reinstallation. This prediction was copied from 10 years ago, but this time it's right.
Android and iPhone kill off all the other mobile phone platforms. Android will be bigger (it will run on all of the "free" cell phones), but iPhone will still be "cooler", and will work more seamlessly with Apple's tablet computer.
Facebook will be a big success, possibly as big as Google. I'm probably over-optimistic because I work there, but the people are smart and ambitious.
3D displays will be popular because people want to watch Avatar and all future 3D movies and games at home (plus porn, of course -- think of the porn!).
No human-like artificial intelligence, but computers get a lot better at both high level and low level intelligence. At the high level, Google will release an amazing question answering service that can answer complex questions and is in many ways smarter than any human. Low-level, insect-like intelligence will become common enough that I'll be able to quickly build a Lego robot that uses computer vision to spot ants and physically squash them (and it will be awesome).
There will be an even bigger economic crash. The system becomes increasingly unstable as we try to paper-over the damage from one bubble by creating an even bigger bubble elsewhere. Nevertheless, things continue semi-working for some reason.
The health care system continues to be badly broken, and attempts at reform probably only make it worse (here is a nice summary). Fortunately, solutions will come from outside of the system. People will get better at avoiding disease (start with Taubes), and bio-tech will start to deliver in a big way. Unfortunately, the bio-tech will probably come from China, since they don't have so many rules to slow things down. The Chinese treatments won't be approved in the United States, but we can simply travel to Mexico for care. Urgent care will be local, but non-urgent procedures will be performed outside the US. Not everyone will do this, but it will be a very visible and growing trend by 2020.
The drug war will be over, mostly. An increasing number of people will realize that the war is causing a lot more harm than the drugs are (it's killing Mexico, for example). The end of marijuana prohibition in California will demonstrate that non-addictive drugs are not as dangerous as the government claims. It's unclear what will happen with highly addictive drugs such as heroin, but bio-tech may offer a solution that removes the addiction.
The energy problem is largely solved by cheap solar. It's not 100% done obviously, but we'll be on the ramp. Nuclear technology (Thorium?) could do it too, but probably isn't politically feasible. Edit: To be clear since not everyone has the same definition of "solved", for me it means that we have a relatively cheap and scalable energy solution, not that we're fully converted to it.
Politics will evolve much faster than in the past due to the Internet and social networks changing the core architecture of society (what people think and how they came to think it -- the professional media and government are no longer at the center). New movements will arise and gain power very quickly. Obama beating Clinton was the first hint of this. Political outsiders will begin winning elections. Steven Colbert will win the 2016 election -- the left will think he's joking, the right will think he's serious, and both sides will think that they are tricking the other ;)

My poorly remembered and partially imagined predictions from Jan 1, 2000

2010-01-01T22:49:00.000-08:00

Reviewing old predictions is fun. Unfortunately, I didn't bother to write any down, so I'm working from memory here, and of course human memory is very unreliable and selective, so this is rather bogus.

Predictions for the next 10 years, from Jan 1, 2000 (as remembered on Jan 1, 2010):

Prediction: Linux will continue expanding into new spaces and will eventually make Microsoft irrelevant. Windows 2000 is the last release that anyone will care about.

Result: I was over-optimistic about Linux -- the community is unable to produce anything worthwhile on the desktop, most development has moved to the web anyway (making the desktop OS irrelevant), and OSX popularity among developers took away a lot of energy (I develop in Linux, but use OSX to host my web browser and other apps). Linux is a huge success on the server-side though and continues to grow in more "embedded" contexts, such as Android and Chrome OS. Microsoft made itself irrelevant -- they still make a lot of money, but they are no longer changing the world.

Prediction: Wireless data access (such as that provided by Ricochet) will become fast and practically ubiquitous, meaning that the Internet is always with us.

Result: It took longer than I expected, but we're finally starting to get there, and it is awesome.

Prediction: "Computer Aided Reality" will provide a cool visual overlay by using computer vision to identify objects and then fetching info about them from the Internet.

Result: I was very over-optimistic. There are a few basic "augmented reality" apps around, but nothing major. I still think that this will happen, though the display technology is still very uncertain (I haven't even heard about direct retinal projection lately).

Prediction: All data lives "online" and can be accessed from anywhere. Your computer is a stateless cache which can be replaced without any data-loss or need for reconfiguration or reinstallation.

Result: I was over-optimistic. We made some progress with things like Gmail, but computers still store information (configuration at the very least). Chrome OS may be more like what I had in mind, and the iPhone is too if you set aside the fact that it requires manual syncing.

Prediction: The Java VM will get good enough that people will finally stop using C++.

Result: I was over-optimistic. The JVM got somewhat better (though it still has significant GC problems), but Java got worse due to cultural problems. Fortunately, a lot of other interesting languages became popular, including Javascript, and there are a number of fast virtual machines in the works.

Prediction: Google will be a big success, possibly as big as Yahoo. I'm probably over-optimistic because I work there, but the people are smart and ambitious.

Result: I was under-optimistic. Google is bigger than Yahoo ever was, and is getting close to Microsoft (their market cap is 70% of Microsoft's).

Prediction: Humans will be cloned. After the initial outrage, people will stop caring once they see that the result is just a regular baby (like the "test tube babies" before them), not a "soulless monster".

Result: To my knowledge, that hasn't been any successful human cloning, though I wouldn't be surprised if there has and they are just keeping quiet about it.

Prediction: The stock market will crash, and take Silicon Valley (and other bubble-zone) real estate markets down with it.

Result: The market crash came (sort of), but instead of going down, real estate kept going up! Even when it did finally crash, local prices (especially Palo Alto) remained remarkably high.

Prediction: A meteor strike will destroy all life on Earth on November 5, 2007, so I don't need to waste time writing down my predictions for 2010.

Result: I meant 2012, there was an off-by-5 error in my calculation ;)

Tablet thoughts

2009-12-31T15:15:00.000-08:00

I don't know anything about Apple's tablet, and I generally don't pay much attention to the speculation about their unannounced products. However, John Gruber has a nice post today which discusses the hypothetical Apple tablet.

The best part is the core product question -- how does this new product fit in with all existing products? Revolutionary products are underestimated because we evaluate them relative to existing products. This quote gets it:

Like all Apple products, The Tablet will do less than we expect but the things it does do, it will do insanely well. It will offer a fraction of the functionality of a MacBook‚ but that fraction will be way more fun. The same Asperger-y critics who dismissed the iPhone will focus on all that The Tablet doesn't do and declare that this time, Apple really has fucked up but good.

That was certainly the case with the iPod, or as slashdot put it, "No wireless. Less space than a nomad. Lame."

When considering revolutionary new products, we can not simply compare them with existing products, but must instead compare them with the products that don't yet exist, but should. For example, the PC was more than just an expensive, hard-to-use typewriter -- it was a whole new thing that just happened to have some typewriter features. Obviously this comparison is much more difficult than the "count the checkboxes" approach that we like to use when evaluating the "better mousetrap", but it's critical if we're going to understand or create anything truly new.

I have no idea what Apple is planning to release, but to me the revolutionary product need is in bridging the virtual and physical worlds. If you spend your entire day in front of the computer, this need may not seem real, but if you move between the two worlds you may notice that they are strangely disconnected. For example, imagine that I'm looking at a picture on my computer and want to give you a copy. In the physical world, I would simply hand you the print (I would have gotten double-prints), but with computers it's nearly impossible. Yes, there may be some complicated 10-step process that I can use to share the image, or maybe I can download and install some obscure software, but I'm not going to do that and neither are most other people. Imagine if I instead had a simple (built-in) gesture for passing the photo off to person standing next to me, and it were just as easy as handing them a real photo.

Of course exchanging photos is just one small example of these physical/virtual interactions. It's a whole new category, so many of them, including the most important, haven't been invented yet. However, you can get some ideas by thinking of the marketing cliche where two people are standing around a computer collaborating on something, taking quick notes, working off a recipe, etc. Those images occur in marketing because they are appealing, but they don't occur much in real life because our existing devices and software are awful. Current laptop computers are too bulky, awkward, and keyboard centric (the ui needs to be gesture-centric), and the iPhone is too small and limited. I want something about the size of a notepad that can be used naturally while standing up and walking around, just like an actual pad of paper, except that it's fully integrated with the virtual world as well as the physical world.

I hope this is what Apple is building -- it would be a great product. (or someone else could build it, though honestly I can't imagine anyone besides Apple getting it right)

So I finally tried Wave...

2009-11-28T15:46:00.001-08:00

Last week, TechCrunch published a story about me not yet trying Google Wave ("Gmail Creator Thinks Email Will Last Forever. And Hasn't Tried Google Wave"). The is apparently unacceptable, or as one commenter put it, "Paul may have been trying to be cool and ironic, but really he should be ashamed for not having tried Wave yet." I'm not sure if this is because I have an obligation to try all new products, or because my views on the longevity of email will seem hopelessly naive once I try Wave, but either way, I mustn't disappoint the good people of TechCrunch :)

The Google Wave About page and video does a good job of summarizing what Wave is and how it works. If you want to learn more about Wave, I would start there and skip this post. That said, here are my thoughts on Wave:

First off, Wave is clever and full of interesting ideas.

Second, comparisons to Facebook and Twitter are nonsensical. If Twitter were CNN Headline News, Google Wave would be Microsoft Office. Wave is less of a social network and more of a productivity tool. It's Google Docs meets Gmail, or as Google puts it, "A wave is equal parts conversation and document. People can communicate and work together with richly formatted text, photos, videos, maps, and more."

Third, although Wave is very promising, it's clear that it still needs some refinement. This is why Google calls it a "preview release". The trouble with innovative new ideas is that not all of them are worth keeping. While developing Gmail, we implemented a lot of features that were either not released, or not released until much later. Some of the most interesting ideas (such as automatic email prioritization) never made it out because we couldn't find simple enough interfaces. Other ideas sounded good, but in practice weren't useful enough to justify the added complexity (such as multiple stars). Other features, such as integrated IM, simply needed more time to get right and were added later. Our approach was somewhat minimal: only include features that had proven to be highly useful, such as the conversation view and search. It's my impression that Wave was released at an earlier stage of development -- they included all of the features, and will likely winnow and refine them as Wave approaches a full launch. The Wave approach can be a little confusing, but it allows for greater public feedback and testing.

From what I've seen, the realtime aspects of Wave are both the most intriguing, and the most problematic. I think the root of the issue is that conversations need to be mostly linear, or else they become incomprehensible. IM and chat work because there is a nice, linear back-and-forth among the participants. Wave puts the conversation into little Gmail-like boxes, but then makes them update in realtime. The result is that people end up responding (in realtime) to things on other parts of the page, and the chronological linkage and flow of the conversation is lost. I suspect it would work better if each box behaved more like a little chat room. A single Wave could contain multiple chats (different sub-topics), but each box would be mostly self-contained and could be read in a linear fashion.

So now that I've tried Wave, do I expect it to kill email? No. The reason that nothing is going to kill email anytime soon is quite simple: email is universal (or as close to it as anything on the Internet). Email has all kinds of problems and I often hate it, but the fact is that it mostly works, and there's a huge amount of experience and infrastructure supporting it. The best we can do is to use email less, and tools like Wave and Docs are a big help here.

I don't know what Google has planned for Wave or Gmail, but if I were them I would continue improving Wave, and then once it's ready for the whole world to use, integrate it into Gmail. Moving Wave into Gmail would give it a huge userbase, and partially address the "email is universal" problem. They could use MIME multi-part to send both a non-Wave, HTML version of the message, and the Wave version. Wave-enabled mail readers would display the live Wave, while older mailers would show the static version along with a link to the live Wave.

Open as in water, the fluid necessary for life

2009-11-20T15:40:00.001-08:00

"Open" is a great thing. Everyone likes it. Unfortunately, nobody agrees what open is. There are many meanings, but in general, I think "open" must be the opposite of "closed". In the world of abstract things like software, protocols and society, closed is secret, hidden, or locked.

"Closed" limits our mobility, prevents discovery, and discourages new connections. Imagine being in a building where all of the doors are locked or guarded, and it's difficult to move from room to room or leave. A closed world is one where people are forced to stay in their place, sometimes because of physical constraints, but more commonly because they simply don't know where else to go. A closed world is giant prison.

In an open world, people are able to see more clearly, and more easily explore new ideas and possibilities. An open world is more fluid -- people and ideas easily flow over boundaries and other borders. This openness is what makes the Internet so powerful. The Internet is melting the world, but in a good way.

Open standards and open source software are important for making technology open and available to everyone, but it's important to remember that open goes beyond tech. Wikipedia makes knowledge open to everyone. Blogs and YouTube make broadcasting and mass communication open to everyone -- news and events that would have been suppressed in the past are now reaching the whole world.

These things have been discussed to death, but there's another "open" that still seems a little frivolous: our lives. We like to joke (or complain) about people who share every boring detail of their lives and thoughts on Facebook or Twitter, but they may be doing something important.

Most of our happiness and productivity comes from the everyday details of our lives: the people we live and work with, the books we read, the hikes we take, the parties we attend, etc. But how do we choose these things? How do we know what to do, and how do know if we'll like it? The obvious answer is that we do and like whatever the TV tells us to do and like. I'm not certain that's the best answer though.

By sharing more of our own thoughts and lives with the world, we contribute to the global pool of "how to live", and over time we also get contributions back from the world. Think of it as "open source living". This has certainly been my experience with my blog and FriendFeed. Not only do people occasionally say that it has helped them, but I've also met interesting new people and gotten a lot of good leads on new ideas. These are typically small things, but our lives are woven from the small details of everyday living. For example, I saw a good TED talk on "The science of motivation", shared it on FriendFeed, and in the comments Laura Norvig suggested a book called Unconditional Parenting, which turns out to be very good.

The next step is for people to open more of their current activities and plans. This is often referred to as "real-time", but since real-time is also a technical term, we often focus too much on the technical aspect of it. The "real-time" that matters is the human part -- what I'm doing and thinking right now, and my ability to communicate that to the world, right now. We see some of this on Facebook, FriendFeed, and Twitter, and also location-aware apps such as Foursquare, but it's still fairly primitive and fringe. When this activity reaches critical mass, it should be very interesting for society. It dramatically alters the time and growth coefficients in group formation. It enables a much higher degree of serendipity and ad hoc socializing.

The basic pattern of openness is that better access to information and better systems lead to better decisions and better living. This general principal is broadly accepted, but we're just now discovering that it also applies to the minutiae of our lives.

Sharing your boring thoughts and activities may seem narcissistic and self-absorbed at first (I'm still kind of embarrassed about having a blog), but there is virtue and benefit in it. Naturally there will be challenges and fear along the way, but in the long term we're contributing to a more open, fluid society, where people are more able to find happy, productive lives. It also encourages us to be more accepting of others. Everyone is flawed, and the more we see that we aren't alone, the less we need to fear that truth.

People can not truly live and thrive in a prison -- we require freedom and mobility. This may explain my incomprehensible analogy, "Open as in water, the fluid necessary for life".

Go forth and share.

Applied Philosophy, a.k.a. "Hacking"

2009-10-13T21:53:00.001-07:00

Every system has two sets of rules: The rules as they are intended or commonly perceived, and the actual rules ("reality"). In most complex systems, the gap between these two sets of rules is huge.

Sometimes we catch a glimpse of the truth, and discover the actual rules of a system. Once the actual rules are known, it may be possible to perform "miracles" -- things which violate the perceived rules.

Hacking is most commonly associated with computers, and people who break into or otherwise subvert computer systems are often called hackers. Although this terminology is occasionally disputed, I think it is essentially correct -- these hackers are discovering the actual rules of the computer systems (e.g. buffer overflows), and using them to circumvent the intended rules of the system (typically access controls). The same is true of the hackers who break DRM or other systems of control.

Writing clever (or sometimes ugly) code is also described as hacking. In this case the hacker is violating the rules of how we expect software to be written. If there's a project that should take months to write, and someone manages to hack it out in a single evening, that's a small miracle, and a major hack. If the result is simple and beautiful because the hacker discovered a better solution, we may describe the hack as "elegant" or "brilliant". If the result is complex and hard to understand (perhaps it violates many layers of abstraction), then we will call it an "ugly hack". Ugly hacks aren't all bad though -- one of my favorite personal hacks was some messy code that demonstrated what would become AdSense (story here), and although the code was quickly discarded, it did it's job.

Hacking isn't limited to computers though. Wherever there are systems, there is the potential for hacking, and there are systems everywhere. Our entire reality is systems of systems, all the way down. This includes human relations (see The Game for an very amusing story of people hacking human attraction), health (Seth Roberts has some interesting ideas), sports (Tim Ferriss claims to have hacked the National Chinese Kickboxing championship), and finance ("too big to fail").

We're often told that there are no shortcuts to success -- that it's all a matter of hard work and doing what we're told. The hacking mindset takes there opposite approach: There are always shortcuts and loopholes. For this reason, hacking is sometimes perceived as cheating, or unfair, and it can be. Using social hacks to steal billions of dollars is wrong (see Madoff). On the other hand, automation seems like a great hack -- getting machines to do our work enabled a much higher standard of living, though as always, not everyone sees it that way (the Luddites weren't big fans).

Important new businesses are usually some kind of hack. The established businesses think they understand the system and have setup rules to guard their profits and prevent real competition. New businesses must find a gap in the rules -- something that the established powers either don't see, or don't perceive as important. That was certainly the case with Google: the existing search engines (which thought of themselves as portals) believed that search quality wasn't very important (regular people can't tell the difference), and that search wasn't very valuable anyway, since it sends people away from your site. Google's success came in large part from recognizing that others were wrong on both points.

In fact, the entire process of building a business and having other people and computers do the work for you is a big hack. Nobody ever created a billion dollars through direct physical labor -- it requires some major shortcuts to create that much wealth, and by definition those shortcuts were mostly invisible to others (though many will dispute it after the fact). Startup investing takes this hack to the next level by having other people do the work of building the business, though finding the right people and businesses is not easy.

Not everyone has the hacker mindset (society requires a variety of personalities), but wherever and whenever there were people, there was someone staring into the system, searching for the truth. Some of those people were content to simply find a truth, but others used their discoveries to hack the system, to transform the world. These are the people that created the governments, businesses, religions, and other machines that operate our society, and they necessarily did it by hacking the prior systems. (consider the challenge of establishing a successful new government or religion -- the incumbents won't give up easily)

To discover great hacks, we must always be searching for the true nature of our reality, while acknowledging that we do not currently possess the truth, and never will. Hacking is much bigger and more important than clever bits of code in a computer -- it's how we create the future.

Or at least that's how I see it. Maybe I'll change my mind later.

See also: "The Knack" (and the need to disassemble things)

Left brain, Right brain, and the other half of the story

2009-09-13T17:20:00.001-07:00

In my head, this post and yesterday's post on risk and opportunity are deeply connected, but logically they needed to be split apart.

The theory of the left-brain / right-brain split is that the left hemisphere of our brain handles linear, logical processing (cold logic) while the right hemisphere is more emotional, intuitive, and holistic (evaluating the whole picture instead of considering things one component at a time). Naturally, some people are more left-brain dominant while others are more right-brain dominant. This divide is discussed quite a bit elsewhere -- I recommend starting with the TED talk by Jill Bolte Taylor, a neuroanatomist whose left hemisphere was damaged by a stroke, causing her to become right-brain dominant.

I'm actually somewhat skeptical that the left-brain / right-brain split is as real as people assume, however it seems to be metaphorically correct, so for my non-surgical purposes, it's "good enough".

To me, one of the most interesting aspects of this right/left divide is that many people seem to identify strongly with one side or the other, and actually despise the other half of their brain (see here for a few examples, and even Jill Taylor seems to be doing it to some extent). This seems kind of dumb. My theory is that both halves of our brain are useful, and that for maximum benefit and happiness, we should learn how to use each half to its maximum potential.

This is where I link in to yesterday's post on Risk and Opportunity. My suggestion was to simultaneously seek big, exciting opportunities ("dream big"), while carefully avoiding unacceptable risks ("don't be stupid"). In my mind, that is the right/left divide.

The left-brain ability to carefully double-check logic and evaluate the risks is very important because it helps to protect us from bad decisions. When we imagine the kind of person who believes things that are obviously false, falls for scams, ends up joining a cult, etc, we probably picture a stereotypically right-brain person.

However, what the left brain has in cold, efficient logic, it lacks in passion and grandiosity.

When I wrote about evaluating risks and opportunities, it was as though we use a logical process when make decisions, but of course that's not actually true, nor should it be. Our actual decision making is much more emotional (and emotions are just another mental process).

The right-brain utility is in integrating millions of facts (more than the left brain can logically combine) and producing a unified output. However, that output is in the form of an intuition, "gut feeling", or just plain excitement, which can sometimes be difficult to communicate or justify ("it seems like a good idea" isn't always convincing). Nevertheless, these intuitions are crucial for making big conceptual leaps, and ultimately providing direction and meaning in our lives.

So to reformulate yesterday's advice, I think we do best when using our right-brain skills to discover opportunity and excitement, while also engaging our left-brain abilities to avoid disasters, find tactical advantages, and rationalize our actions to the world. Left and Right are both stuck in the same skull, but not by accident -- they actually need each other. (the same could probably be said for politics, but that would be another post)

Coincidentally, I just saw another good TED talk that mentions these right-brain/left-brain issues in the context of managing and incentivizing creative people. It's worth watching.

Evaluating risk and opportunity (as a human)

2009-09-12T17:44:00.000-07:00

Our lives are full of decisions that force us to balance risk and opportunity: should you take that new job, buy that house, invest in that company, swallow that pill, jump off that cliff, etc. How do we decide which risks are smart, and which are dumb? Once we've made our choices, are we willing to accept the consequences?

I think the most common technique is to ask ourselves, "What is the most likely outcome?", and if that outcome is good, then we do it (to the extent that people actually reason through decisions at all). That works well enough for many decisions -- for example, you might believe that the most likely outcome of going to school is that you can get a better job later on, and therefore choose that path. That's the reasoning most people use when going to school, getting a job, buying a house, or making most other "normal" decisions. Since it focuses on the "expected" outcome, people using it often ignore the possible bad outcomes, and when something bad does happen, they may feel bitter or cheated ("I have a degree, now where's my job!?"). For example, most people buying houses a couple of years ago weren't considering the possibility that their new house would lose 20% of its value, and that they would end up owing more than the house was worth.

When advising on startups, I often tell people that they should start with the assumption that the startup will fail and all of their equity will become worthless. Many people have a hard time accepting that fact, and say that they would be unable to stay motivated if they believed such a thing. It seems unfortunate that these people feel the need to lie to themselves in order to stay motivated, but recently I realized that I'm just using a different method of evaluating risks and opportunities.

Instead of asking, "What's the most likely outcome?", I like to ask "What's the worst that could happen?" and "Could it be awesome?". Essentially, instead of evaluating the median outcome, I like to look at the 0.01 percentile and 95th percentile outcomes. In the case of a startup, the worst case outcome is generally that you will lose your entire investment (but learn a lot), and the best case is that you make a large pile of money, create something cool, and learn a lot. (see "Why I'd rather be wrong" for more on this)

Thinking about the best-case outcomes is easy and people do it a lot, which is part of the reason it's often disrespected ("dreamer" isn't usually a compliment). However, too many people ignore the worst case scenario because thinking about bad things is uncomfortable. This is a mistake. This is why we see people killing themselves over investment losses (part of the reason, anyway). They were not planning for the worst case. Thinking about the worst case not only protects us from making dumb mistakes, it also provides an emotional buffer. If I'm comfortable with the worst-case outcome, then I can move without fear and focus my attention on the opportunity.

Considering only the best and worst case outcomes is not perfect of course -- lottery tickets have an acceptable worst case (you lose a $1) and a great best case (you win millions), yet they are generally a bad deal. Ideally we would also consider the "expected value" of our decisions, but in practice that's impossible for most real decisions because the world is too complicated and math is hard. If the expected value is available (as it is for lottery tickets), then use it (and don't buy lottery tickets), but otherwise we need some heuristics. Here are some of mine:

Will I learn a lot from the experience? (failure can be very educational)
Will it make my life more interesting? (a predictable life is a boring life)
Is it good for the world? (even if I don't benefit, maybe someone else will)

These things all raise the expected value (in my mind at least), so if they are mostly true, and I'm excited about the best-case outcome, and I'm comfortable with the worst-case outcome, then it's probably a good gamble. (note: I should also point out that when considering the worst-case scenario, it's important to also think about the impact on others. For example, even if you're ok with dying, that outcome may cause unacceptable harm to other people in your life.)

I've been told that I'm extremely cynical. I've also been told that I'm unreasonably optimistic. Upon reflection, I think I'm ok with being a cynical optimist :)

By the way, here's why I chose the 0.01 percentile outcome when evaluating the worst case: Last year there were 37,261 motor vehicle fatalities in the United States. The population of the United States is 304,059,724, so my odds of getting killed in a car accident is very roughly 1/10,000 per year (of course many of those people were teenagers and alcoholics, so my odds are probably a little better than that, but as a rough estimate it's good). Using this logic, I can largely ignore obscure 1/1,000,000 risks, which are too numerous and difficult to protect against anyway.

Also see The other half of the story

Collaborative Charity

2009-06-25T12:41:00.001-07:00

Donating money to worthwhile causes seems like a good idea, but doing it right requires knowledge, wisdom, intelligence, time, and of course money. I have at most two of those things. The traditional solution is to rely on "experts", but that has its own problems.

One of the great things about the Internet (other than the obvious) is that it enables people to collaborate in new ways, and each contribute little bits of their time and knowledge. Wikipedia is probably the best example of this, but I think it's possible to do much more. I'm not quite sure how to make this work, but I expect that in 10 years we will have much smarter "collective" systems that leverage small bits of time, knowledge, etc from large groups.

This is my first experiment in solving this problem. Actually, in some ways it's my second experiment -- a few months ago I posed a question about the "best use of money", and although it was only meant as a thought experiment, people also provided a lot of specific suggestions. That was rather encouraging.

Now I'm trying it for real -- I have a lot of ideas, but not much time, so I'm starting with the simplest solution that I could find. It may not work, but it should be interesting.

Here's how it works: I'm going to donate a bunch of money, but I want random people on the Internet to decide where it goes.

Here are the rules:

The money MUST go to an IRS recognized public charity. No exceptions.
Don't contact me. I already don't read the email I have -- I don't need more.
I've created a topic on Google Moderator where people can submit and vote on ideas. I've never used Google Moderator, but someone told me that it's good, so hopefully it works :)
Ultimately, this is just a recommendation and I may completely ignore the results if they are stupid, so don't bother spamming.
I also created a group on FriendFeed where people can submit links and discuss ideas.
I'd like to see broad support (from real people, not spam accounts) along with some evidence that it's a good idea, and perhaps endorsements from knowledgeable people.

In terms of which causes I'd like to support, I'd consider anything, but am probably most sympathetic to health, freedom, and education. In terms of solutions, I'm very skeptical of centralization, one-size-fits-all solutions, and people who are certain of the answer. I also prefer to support things that have tangible, objective outcomes (where you could say, "this money was used to purchase X" or "this money was used to fund study Y, which will be published this fall").

Make your site faster and cheaper to operate in one easy step

2009-04-17T02:21:00.000-07:00

Is your web server using using gzip encoding? Surprisingly, many are not. I just wrote a little script to fetch the 30 external links off news.yc and check if they are using gzip encoding. Only 18 were, which means that the other 12 sites are needlessly slow, and also wasting money on bandwidth.

Check your site here.

Some people think gzip is "too slow". It's not. Here's an example (run on my laptop) using data from one of the links on news.ycombinator.com:
$ cat < /tmp/sd.html | wc -c
146117
$ gzip < /tmp/sd.html | wc -c
35481
$ time gzip < /tmp/sd.html >/dev/null
real    0m0.009s
user    0m0.004s
sys    0m0.004s

It took 9ms to compress 146,117 bytes of html (and that includes process creation time, etc), and the compressed data was only about 24% the size of the input. At that rate, compressing 1GB of data would require about 66 seconds of cpu time. Repeating the test with a much larger file results yields about 42 sec/GB, so 66 sec is not an unreasonable estimate.

Inevitably, someone will argue that they can't spare a few ms per page to compress the data, even though it will make their site much more responsive. However, it occured to me today that thanks to Amazon, it's very easy to compare CPU vs Bandwidth. According to their pricing page, a "small" (single core) instance cost $0.10 / hour, and data transfer out costs $0.17 / GB (though it goes down to $0.10 / GB if you use over 150 TB / month, which you probably don't).

Using these numbers, we can estimate that it would cost $1.88 to gzip 1TB of data on Amazon EC2, and $174 to transfer 1TB of data. If you instead compress your data (and get 4-to-1 compression, which is not unusual for html), the bandwidth will only cost $43.52.

Summary:
with gzip: $1.88 for cpu + $43.52 for bandwidth = $45.40 + happier users

without gzip: $174.00 for bandwidth = $128.60 wasted + less happy users

The other excuse for not gzipping content is that your webserver doesn't support it for some reason. Fortunately, there's a simple solution: put nginx in front of your servers. That's what we do at FriendFeed, and it works very well (we use a custom, epoll-based python server). Nginx acts as a proxy -- outside requests connect to nginx, and nginx connects to whatever webserver you are already using (and along the way it will compress your response, and do other good stuff).

Communicating with code

2009-01-22T02:36:00.001-08:00

Some people can sell their ideas with a brilliant speech or a slick powerpoint presentation.

I can't.

Maybe that's why I'm skeptical of ideas that are sold via brilliant speeches and slick powerpoints. Or maybe it's because it's too easy to overlook the messy details, or to get caught up in details that seem very important, but aren't. I also get very bored by endless debate.

We did a lot of things wrong during the 2.5 years of pre-launch Gmail development, but one thing we did very right was to always have live code. The first version of Gmail was literally written in a day. It wasn't very impressive -- all I did was take the Google Groups (Usenet search) code (my previous project) and stuff my email into it -- but it was live and people could use it (to search my mail...). From that day until launch, every new feature went live immediately, and most new ideas were implemented as soon as possible. This resulted in a lot of churn -- we re-wrote the frontend about six times and the backend three times by launch -- but it meant that we had direct experience with all of the features. A lot of features seemed like great ideas, until we tried them. Other things seemed like they would be big problems or very confusing, but once they were in we forgot all about the theoretical problems.

The great thing about this process was that I didn't need to sell anyone on my ideas. I would just write the code, release the feature, and watch the response. Usually, everyone (including me) would end up hating whatever it was (especially my ideas), but we always learned something from the experience, and we were able to quickly move on to other ideas.

The most dramatic example of this process was the creation of content targeted ads (now known as "AdSense", or maybe "AdSense for Content"). The idea of targeting our keyword based ads to arbitrary content on the web had been floating around the company for a long time -- it was "obvious". However, it was also "obviously bad". Most people believed that it would require some kind of fancy artificial intelligence to understand the content well enough to target ads, and even if we had that, nobody would click on the ads. I thought they were probably right.

However, we needed a way for Gmail to make money, and Sanjeev Singh kept talking about using relevant ads, even though it was obviously a "bad idea". I remained skeptical, but thought that it might be a fun experiment, so I connected to that ads database (I assure you, random engineers can no longer do this!), copied out all of the ads+keywords, and did a little bit of sorting and filtering with some unix shell commands. I then hacked up the "adult content" classifier that Matt Cutts and I had written for safe-search, linked that into the Gmail prototype, and then loaded the ads data into the classifier. My change to the classifier (which completely broke its original functionality, but this was a separate code branch) changed it from classifying pages as "adult", to classifying them according to which ad was most relevant. The resulting ad was then displayed in a little box on our Gmail prototype ui. The code was rather ugly and hackish, but more importantly, it only took a few hours to write!

I then released the feature on our unsuspecting userbase of about 100 Googlers, and then went home and went to sleep. The response when I returned the next day was not what I would classify as "positive". Someone may have used the word "blasphemous". I liked the ads though -- they were amusing and often relevant. An email from someone looking for their lost sunglasses got an ad for new sunglasses. The lunch menu had an ad for balsamic vinegar.

More importantly, I wasn't the only one who found the ads surprisingly relevant. Suddenly, content targeted ads switched from being a lowest-priority project (unstaffed, will not do) to being a top priority project, an extremely talented team was formed to build the project, and within maybe six months a live beta was launched. Google's content targeted ads are now a big business with billions of dollars in revenue (I think).

Of course none of the code from my prototype ever made it near the real product (thankfully), but that code did something that fancy arguments couldn't do (at least not my fancy arguments), it showed that the idea and product had real potential.

The point of this story, I think, is that you should consider spending less time talking, and more time prototyping, especially if you're not very good at talking or powerpoint. Your code can be a very persuasive argument.

The other point is that it's important to make prototyping new ideas, especially bad ideas, as fast and easy as possible. This can be especially difficult as a product grows. It was easy for me to stuff random broken features into Gmail when there were only about 100 users and they all worked for Google, but it's not so simple when there are 100 million users.

Fortunately for Gmail, they've recently found a rather clever solution that enables the thousands of Google engineers to add new ui features: Gmail Labs. This is also where Google's "20% time" comes in -- if you want innovation, it's critical that people are able to work on ideas that are unapproved and generally thought to be stupid. The real value of "20%" is not the time, but rather the "license" it gives to work on things that "aren't important". (perhaps I should do a post on "20% time" at some point...)

One of the best ways to enable prototyping and innovation on an established product is though an API. Twitter is possibly the best example of how well this can work. There are thousands of different Twitter clients, with new ones being written every day, and I believe a majority of Twitter messages are entered though one of these third-party clients.

Public APIs enable everyone to experiment with new ideas and create new ways of using your product. This is incredibly powerful because no matter how brilliant you and your coworkers are, there are always going to be smarter people outside of your company.

At FriendFeed, we discovered that our API does more than enable great apps, it also reveals great app developers. Gary and Ben were both writing FriendFeed apps using our API before we hired them. When hiring, you don't have to guess which people are "smart and gets things done", you can simply observe it in the wild :)

In my previous post, I asked people to describe their "ideal FriendFeed". Since then, I've been thinking about ideas for my "ideal FriendFeed". Unfortunately, it's very difficult for me to know how much I like an idea based only on words or mockups -- I really need to try it out. So in the spirit of prototyping, I've used my spare time to write a simple FriendFeed interface that prototypes some of the things I've been thinking about. This interface isn't the "future of FriendFeed", it's just a collection of ideas, some that I like, and some that I don't. One thing that's kind of cool about it (from a prototyping perspective) is that it's written entirely in Javascript running in the web browser -- it's just a single web page that uses FriendFeed's JSON APIs to fetch data. This also means that it's relatively easy for other people to copy and change -- you don't even need a server!

If you'd like to try it out, you can see everyone that I'm subscribed to (assuming their feed is public), or if you are a FriendFeed user, you can see all of your public subscriptions by going to http://paulbuchheit.github.com/xfeed.html#YOUR_NICKNAME_GOES_HERE. The complete source code (which is just several hundred lines of HTML and JS) is here. In this prototype, I'm experimenting with treating entries, comments, and likes all as simple "messages", only showing comments from the user's friends (which can be a little confusing), and putting it all in reverse-chronological order. As I mentioned, this interface isn't the "future of FriendFeed", it's just a collection of ideas that I'm playing with.

If you're interested in prototyping something, feel free to take this code and have your way with it. As always, I'd love to see your prototypes action!

If you're the kind of person who likes to vote...

2009-01-06T12:44:00.000-08:00

Now is your opportunity!

FriendFeed was nominated for three "Crunchies". Please vote for us in all three categories:

I can't promise that your vote will end the war, fix the economy, or save the environment (that one is here), but I can promise that your vote might be counted.

Overnight success takes a long time

2009-01-04T15:31:00.000-08:00

For some reason, this weekend has seen a lot of talk about what FriendFeed is/isn't/should be doing (see Louis Gray and others). One person even predicted that we will fail.

I considered writing my own list of complaints about FriendFeed. I think and care about it a lot more than most people, so my list of FriendFeed issues would be a lot longer. I may still do that, but there's something else also worth discussing...

One of the benefits of experience is that it gives some degree of perspective. Of course there's a huge risk of overgeneralizing (someone took a picture!), but with that in mind...

We starting working on Gmail in August (or September?) 2001. For a long time, almost everyone disliked it. Some people used it anyway because of the search, but they had endless complaints. Quite a few people thought that we should kill the project, or perhaps "reboot" it as an enterprise product with native client software, not this crazy Javascript stuff. Even when we got to the point of launching it on April 1, 2004 (two and a half years after starting work on it), many people inside of Google were predicting doom. The product was too weird, and nobody wants to change email services. I was told that we would never get a million users.

Once we launched, the response was surprisingly positive, except from the people who hated it for a variety of reasons. Nevertheless, it was frequently described as "niche", and "not used by real people outside of silicon valley".

Now, almost 7 and a half years after we started working on Gmail, I see things like this:

Yahoo and Microsoft have more than 250m users each worldwide for their webmail, according to the comScore research firm, compared to close to 100m for Gmail. But Google's younger service, launched in 2004, has been gaining ground in the US over the past year, with users growing by more than 40 per cent, compared to 2 per cent for Yahoo and a 7 per cent fall in users of Microsoft's webmail.

And that probably isn't counting all of the "Apps for your domain" users. I still have a huge list of complaints about Gmail, by the way.

It would be a huge mistake for me to assume that just because Gmail did eventually take off, then the same thing will happen to FriendFeed. They are very different products, and maybe we just got lucky with Gmail.

However, it does give some perspective. Creating an important new product generally takes time. FriendFeed needs to continue changing and improving, just as Gmail did six years ago (there are some screenshots around if you don't believe me). FriendFeed shows a lot of promise, but it's still a "work in progress".

My expectation is that big success takes years, and there aren't many counter-examples (other than YouTube, and they didn't actually get to the point of making piles of money just yet). Facebook grew very fast, but it's almost 5 years old at this point. Larry and Sergey started working on Google in 1996 -- when I started there in 1999, few people had heard of it yet.

This notion of overnight success is very misleading, and rather harmful. If you're starting something new, expect a long journey. That's no excuse to move slow though. To the contrary, you must move very fast, otherwise you will never arrive, because it's a long journey! This is also why it's important to be frugal -- you don't want to starve to death half the way up the mountain.

Getting back to FriendFeed, I'm always concerned when I hear complaints about the service. However, I'm also encouraged by the complaints, because it means that people care about the product. In fact, they care so much that they write long blog posts about what we should do differently. It's clear that our product isn't quite right and needs to evolve, but the fact that people are giving it so much thought tells me that we are at least headed in roughly the right direction. I would be much more concerned if there were silence and nobody cared about what we are doing -- it would mean that we are "off in the weeds", as they say. Getting this kind of valuable feedback is one of the major benefits of launching early.

If you'd like to contribute (and I hope you do), I'd love to read more of your visions of "the perfect FriendFeed". Describe what would make FriendFeed perfect for YOU, and post it on your blog (or email post@posterous.com if you don't have a blog -- they create them automatically). Feel free to drop or change features in any way you like. Yes, technically you're doing my work for me, but it's mutually beneficial because we'll do our best to create a product that you like, and even if we don't, maybe someone else will (since the concepts are out there for everyone).

The question is wrong

2009-01-03T12:41:00.001-08:00

On "Coding Horror", Jeff Atwood asked this question:

Let's say, hypothetically speaking, you met someone who told you they had two children, and one of them is a girl. What are the odds that person has a boy and a girl?

He then argues that our intuition leads us to the "wrong" answer (50%) instead of the "correct" answer (2/3 or 67%).

However, the question does not include enough information to determine which of these answers is actually correct, so the only truly correct answer is, "I don't know" or "it depends". I skimmed though the comments on the post (there are about a million), and didn't see anyone addressing this issue (though someone probably did). They mostly argued about BG vs GB for some reason.

The reason that this question is wrong is because it doesn't specify the "algorithm" for posing the question.

If we assume that boys and girls are born with equal probability (50/50, like flipping a coin), then families with two children will have two girls 25% of the time, two boys 25% of the time, and a boy and a girl 50% of the time.

If the algorithm for posing the question is:

Choose a random parent that has exactly two children
If the parent has two boys, eliminate him and choose another random parent
Ask about the odds that the parent has both a boy and a girl

Then we can see that step two eliminated the "two boy" possibility, which leaves the 25% probability of two girls and the 50% probability of both a boy and a girl. Of course probabilities should add up to 100%, so the final probabilities are 25/75 (1/3) for two girls and 50/75 (2/3) for both a boy and girl. This is the "correct" answer described by Jeff, and it occurs because of the elimination performed at step two.

However, if the algorithm for posing the question was instead:

Choose a random parent that has exactly two children
Arbitrarily announce the gender of one of the children
Ask about the odds that the parent has both a boy and a girl

Now we're back to having a 50% probability of there being both a boy and a girl. The difference is that there was no elimination at step two, and simply announcing the gender of one of the children does not affect the gender of the other child or change the probability distribution.

The problem with the question as originally posed was that it didn't specify which of these algorithms was being used. Were we arbitrarily told about the girl, or was a selective process applied?

By the way, if we're applying a selective process, then 100% is also a possibly correct answer, because at step two we could have eliminated all parents that don't have both a boy and a girl. Likewise, all other probabilities are also potentially correct depending on the algorithm applied.

Update: Surprisingly, some people are still thinking that my second algorithm yields 2/3 instead of 1/2 (see the confused discussion on news.yc). I think part of the reason is that I was somewhat imprecise with the concept of "elimination". The second algorithm does not eliminate any of the families, but if I announce that there is a boy, that does eliminate the possibility of two girls. This is where some people are getting lost and thinking that the boy+girl probability has become 2/3. The catch is that announcing the boy also reduced the boy+girl probability by an equal amount, so the result is still the same (it eliminated either BG or GB, I don't know which, but it doesn't matter).

communication and collaboration - the big upgrade

2008-12-30T18:26:00.001-08:00

The responses to my blog are always a little surprising to me. Yesterday's post didn't have a whole lot of substance, but it did include one good product idea, which is to somehow let other people edit my posts.

Someone on news.yc was not impressed by my idea though:

I was going to disagree with those negative comments below but then read the blog and damn; this guy has a freaking ego to think people would want to edit his ramblings for him in any other way than comical..

Next, I looked at the comments that were on my blog directly. One of the first ones was from Paul Graham, and it said simply:

deniable -> deniability :-)

Well, apparently Paul Graham wants to edit my ramblings, and in a way that would make me look smarter too... I'm pretty sure that he would have just made the correction himself had it been as easy and obvious as leaving a comment, but unfortunately no blog software seems to do that, most especially not Blogger.com.

Last year, someone translated one of my posts into Chinese (and I had Google translate it back).

This all reminds me of one of the blog posts that has been trapped in my head for a long time...

It starts off with something about ants, because my house must have been built on top of a giant anthill or something, because they are continually staging giant invasions and I'm always having to set them on fire or vacuum them up or something. So I'm always thinking about ants, and ants are kind of interesting because, more so than a lot of animals, the individuals are not really viable, and the hive (or colony or whatever) is kind of like a creature of its own (yeah, I know, I'm not the first person to notice this). It even has a short term memory in the form of pheromone trails left on my floor, and I erase those memories with a paper towel and some soapy water. So the ant colony is fairly sophisticated, but each ant's behavior is relatively simple -- they are just following some simple rules and don't really comprehend why or how the colony works. They don't see the "big picture".

And that reminds me of our brains, which are built out of relatively simple neurons. Each neuron simply sums up it's inputs, and then generates an output that gets passed along to some other neurons (or something like that, I'm sure it's a huge simplification, but you get the point). Certainly no individual neuron can possibly comprehend what it's doing -- it just cranks along summing up inputs and generating outputs. The magic is in the wires, the connections among the neurons.

Individual humans aren't terribly viable animals either. They almost went extinct not that long ago (100,000 years?). However, since then we've managed to pretty much take over the entire planet and build all kinds of amazing things like airplanes, computers, and burritos. Humans started out kind of similar to other animals (but weaker and less numerous) and then became something fundamentally different. That transition occurred because we are able to communicate and collaborate like nothing else. We can communicate though both time and space. We learn from people who died thousands of years ago on the other side of the planet. Even a survivalist hunter who goes off into wilderness alone is still relying on all the training and knowledge that was passed on before the journey began.

So in many ways, the human society (or human superorganism) is kind of like the human brain -- the magic is in the connections. Significant advances have occurred when we upgraded the wiring that connects everyone. The inventions of spoken language, written language, and the printing press were all revolutionary because they enabled more sophisticated communication and collaboration.

And now I can ramble on about ants and neurons and stuff, and people all of the world can read it, and digest it, and tell me I'm an idiot, and make their own ideas, and pass them on to other people, and it all happens in a matter of minutes. As much hype and excitement as there has been around the Internet, I think that people may still be misunderestimating its importance. We are literally upgrading the wiring that drives human society.

This is also why I'm excited about things like FriendFeed. The flow of information and influence is rather fundamental to way our world works. In the past much of that information flow was slow and hierarchical. It had to pass through one of a relatively small number of tightly controlled networks and publishers. But suddenly, the information can come from anywhere, and go anywhere, and it doesn't need anyone's approval. If it's completely random, it won't work any better than a bunch of randomly wired neurons (which I assume isn't very good), but with the right wiring, everyone starts to get the right information for them, and maybe we can stop being so stupid. I'm not yet sure what this new human architecture looks like, but that's what makes it an interesting (and extremely important) problem.

I sometimes think of FriendFeed as a kind of "distributed broadcast channel", but that's just part of picture. Better collaboration, like having other people edit my blog posts, is another part. It enables each of us to do what we do best, which improves the overall system efficiency and intelligence (and more importantly, I can avoid things that I don't like doing).

Keeping with the brain anology, it's very likely that we can't even comprehend what's going on. I certainly don't. I'm just a little neuron, summing up my inputs, and then passing the result along to you.

blog, v2

2008-12-29T18:39:00.001-08:00

I haven't posted anything here in about eight months, mostly because I've just been very busy, but also because:

Blogging is too hard
I post a lot of things over on FriendFeed, which is easier, and I'm lazy (and you really should subscribe to my FriendFeed if you find anything I post here at all interesting)
I got tired of my blog posts. When I read them, there's something I don't like.

Nevertheless, I sometimes want to share something more than a FriendFeed message or interesting link, so I'm going to give it a second try.

However, I've decided that in the future my posts will be more rambling, and more pointless. I think part of what I don't like about the older posts is that they are sometimes arguing a point or something, but my real point (or my intention, at least) is just to share some kind of idea or thought, not convince anyone of anything. Also, I think this will be a lot easier to write because I can just type a bunch of words and they don't have to fit together in any particular way, and it's also a good excuse to not bother with any editing, so I should be able to crank these things out really fast.

I also have this idea to outsource the writing of my blog posts to someone, ideally everyone. The idea is that I'd write a bunch of stuff and then someone else (maybe wiki-style) would turn it into something coherent and readable. That would save me a lot of time and also provide plausible deniable when I write something that turns out to be especially stupid or offensive. But that's in the future. For now, it will just be a bunch of words that keep going until I get bored or distracted, and then I'll hit "send" :) (I'm also writing these things in Gmail since the blogger interface upsets me)

The power of links and the value of global knowledge

2008-04-23T01:30:00.000-07:00

Long, long ago, before Google, search engines evaluated and ranked web pages by considering each page in isolation, examining the size of the fonts, the contents of the meta tags, etc. In some cases, it was even possible to "hijack" another site's listings by simply cloning their HTML. Perhaps a few search engines attempted to improve on this with simple tactics such as counting the number of links to a page, but that was generally useless since it's so easy to create "fake" links in order to boost your count.

With Pagerank, Google took a very different approach. Instead of considering each page in isolation, they examined the link structure of the entire web and computed a global evaluation of that structure. In other words, they began looking at the entire forest instead of just the individual trees. Google did other things too -- Pagerank is just one of many factors, but this general approach of evaluating information in a global context is fundamental to many of the algorithms. These algorithms made it easier for Google to spot which web sites were actually important, and which were just pretenders. Of course Google isn't perfect, and people can still manipulate rankings to some extent, but it was substantially better than the old way, and good enough to form the foundation of what is now a $174 billion dollar company.

Last week I wrote about Facebook gathering similar information about people. By collecting information about people and the links between them, they can start to get a global view of the human "forest". Unfortunately, based on many of the responses, that post wasn't very well written. A lot of people focused on how annoying Facebook applications are (true), how search results limited to your friends would be useless (also true), or other things completely unrelated to my point. A few people mentioned that Facebook hasn't done anything useful with this data, which is actually a good point, but I think that has more to do with Facebook and the newness of the data than it does with the value of the data. After all, the web was around for many years before Google came along and started profitably mining the link structure.

Will Facebook ever do anything useful with the human link data? I have no idea, and it's not particularly important to me. However, I'm confident that SOMEONE will begin mining this data, and that it could ultimately be more valuable than the link data from the web. Facebook is a convenient example because they happen to have a head start on collecting the data, but others might be the first to actually profit from it. Google, in particular, is much better at data mining and also has quite a bit of human link data (from Gmail and Orkut). Microsoft+Yahoo will also have a nice data set, though I doubt that they will know what to do with it. Of course none of this data is perfectly clean and noise-free, but real data never is -- the web certainly isn't.

Facebook knows who you are, and that's worth more than you think

2008-04-17T02:20:00.000-07:00

It's very fashionable to declare that Facebook is an over-hyped fad and will never make any real money, certainly not enough to justify its insane $15 billion valuation. At first glance, it's easy to understand why some people might think it's a toy -- most of the activity there seems to involve biting, poking, and joining groups with funny names.

However, I think that assessment misses out on something very interesting: Facebook is capturing everyone's identity and relationships. Of course there's some noise caused by random friending, but by examining the larger graph as well as other details such as location, affiliations, interactions, and of course explicitly entered relationship details ("how do you know Paul?"), they can get a pretty good idea of which people are actual friends and acquaintances.

The lack of reliable identity information has always been an issue on the web. It's the reason why we don't have a useful directory of email addresses -- everyone in the directory would get bombarded by spam or other unwanted messages, and even if it did exist, how would you know which of the thousands of Adam Smiths is the one that you are looking for? Facebook has already solved this problem for a large fraction of people. It's easy to search for a name and then pick out the right person based on their picture, location, or friends. I get a lot of messages on Facebook, but unlike email, I have yet to receive any spam. That's pretty remarkable.

Perhaps a people directory doesn't seem terribly valuable, but if you can't imagine how to make money from knowing everyone's identity and trust networks, then you aren't being very imaginative. Spam and fraud are two of the biggest problems on the internet, and they are very difficult to stop because it's so easy to create new identities, and we have no good way of differentiating between real identities and fake ones. Even in "real" life, people are able to skip town-to-town, defrauding people again and again because to the people in the new town, they have a new and unknown identity.

One of the best examples of this problem on the internet is eBay. If you try to buy or sell something on eBay (especially computers or electronics, apparently), there is a very good chance that someone will try to rip you off -- just search Google for ebay scammers and you will find pages such as "How scammers run rings round eBay" and "eBay Forums: Today's Scams In Progress". Ebay has had a relatively solid lock on the auction market due to network effects, but with billions of dollars in profits, a $42 billion market cap, and 10 years of not innovating, I'm willing to bet that won't last. With reliable identity information, most of these fraud schemes would become impractical, which would obviously be a real advantage for an eBay competitor.

What else is highly profitable on the internet? Search. I doubt that anyone will ever beat Google at Google-style search, certainly not Microsoft or Yahoo, even if they do tie their horses together. The only way anyone will create something significantly better than today's Google is if they add a new and important ingredient to the mix. Many people have suggested that demographic information, or perhaps knowing what your friends have searched for will help, but I doubt it. What could work is actual, direct, human involvement by the users. In fact, it's already helping in a very limited form -- Wikipedia pages are written and edited by random people on the internet and they frequently occupy the top spots on Google (and I always click on them). Of course the problem with letting random users edit or reorder the search results is that you will quickly be overwhelmed by spam and fraud. But what if you knew who the users were and which ones you could trust?

Those are just the first few things that come to mind -- the uses of identity information are endless. Of course there's no guarantee that Facebook will actually realize any of this potential -- there were many search engines before Google, and they all fumbled the opportunity they had, but it's important to at least understand the potential for big things.

Update: This post was supposed to be about data more so than Facebook (Facebook just happens to have the data). See this post for a (hopefully) better explanation.

Ideas vs Judgment and Execution: Climbing the Mountain

2008-03-30T20:15:00.000-07:00

How much is an idea worth? Many normal people assume that ideas are valuable, and that if only they could think of one, they might be able to sell it for millions of dollars, like the Pet Rock. On the other hand, many engineers, VCs, and successful entrepreneurs claim that ideas are worthless. Paul Graham provides a sort of "proof" that ideas are worthless:

Startup ideas are not million dollar ideas, and here's an experiment you can try to prove it: just try to sell one. Nothing evolves faster than markets. The fact that there's no market for startup ideas suggests there's no demand. Which means, in the narrow sense of the word, that startup ideas are worthless.

People in the "ideas are worthless" camp usually claim that it's all about execution -- they have plenty of great ideas that just need great teams to execute on them.

I have ideas all of the time, many more than I have time for, and so I tend towards the "ideas are worthless" camp. However, there's a nagging inconsistency -- something isn't quite right.

Quoting yet again from Marc Andreessen's "Guide to Startups, part 4: The only thing that matters"

I'll assert that market is the most important factor in a startup's success or failure.
...
The product doesn't need to be great; it just has to basically work. And, the market doesn't care how good the team is, as long as the team can produce that viable product.
...
Conversely, in a terrible market, you can have the best product in the world and an absolutely killer team, and it doesn't matter -- you're going to fail.

In other words, you just need to build the right product. A mediocre team building the right product will succeed and a brilliant team building the wrong product will fail.

Isn't that a little bit like saying that having the right idea DOES matter? And if ideas are so plentiful, then why do we see great teams executing perfectly on bad ideas?

I've thought about this for a bit and realized that both camps ("ideas are valuable" and "ideas are worthless") are wrong, at least when stated so simply.

Imagine that products are mountains. To build a product, you will need to climb that mountain. Some mountains have a big pot of gold at the top, and some do not. In order to make money, you will need to pick the right mountain and then successfully climb to the top and gather up the gold. You can fail by choosing a mountain that has little or no gold at the top, or by dying on the way up.

Taking this metaphor a little further, there are also multiple paths up the mountain. According to Wikipedia, Mount Everest has fifteen recognized routes to the top. Some routes are easier than others.

Successfully executing a trip to the top of the mountain requires a certain level of technical ability -- how much will depend on the mountain and route. It also requires good judgment in order to choose the right route, or to change course when you realize that the current path isn't working out.

Judgment isn't talked about as much as execution, but it's obviously very important. A technically brilliant team, upon encountering a sheer cliff, may excitedly think to themselves, "this is the perfect opportunity to use Erlang!" (or some other fancy tech -- Erlang is just a funny example) A team with better judgment would notice that there's an easier route that goes around the other side.

Judgment also plays a critical role in choosing which mountain to climb. Our landscape of product-mountains has millions of different mountains, many of which have never been climbed. Other mountains have been attempted in the past, but the team froze on the way up, or there was no gold when they got to the top (apparently the gold flows intermittently in this analogy).

There are also people wandering around in the flat lands near the mountains. Many of these people have ideas about which mountains have gold at the top, and some of them have even drawn crude maps showing what they believe to be an easy route to the top. Inevitably, they try to sell their ideas and maps to the mountain climbers, but the climbers just brush them off and say that their ideas and plans are worthless.

Eventually, a team of climbers will discover a huge cache of gold on one of the mountains. Naturally, the people who were hanging around at the base trying to sell their ideas and plans will say, "I had that idea first! They stole my idea! I knew there was gold at the top of the mountain!"

And it's true that they had the idea, as did many other people. Ideas are plentiful. The problem is that most ideas are bad -- either there's no gold at the top of the mountain, or the ascent is too difficult with today's technology. What's valuable is the judgment to know which mountains have the gold, and the team that can get to it.

So are ideas worthless? Not quite. If a skilled climber who has successfully chosen the right mountains in the past thinks he knows the location of another gold-rich mountain, people will listen. The idea has value because it comes from someone who has a history of being right.

If the exact same idea were presented by a random person with no experience and no ability to execute, it would probably be ignored -- there's just not enough evidence that it's a good idea. If that person truly believes in their idea, they will have to prove it on their own. (The beauty of our system is that they often can, even if everyone else thinks it's a bad idea)

If someone with a history of being right also has a capable team of climbers who have demonstrated the technical skill and judgment to climb other mountains, then that is very valuable, and they will have no trouble getting their idea funded.

Summary:
Idea * Judgment * Ability * Determination * Luck = $$$

FriendFeed from the command line

2008-03-27T02:47:00.000-07:00

Sometimes, it's faster and easier to just use the command line. Thanks to the new FriendFeed API, I was able write a little script that connects my command line to my FriendFeed.

This probably would have been easier to write in Python, but bash is so awkward that it makes for a somewhat more interesting challenge. (most of this code is just dealing with image files -- the real work is done by curl)

Here you go:

#!/bin/bash
# Replace with your nickname:remote-key
# Go to http://friendfeed.com/account/api to get your remote key
USER="paulapitest:buggy696hoist"

function usage {
    echo "Usage: $0 [-t title] [-l link] [-u nickname:remotekey] [images ...]"
    exit 1
}

MAXSIZE=""
while getopts m:u:t:l: opt ; do
    case "$opt" in
        t)  TITLE="$OPTARG";;
        l)  LINK="$OPTARG";;
        u)  USER="$OPTARG";;
        m)  MAXSIZE="$OPTARG";;
        \?) usage;;
    esac
done
shift $[OPTIND - 1]

TITLE="${TITLE:-$LINK}"
TITLE="${TITLE:-$1}"

[ "$TITLE" = "" ] && usage

ARGS=("-F" "title=$TITLE" "-F" "link=$LINK" "-u" "$USER")
FILES=("$@")

for F in "${FILES[@]}" ; do
    if [ "$MAXSIZE" != "" -a -x /usr/bin/sips ] ; then
        T=`mktemp /tmp/ffshare.XXXXXX`
        sips --resampleHeightWidthMax "$MAXSIZE" --out "$T" "$F" 2>/dev/null
        F="$T;filename=$F"
    fi
    N="${#ARGS[@]}"
    ARGS[N]="-F"
    ARGS[N+1]="img$RANDOM=@$F"
done

CODE=`curl -o /dev/null -w "%{http_code}" "${ARGS[@]}" http://friendfeed.com/api/share`
if [ "$CODE" == "200" ] ; then
    echo "Shared on http://friendfeed.com/`echo "$USER" | sed -e 's/:.*//'`"
else
    echo "Failed: HTTP response $CODE"
fi

Is fragmentation bad?

2008-03-17T13:05:00.000-07:00

Imagine that you've just finished watching a movie and are in the mood to talk about it. How are you going to do that? You could chat with random, semi-anonymous people in the movie theater lobby (assume you went to a theater). You could find a community of people who are big fans of the director or the book that the movie was based on. Or, if you saw the movie with friends or family, maybe you'll discuss it with them.

Which of these options you choose will probably depend on your situation. Sometimes it's fun to hear what "random" people think. If the movie is a little more niche and you're somewhat of a connoisseur, you may not care what random people, or even your friends, think. On the other hand, going to movies is often more about shared experience than it is about the movie itself. We enjoy spending time with our friends and the movie is just something interesting to discuss.

Ultimately, a single movie may spawn millions of separate discussions among millions of different people, all in different situations and contexts.

However, there's a question that no one is asking: Isn't all that fragmentation bad? Instead of having millions of separate discussions, shouldn't we have a single, unified discussion, preferably under the control and ownership of the movie studio?

No?

I enjoy our fragmented movie discussions, and I suspect that I would hate the single, unified, shouting match that would occur if we tried to unify all of those separate discussions. This issue of unified discussion may seem a little silly, but I keep seeing it repeated in the context of blogs and other online content.

People sometimes complain that specialized communities such as news.ycombinator.com are taking the conversation away from the sites that they link to, but I go to news.yc in large part because it has an intelligent and well behaved community. That community is kind of niche -- they mostly talk about programming and startups -- but I'm interested in those same things, so I like it.

On the other hand, I occasionally read the comments on YouTube, but I would never comment there myself. It's too random and belligerent for me.

Most recently, this issue of fragmentation has been brought up a lot when debating FriendFeed. One of things that people really love about FriendFeed are the comments -- it's the only place on the web where I can easily share and discuss things with my actual friends (to see what this looks like, view the things I've shared or the things that I've liked or commented on).

Although comments are one of our most popular features, they are also our most controversial feature. If you believe that there should only be a single, unified discussion, then the extra fragmentation caused by FriendFeed will seem like a step in the wrong direction. In fact, not only is there a separate discussion on FriendFeed, there may be hundreds of separate discussions within FriendFeed on the very same topic or link (because different people are sharing the link, and different people have different friend groups).

I, for one, enjoy the fragmentation. It's important to understand that FriendFeed isn't trying to replace the specialized communities on places such as news.yc, or the screaming hordes on YouTube. We're creating a third option: discussion with friends. It may not be for everyone, and that's fine, but many people really like it, including people who would never participate in broader forums such as TechCrunch or YouTube.

Good news, everyone!

2008-02-25T22:08:00.000-08:00

FriendFeed is officially launching! (and also announcing our funding)

See Louis Gray, VentureBeat ("Friendfeed, the best software for conversations"), and TechCrunch for more detailed reviews.

The most important thing to understand about new products and startups

2008-02-17T03:22:00.000-08:00

First, a quote from Marc Andreessen's "Guide to Startups, part 4: The only thing that matters"

If you ask entrepreneurs or VCs which of team, product, or market is most important, many will say team.
...
Personally, I'll take the third position -- I'll assert that market is the most important factor in a startup's success or failure.

Why?

In a great market -- a market with lots of real potential customers -- the market pulls product out of the startup.

The market needs to be fulfilled and the market will be fulfilled, by the first viable product that comes along.

The product doesn't need to be great; it just has to basically work. And, the market doesn't care how good the team is, as long as the team can produce that viable product.
...
Conversely, in a terrible market, you can have the best product in the world and an absolutely killer team, and it doesn't matter -- you're going to fail.

Mark's blog post did not immediately resonate with me, because his terms are somewhat different from the way I think. After all, how great is your product if nobody wants it? How great is your team if they persist in building something that nobody wants?

However, his main point has stayed in the back of my mind since then, and I'm continually reminded of how important it is, and how often I see people who clearly don't get it.

In my mind, there's really two points. One: You can take the smartest, most experienced, most connected, most brilliant people in the world and have them build the most stunningly designed and technically advanced product in the world, but if people don't want it, then you will fail. This is roughly what happened with the Segway, for example.

Perhaps that seems a little discouraging. After all, if really smart people with all the right resources can fail, then what hope is there for the rest of us? Perhaps success is random, and maybe startups are more like the lottery than we'd like to admit.

I don't believe that's true though. There is an optimistic way of understanding my first point, and that's my second point: Even if you aren't the smartest person around, and your product is kind of ugly and broken, you can still be very successful, if you just build the right product. YouTube and MySpace are both fine examples of this.

But if your team is so great, why aren't they building the right product? Simply put, they have the wrong attitude. Firstly, they overestimate the importance of their own skills. Engineers think that success is all about fancy technology and complex engineering (hello Google). Designers think that success is all about beautiful design. MBAs think that success is all about knowing the right people, or spreadsheets, or something. If you have especially smart or successful people, then this problem could be even worse, because then the team is also likely to be arrogant and overconfident, which makes them less likely to question these assumptions or the value of their own skills.

It's easy to find examples of this wrong attitude. When Google acquired YouTube, many people inside the company were flabbergasted, "But they have no technology!?" They didn't understand that you only need enough technology to make the product work. Any more and you probably have the wrong priorities. I regularly see similar complaints about Facebook, MySpace, and a lot of other popular sites. Similarly, people will often complain that MySpace or even Google has "no design" or "bad design". Again, they have enough design (or the right design) to work for their users.

So what's the right attitude? Humility. It doesn't matter how smart and successful and qualified you are, you simply don't know what you're doing. The good news is that nobody else does either, though some are foolish enough to think that they do (and that's why you can beat them).

What is the humble approach to product design? Pay attention. Notice which things are working and which aren't. Experiment and iterate. Question your assumptions. Remember that you are wrong about a lot of things. Watch for the signals. Lose your technical and design snobbery. Whatever works, works.

MySpace is a great example of this. I'm pretty sure that their custom profile page layouts were an accident. They didn't know enough to properly escape the text that people put on their profiles, and that allowed their users to start including arbitrary html and css in their pages. This is a common bug, and most people would have fixed the bug and that would have been the end of it (really great engineers wouldn't have had the bug in the first place). But they did something smarter. They noticed that the feature was popular and found a way to preserve it. The result is mostly ugly, but it's extremely popular.

There are many other accidental inventions besides MySpace, but it's important to understand that "accidental" isn't the same as "random". There are clues all around us, we just need to watch more closely.

For web based products at least, there's another very powerful technique: release early and iterate. The sooner you can start testing your ideas, the sooner you can start fixing them.

I wrote the first version of Gmail in one day. It was not very impressive. All I did was stuff my own email into the Google Groups (Usenet) indexing engine. I sent it out to a few people for feedback, and they said that it was somewhat useful, but it would be better if it searched over their email instead of mine. That was version two. After I released that people started wanting the ability to respond to email as well. That was version three. That process went on for a couple of years inside of Google before we released to the world.

Startups don't have hundreds of internal users, so it's important to release to the world much sooner. When FriendFeed was semi-released (private beta) in October, the product was only about two months old (and 99.9% written by two people, Bret and Jim). We've made a lot of improvements since then, and the product that we have today is much better than what we would have built had we not launched. The reason? We have users, and we listen to them, and we see which things work and which don't.

Find the gradient, then follow it.

Ultra-immersive, long-form video games from the past or future

2008-01-27T19:27:00.000-08:00

As our technology and understanding of nature improves, we are living longer, and many predict that this trend will continue to the point that humans will become nearly immortal. When confronted with the possibility of living for hundreds, thousands, or even millions of years, most people express several concerns: "Will I still be able to retire at age 65?", and "Won't that get boring after a while?"

Fortunately, technology is also improving it other areas (and not just more deadly weapons). Video games, for example, are getting quite sophisticated. As the graphics and other interfaces improve, video games become increasingly immersive and involved, and we begin to feel as though we are really inside the game. As this trend continues, will we get to the point that the games feel so real that we become completely immersed and forget about the outside reality?

So what will million-year old people do to manage their boredom? Perhaps they will play long, complicated, multi-player, fully-immersive video games. If your regular life lasted millions of years, occasionally spending a hundred years playing some fancy game might seem reasonable. Perhaps you would play-out your character's entire life span, from birth to death, in one "sitting". In order to really feel the experience and keep the game authentic, you would of course make it so that everyone playing would forget that it was just a game (though maybe some people would try to cheat).

The obvious question: Is that the future, or the past?

Once we eliminate the certainty of our perceived reality, then how can we justify our certainty of anything else? Accepting true reality, whatever it may be, requires letting go of everything specific.

Building a great team

2008-01-07T13:01:00.000-08:00

Unless you happen to be really great at everything, it's very important to build a well matched team of people who have complimentary skills and can work well together. Unfortunately, that's much easier said than done, and most startups really struggle to find the right people. That's why I'm excited to announce another great addition to the FriendFeed team. Check out my post on the FriendFeed blog to understand why this is a big announcement.

Should Gmail, Yahoo, and Hotmail block Facebook?

2008-01-03T17:23:00.000-08:00

Apparently Facebook will ban you (or at least Robert Scoble) if you attempt to extract your friend's email addresses from the service.

Automated access is a difficult issue for any web service, so I won't argue with their decision -- it's their service and they own you.

However, when I signed up for Facebook I gave them my Gmail address and password, using their find friends feature:

It was very helpful -- I didn't think that I would know anyone on Facebook, but it turns out that I knew hundreds of people.

However, Gmail's Terms of Use seems to prohibit this:

You also agree that you will not use any robot, spider, other automated device, or manual process to monitor or copy any content from the Service.

Facebook can also import contacts from Yahoo and Hotmail. Yahoo TOS says:

You agree not to access the Service by any means other than through the interface that is provided by Yahoo! for use in accessing the Service.

And Hotmail TOS says:

In using the service, you may not:
...
Use any automated process or service to access and/or use the service (such as a BOT, a spider, periodic caching of information stored by Microsoft, or "meta-searching")

So the question is, should Gmail, Yahoo, and Hotmail block Facebook (or close the accounts of anyone who uses Facebook's "friend finder") for violating their Terms of Use?

Twas the night before Festivus...

2007-12-22T00:40:00.000-08:00

And all through the FriendFeed Global International World Headquarters, not a creature was coding, not even Bret..

But before that, we were all very busy...

And not just busy posing for pictures -- we're also creating new products! I'm thrilled to announce the release of our latest platform initiative, AirGrievance.

Brilliantly wrong

2007-12-14T03:28:00.000-08:00

Being a little bit wrong is easy, but it's much more interesting to be completely wrong.

Google Blogoscoped had an entry yesterday about my statement that, whenever possible, everything should have undo.

The comments on that story had the usual mix of no-so-great usability suggestions (putting options and settings everywhere), and a few good ideas, but it also had one comment that was so remarkably wrong that it brightened my whole day:

While an Undo feature could be useful, isn't this just coddling people who should otherwise be paying closer attention to what they are doing? A mistake is a mistake, and people need to learn to live with the consequences of the mistakes they make.

This comment may have been a joke, but I really like this "tough love" approach to usability, because in just a few words it perfectly captures the exact opposite of what we should be doing.

To design great products, we must truly empathize with our users, and understand that if they are having problems using our products, is more likely our fault, not theirs. This isn't the same as the disdainful or patronizing attitudes too often expressed by engineers and other technical people. Our users aren't dumb, they just have better things to do than waste time understanding poorly designed software.

As for "undo", in general, the more we can lower the costs of making mistakes, the faster we can move. This applies to everything from interfaces (I type fast because I have a backspace key) to societies (people are more likely to start new companies in cultures that are accepting of failure).

Is there more to life than money?

2007-12-09T15:10:00.000-08:00

Whenever there is a discussion about joining a startup vs working at regular job, someone will defend the boring 9-5 job by saying, "there is more to life than money".

These people are, of course, right. There is plenty of evidence that happiness is only loosely correlated with wealth. People seem to derive a much greater sense of satisfaction from good relationships, and having a sense of purpose and meaning in their life. Money matters too, but not as much once the basic need for food and shelter are addressed, and those aren't big issues for most people deciding between joining a startup or a big company.

Furthermore, most people who do join startups will never experience a huge payday. Google distributed billions of dollars to thousands of employees, but that was truly exceptional. Even moderately successful startups that eventually sell for $50 million dollars or so will only make a couple of people rich.

If all you care about is money, I doubt that joining a startup is the right way to go. You'll probably make a lot more at a hedge fund, or by becoming a lawyer, or something like that.

Maybe it now seems like I'm defending the boring 9-5 job, but it's actually just the opposite. Those 8 hours/day are a huge chunk of of your waking life, and don't forget that you'll probably also spend a few hours preparing, commuting, and "unwinding". Even worse, bad or boring jobs can sap our energy, so that at the end of the day all we feel like doing is sitting in front of the tv.

That's no way to live, if you can help it. Why surrender such a huge chunk of your life just to get some money? For some people, that's the only option, but for those fortunate enough to be smart and educated, there's a better way.

Instead of throwing away your "working hours", why not make every minute count? Why not find work that you can actually enjoy, work that's fun and meaningful?

Of course that's easier said than done, but it's not impossible.

For reasons that will have to be left for another post, the structure and systems in big companies tend to make work meaningless and life unpleasant, at least for me. Of course smaller companies can be awful too, but they have a greater potential to be good.

If your job isn't working for you, if it isn't making you happy and energized, then why are you still there? I can't guarantee that you'll find something better, but perhaps you should at least try something new. Don't just give up on life.

I truly enjoy writing code. I love creating new products and features. I like getting feedback from users and finding ways to solve their problems. I like the game of business. I like helping people. Of course there is always some amount of unpleasant work that must be done, but that can be contained (my rule is that work should be no more than 10% awful).

That's why, even though I don't financially need to work, I choose to work (and end up staying up until 4am pushing new code). It's why we created FriendFeed, to have a great place to work, a place where we can build great products and have happy users. Of course I'd also like to earn a few billion dollars, and I plan to make all of our employees very wealthy, but that's more like a bonus. I don't believe that you can be happy and make great products and treat your users right if all you think about is money.

There's more to life than money.

My first Google pay-stub (including note from Heather). Money isn't everything, but this zero-dollar check did make me a little nervous.

There's no such thing as a "social network"

2007-12-02T13:40:00.000-08:00

Of course there are many things that people call "social networks", but that's more confusing than enlightening. The "social network" aspect of these products is just a mechanism, not the purpose, and their purposes are often more different than similar. It's like saying that something is a "Javascript". Many websites use Javascript, but hopefully the Javascript is only there in order to accomplish some other purpose. LinkedIn and Twitter, for example, both use Javascript and social networking mechanisms, but clearly they are very different products and have very different uses. Facebook likes to call itself a "social utility", but to me that's just as meaningless.

That said, there is something "social networky" that unites these otherwise-dissimilar "social networking" sites. The two features that seem to define the social network aspect of a product are:

Some kind of page or profile for each user. The contents of this page vary wildly, but it always includes a name and often a picture.
"Friend" links among the profile pages, which may or may not be bi-directional. These "friends" only sometimes correspond to real-life friends.

Those two features alone are very simple but also very useless. Real products need more functionality in order to somehow deliver value to their users. It is this other functionality that defines the real purpose of a product, not the "social network", which exists only to enable or enhance the core purpose.

So what ARE the purposes of these many "social networking" products? (in random order)

Enable people to send messages to their friends. This functionality basically defines email, which can be considered one of the earliest "social networks", though most email systems lack browsable profiles. Annoyingly, many other social networking products reimplement this function, with mixed results.
Enable people to send messages to non-friends. There are two aspects to this feature: discovery (finding the people) and control (preventing unwanted messages). Email is weak on both aspects, and for many people, this is one of the most useful features of products such as Facebook and LinkedIn (I can easily find and contact people that I don't know).
"Live" address/phone book. Sites like Facebook have fields for phone, email, etc, and since everyone maintains their own page, the information is generally up-to-date (vs the bad-old-world of everyone having to broadcast "please update your address books" every time they move).
Less private communication. Email and IM are often too private. Life would be much less interesting if all of our conversations took place one-on-one in closed rooms. Part of what makes parties, offices, and other social environments interesting is that we can observe other people interacting, overhear conversations, and often join-in. Features such as the "wall" create a kind of public or semi-public email, allowing our friends to overhear conversations. Twitter is often used as a public IM/chat.
Passive communication about my life. I don't want to interrupt or spam my friends every time I have some little bit of news, take some new pictures, or get a random thought, but I have no problem blogging or Twittering those things, or uploading to photo sharing sites (which all have social features now). Similarly, I enjoy seeing this information from my friends (but I wouldn't want them calling me on the phone to tell me about their lunch every day).
Passive sharing of non-personal content. I'll occasionally see an interesting or amusing web page and want to share it with friends. I used to put these things into my IM status, but now I add them to my FriendFeed. Email, Twitter, Facebook, and Digg/Reddit are also sometimes used for this purpose (though obviously Digg is less "friend" oriented).
Self expression. This is most prominent on MySpace, where user profiles are highly customizable and can have embedded music (which starts playing as soon as I visit the page). This behavior is very similar to decorating your house or bedroom in "real life" -- there seems to be some human instinct to define our identity through decorations, fashion, music, art, etc. Self expression is an element of all products which have browsable profiles, but on most sites identity is expressed more in terms of content (favorite movies on Facebook, job history on LinkedIn, tweets on Twitter, etc).
Background information. If I meet someone new (or am about to meet them), I sometimes checkout their Facebook or LinkedIn pages to learn more about their background, see if we have common friends, etc. Obviously this also ties in with self-expression and identity in more social settings -- people "friend" each other after meeting and their MySpace page (or whatever) becomes part of that "first" impression.
Dating. One of the earliest and still most popular uses of sites like MySpace, Facebook, Orkut, etc. It's less "explicit" than on dating sites, but by putting themselves online, people can see and be seen, plus get background info, see who their common friends are, etc. The Facebook newsfeed even tells us when friends break-up (or start dating).
Jobs. Job hunting and hiring are essentially the "professional" analog of dating and seem to work in somewhat similar ways.
Finding old friends. By enabling friend-of-friend and school class browsing, it's much easier to locate old friends, and therefore people do more of it. This also lets people track down old friends without appearing crazy (they can "bump" into them on classmates.com, or whatever -- no private detective needed).
Keeping in touch with "unclose" friends (related to "passive communication about my life"). It's fun to know what's going on with people who we've known in the past, even if we aren't close friends. Profile browsing and newsfeeds make this easier. This can also make unclose friends into closer friends, as we may coincidentally be in the same city at the same time, be attending the same events, or just start chatting.
"Micro-socializing" (I need a better term here). A lot of wall posts, poking, winking, commenting, Twittering, etc can go into the category. The comments on FriendFeed generally fall into this category as well -- someone shares a link or posts some photos, then they and their friends will end up chatting about it. This is somewhat analogous to the traditional "water-cooler" conversation.
Word-of-mouth recommendations. Shared links, news, Yelp reviews, some Twitters, and the like provide us with information about what our friends are doing. Word-of-mouth can be very powerful since we typically trust our friends much more than random people. The new Facebook "Beacon" advertising system is trying to use this effect to sell things.
General amusement for bored people. Clicking around on friends and friends-of-friends and even random people's pages can be kind of interesting. As sites provide more content (such as the links and videos on FriendFeed) or games (many popular Facebook apps are games) there will be even more of this.
Publishing. This especially applies to blogs, twitter, flickr, and the like, but being able to subscribe to the content produced by our favorite authors and artists as well as friends and family is very powerful for both the producer and the consumer.
Group sharing and socializing. This was traditionally done via mailing lists and that's the model behind Google/Yahoo Groups, but of course Facebook, Orkut and the like have have some basic group functionality too, and Ning seems to specialize in it by enabling each group to create their own social network.

This list is certainly not complete, but I'm way over my 30 minute limit... I look forward to seeing your suggested additions. I'll probably post an improved list at some point.

I also hope to explain how my company, FriendFeed, fits into all of this. It definitely has social aspects (the word "friend" is right in the name!), but it certainly isn't a social network in the style of MySpace or Facebook.

We all have tunnel vision

2007-11-17T04:22:00.000-08:00

Supposedly people can hold about seven "items" in their mind at any one time. I was never sure what that meant -- what qualifies as an "item"? I recently realized that it means our brains have roughly seven "registers", similar to the registers on a microprocessor. These registers don't store much information, they really just hold pointers to something in longer term memory.

Here's the important realization: There's a million important things going on in the world and in our lives, but we're really only aware of seven of them. This means that we all have a very narrow and limited understanding of the world and our own lives. The seven things on our mind all seem very important, while everything else is just kind of forgotten.

We must be very careful about what gets loaded into those seven registers. I like to read reddit occasionally, but I've found that it can be a little dangerous. Some days it is filled with bad news -- my seven registers get filled with scary and depressing things, and it feels like the world is crumbling.

On the other hand, I can go for a walk outside and after a little while I begin to notice how nice the trees are, the various smells of nature, the construction progress on the house down the street, and other pleasant (for me) things, and the world seems like a really nice place.

This seven register limitation also makes people very subject to manipulation. If you can control what is getting loaded into their attention, you can largely control what they think and how they feel. For example, if people keep talking about Iran and how scary they are and debating what to do about them, then pretty soon Iran will seem like the biggest, scariest problem in the world, and no solution will seem too extreme. The truth is that there are probably 100 more important problems, but it won't seem that way because all seven registers are loaded with the same topic. The subject of the debate is more important than the content.

At any one time, a million things are going wrong, the world is falling apart. At any one time, a million things are going right, the world just keeps getting better. Those are both true statements, but one will feel much more true depending one which seven things are loaded into your attention now.

The same thing can happen inside of a company. We can easily become obsessed with one issue or threat, and it ends up taking all of our attention and energy even though it's not the most important problem. And because it seems like the only problem, our response is often completely irrational. I have some amusing Google anecdotes, but I'm almost out of time for this post, so they will have to wait...

How do we deal with this? I'm not certain, but the first step is to simply be aware of it. Take a break. Go for a walk. Try to clear out your attention. Maintain balance. Keep four good things in mind at all times (make a list). We shouldn't ignore the bad things, but we mustn't allow them to overwhelm us, because that's what will happen if the bad completely pushes out the good.

27 minutes.

Update: To be clear, the actual value "seven" is not important -- it's just an idea that many people are familiar with. The point is that our attention has a very limited capacity, and therefore we are always missing out on the bigger picture.

Back to blogging again, I hope

2007-11-15T02:21:00.000-08:00

I have about a 1000 posts queued up in my brain, but these things take forever to write, and now that I'm working again, I don't have forever to waste. So, in the spirit of "Good enough" is the enemy of "At all", I'm going to start blogging again, but with an absolute limit of 30 minutes per post. If after 30 minutes I'm not happy with what I've written, I'll delete the post and move on for the day. If anything seems less thought-out or more confusing and inflammatory, this is why (and I didn't mean whatever you thought I meant, I meant something intelligent and non-insulting).

Meanwhile, check out My FriendFeed.

The next big adventure

2007-10-01T01:41:00.000-07:00

We have a new startup: FriendFeed!

It's still early beta, and I'll write more about the product and company as soon as I get a chance. Until then, read the NY Times article and checkout our "about" page.

Quick: Read this if you ever store password data

2007-09-17T14:35:00.000-07:00

Thomas Ptacek has a good post explaining why your password storage scheme is probably bad.

His post has a lot of words, so I'll summarize:

Storing passwords in plaintext is obviously bad (most developers already know this)
Using a salted-hash such as MD5 or SHA-1 isn't much better. (too easy to brute force)
Your other clever password storage ideas are probably bad too.
Use bcrypt instead. It was created by real cryptographers for just this kind of thing.

Here's bcrypt for Java. It has a nice simple API:


// Hash a password for the first time
String hashed = BCrypt.hashpw(password, BCrypt.gensalt());

// Check that an unencrypted password matches one that has
// previously been hashed
if (BCrypt.checkpw(candidate, hashed))
    System.out.println("It matches");
else
    System.out.println("It does not match");

I'm sure other popular languages have similar libraries.

16GB iPhone "SLR" includes high-quality 8MP camera

2007-08-30T21:23:00.000-07:00

And it still fits in my pocket:

Reality, page 191

2007-08-07T04:47:00.000-07:00

...replaces all objects of belief with one single thing: reality itself. We believe only in this universe. We don't believe in the afterlife. We don't believe in the sovereignty of nations. We don't believe in money or power or fame. We don't believe in our idols. We don't believe in our positions or our possessions. We don't believe we can be insulted, or that our honor or the honor of our family, our nation or our faith can be offended...

We just believe in reality. Just this.

...doesn't ask you to believe in anything you cannot confirm for yourself. It does not ask you to memorize any sacred words. It doesn't require you to worship any particular thing or revere any particular person. It doesn't offer any rules to obey. It doesn't give you any hierarchy of learned men whose profound teachings you must follow to the letter. It doesn't ask you to conform to any code of dress. It doesn't ask you to allow anyone else to choose what is right for you and what is wrong.

...is the complete absence of belief. ...is the complete lack of authority. ...tears away every false refuge in which you might hide from the truth and forces you to sit naked before what is real. That's real refuge.

Reality will announce itself to you in utterly unmistakable ways once you learn to listen. Learning to listen to reality, though, ain't so easy. You're so used to shouting reality down, drowning it out completely with your own opinions and views, that you might not even be able to recognize reality's voice anymore. It's a funny thing, though, because reality is the single most glaringly obvious thing there is... Yet we've forgotten how to recognize it.

What does it mean to own a "right"? Was Gandhi a thief?

2007-08-05T16:12:00.000-07:00

From the Wikipedia entry on Gandhi's 1930 Salt March:

The British monopoly on the salt trade in India dictated that the sale or production of salt by anyone but the British government was a criminal offense punishable by law. Salt was readily accessible to coastal area dwellers, but instead of being allowed to collect and use it themselves for free, they were instead forced to purchase it from the colonial government.

In other words, the British government owned the right to produce and sell salt in India.

And another page says:

On April 5, 1930 Gandhi and his satyagrahis reached the coast. After prayers were offered, Gandhi spoke to the large crowd. He picked up a tiny lump of salt, breaking the law. Within moments, the satyagrahis followed Gandhi's passive defiance, picking up salt everywhere along the coast. A month later, Gandhi was arrested and thrown into prison, already full with fellow protesters.

Was Gandhi stealing salt from the British government, or was he simply breaking the law which gave the British the exclusive right to produce salt?

Since Gandhi wasn't actually taking the salt away from the British, and the ocean contained a practically infinite supply of salt (so it can't be claimed that he was taking it from anyone else), I'd argue that he was not stealing anything.

Did Gandhi steal the legal "right" to produce salt? Not really. A right has no substance or physical reality, so it is difficult to outright steal one (you could perhaps trick someone into signing a contract transferring the right, but clearly that's not the case here).

Gandhi's actions DID decrease the value of the British right to produce salt, but many actions could have that effect. For example, he could have convinced the people of India to switch to a lower salt diet, thereby decreasing the demand for salt and the value of the British monopoly. I don't think that anyone would call that stealing.

In fact, I've never seen anyone claim that Gandhi was a thief. It seems like a somewhat silly argument.

So what's my point? Owning a "right", which is a form of "Imaginary Property", is not the same as owning real property. If someone violates your exclusive rights, they may be breaking the law, but they are not stealing. To claim otherwise is silly and dishonest.

Update: Some people have interpreted this post as meaning that I'm opposed to Imaginary Property, or that infringing on other's legal rights is ok. That is not the case at all -- IP can serve an important and beneficial role in society. My only point is that Imaginary Property is not the same as Real Property, and that infringing on someone's legal rights isn't the same as theft. Pretending that they are the same has caused a lot of unnecessary trouble.

The first thing that you need to understand about humans

2007-08-02T16:30:00.000-07:00

Humans aren't rational -- they rationalize. And I don't just mean "some of them" or "other people". I'm talking about everyone. We have a "logic engine" in our brains, but for the most part, it's not the one in the driver's seat -- instead it operates after the fact, generating rationalizations and excuses for our behavior.

I first realized this fact several years ago when I read something about "split brain" people. These people have had their corpus collosum, the structure which connects the left and right brain hemispheres, cut or damaged, leaving the two halves of the brain largely unable to communicate with each other. I don't recall where I read that, but a quick Google turned up this similar article:

"A split-brain patient shown a photograph of Hitler only in the right hemisphere, for example, might exhibit facial expressions indicating anger or disgust. But when asked to explain those emotions, the patient will often invent an answer, such as 'I was thinking about a time when someone made me angry.'" (Newberg and D'Aquili 2001, p. 23)

Kenneth Heilman offers another, more concrete example, writing about the research of Dr. Michael Gazzaniga and his colleagues. In one experiment, they showed sexually suggestive pictures to a woman with callosal disconnection, flashing them only on the left half of a screen so only her right hemisphere could perceive them. The woman giggled and blushed, but when asked why she was doing so, she replied that she was thinking of something embarrassing (Heilman 2002, p. 129).

Are these people lying? In one sense of the word, perhaps; but it seems clear that there is no conscious intent to deceive. Rather, researchers have concluded, what is happening is that the right hemisphere, upon seeing an image with strong emotional connotations, generates the appropriate response. However, due to the callosal disconnection, it cannot transmit the associated sensory data to the left hemisphere and its language centers. The left hemisphere perceives a change in the body's state, but does not know why - and so it "fills in" the missing details, fabricating a logical reason for the emotional reaction. This happens at a subconscious level, so that the person genuinely believes the verbal explanation they provide. In the language of psychology, this filling-in process of unconscious invention is called confabulation.

The important realization is that this process of "confabulation" is not limited to people with brain damage -- everyone does it -- people with "split brain" are just a little more obvious.

It's tempting to think that "most people" have this problem, but that you or I are different, to think that our actions are all logical. That is a mistake -- denying the truth is irrational and dangerous. By accepting that people are fundamentally irrational, we can deal with ourselves and others in a more rational and effective manner. We can learn to manage our irrational selves (somewhat). If, however, we insist that all of our actions and feelings are rational, then we will never be able to deal with them honestly, and are more likely to cling to irrational beliefs and limitations. If we are going to be honest, then we must admit the possibility that everything we know and believe is, in fact, incorrect.

The fascinating (and easy to read) book "Influence: Science and Practice" has some great examples and explanations of how people actually work. Here's a great quote about the effect of attractiveness, and how unaware people are of their true decision making process:

A study of the 1974 Canadian federal elections found that attractive candidates received more than two and a half times as many votes as unattractive candidates (Efran & Patterson, 1976). Despite such evidence of favoritism toward handsome politicians, follow-up research demonstrated that voters did not realize their bias. In fact, 73 percent of Canadian voters surveyed denied in the strongest possible terms that their votes had been influenced by physical appearance; only 14 percent even allowed for the possibility of such influence (Efran & Patterson, 1976). Voters can deny the impact of attractiveness on electability all they want, but evidence has continued to confirm its troubling presence (Budesheim & DePaola, 1994).

A similar effect has been found in hiring situations. In one study, good grooming of applicants in a simulated employment interview accounted for more favorable hiring decisions than did job qualifications—this, even though the interviewers claimed that appearance played a small role in their choices (Mack & Rainey, 1990). The advantage given to attractive workers extends past hiring day to payday. Economists examining U.S. and Canadian samples have found that attractive individuals get paid an average of 12-14 percent more than their unattractive coworkers (Hammermesh & Biddle, 1994).

Equally unsettling research indicates that our judicial process is similarly susceptible to the influences of body dimensions and bone structure. It now appears that good looking people are likely to receive highly favorable treatment in the legal system (see Castellow, Wuensch, & Moore, 1991; and Downs & Lyons, 1990, for reviews). For example, in a Pennsylvania study (Stewart, 1980), researchers rated the physical attractiveness of 74 separate male defendants at the start of their criminal trials. When, much later, the researchers checked court records for the results of these cases, they found that the handsome men had received significantly lighter sentences. In fact, attractive defendants were twice as likely to avoid jail as unattractive defendants. In another study —- this one on the damages awarded in a staged negligence trial -- a defendant who was better looking than his victim was assessed an average amount of $5,623; but when the victim was more attractive of the two, the average compensation was $10,051. What's more, both male and female jurors exhibited the attractiveness-based favoritism (Kulka & Kessler, 1978).

Other experiments have demonstrated that attractive people are more likely to obtain help when in need (Benson, Karabenic, & Lerner, 1976) and are more persuasive in changing the opinions of an audience (Chaiken, 1979).

Understanding our irrational nature is also critical to product design. How can you expect to make something for humans if you don't understand how they think or make decisions?

Another book, "The Culture Code", has some great examples based on the author's experience helping companies improve the design and marketing of their products.

The people at Chrysler had indeed asked hundreds of questions; they just hadn't asked the right ones. They kept listening to what people said. This is always a mistake. As a result, they had theories about moving the Wrangler in multiple directions (more luxurious, more like a traditional car, without removable doors, enclosed rather than convertible, and so on) with no clear path to follow.
...
When I put groups of consumers together, I asked them different questions. I didn't ask them what they wanted in a Jeep; I asked them to tell me about their earliest memories of Jeeps.
...
I returned to those wary Chrysler executives and told them that the Code for Jeep in America is HORSE. Their notion of turning the Wrangler into just another SUV was ill advised.
...
The executives weren't particularly moved. After all, they had vast research that told them consumers said they wanted something else. ... I asked them to test my theory by making a relatively minor adjustment to the car's design: replacing the square headlights with round ones. Why? Because horses have round eyes, not square ones.
...
They tested the new design and the response was instantly positive. Wrangler sales rose and the new "face" of the Wrangler became its most prominent and marketable feature. In fact, the car's logo has incorporated its grille and round headlights ever since.

Using surveys and focus groups to design your product is a great way to produce boring and mediocre products that are the same as everything else.

Another great quote from "The Culture Code" connects us back to the "split brain" example that I started with:

What do Americans look for in a car? I've heard many answers when I've asked this question. The answers include excellent safety ratings, great gas mileage, handling, and cornering ability, among others. I don't believe any of these. That's because the first principle of the Culture Code is that the only effective way to understand what people truly mean is to ignore what they say. This is not to suggest that people intentionally lie or misrepresent themselves. What it means is that, when asked direct questions about their interests and preferences, people tend to give answers they believe the questioner wants to hear. Again, this is not because they intend to mislead. It is because people respond to these questions with their cortexes, the parts of their brains that control intelligence rather than emotion or instinct. They ponder a question, they process a question, and when they deliver an answer, it is the product of deliberation. They believe they are telling the truth. A lie detector would confirm this. In most cases, however, they aren't saying what they mean.

The reason for this is simple: most people don't know why they do the things they do. In a classic study, the nineteenth-century scientist Jean-Martin Charcot hypnotized a female patient, handed her an umbrella, and asked her to open it. After this, he slowly brought the woman out of her hypnotic state. When she came to, she was surprised by the object she held in her hand. Charcot then asked her why she was carrying an open umbrella indoors. The woman was utterly confused by the question. She of course had no idea of what she had been through and no memories of Charcot's instructions. Baffled, she looked at the ceiling. Then she looked back at Charcot and said, "It was raining."
...
Even the most self-examining of us are rarely in close contact with our subconscious. ... Therefore, we give answers to questions that sound logical and are even what the questioner expected, but which don't reveal the unconscious forced that precondition our feelings. This is why polls and surveys are so often misleading and useless.

Of course this understanding is important to more than just product design. It's critical to anything involving humans, which is everything involving us. How does it apply to your life and work? Are you still making decisions with the assumption that we are rational?

Quick: All actions should have 'undo'

2007-06-25T22:45:00.000-07:00

I just slipped and added a misspelled word to the Firefox dictionary. Guess what the process is for undoing that mistake?

Some apps "fix" this problem by prompting with an "are you sure" type dialog before performing the action. That's very annoying, and it doesn't always work since I might hit space or enter or something and accidentally "ok" it. (this happens to me a lot when the dialog pops up while I'm typing)

I really like the solution used in Gmail:

After an action is performed, state what the action was and give an opportunity to undo, but in a non-intrusive way. More people should copy this approach, including Gmail, which needs it on "Send" (this will require adding a short delivery delay, like 10 sec, but it's worth it).

Quick: Maybe you can hack computers

2007-06-23T16:33:00.000-07:00

But do you know how to hack people? It's only code.

Quick: Job culture

2007-06-23T12:37:00.000-07:00

Here's a good link about the "Job culture":

The cultural assumption of a true free enterprise system would be: “Individuals are responsible for their own lives and labors. They trade as equals, but are beholdin' to nobody.”

Free enterprise isn't anything like big-corporate capitalism. We've been told the two are equivalent, but that's just another bit of cultural brainwashing.

Think about it. Job holders by definition aren't capitalists. Job holders, no matter how well paid they might be, function merely as the servants of capitalists, just as medieval serfs functioned as the servants of lords. They are beholdin'. They function in a climate of diminished responsibility, diminished risk, and diminished reward. A climate of institutional dependency.

...

The daily act of surrendering individual sovereignty – the act of becoming a mere interchangeable cog in a machine – an act we have been conditioned to accept and to call a part of “capitalism” and “free enterprise” when it is not – is the key reason why the present Job Culture is a disaster for freedom.

James Madison, the father of the Bill of Rights, wrote:
“The class of citizens who provide at once their own food and their own raiment, may be viewed as the most truly independent and happy. They are more: They are the best basis of public liberty, and the strongest bulwark of public safety. It follows, that the greater the proportion of this class to the whole society, the more free, the more independent, and the more happy must be the society itself.”

Madison was speaking specifically about independent farmers, but he was also a believer in the independent entrepreneur – and for the same reasons.

Madison (and his like-minded friend Jefferson) knew that people who are self-sufficient in life's basics, who make their own decisions, whose livelihood relies on their own choices rather than someone else's, are less likely to march in lockstep. Independent enterprisers are far more likely to think for themselves, and far more capable of independent action than those whose first aim is to appease institutional gods.

Living in the Job Culture, on the other hand, has conditioned us to take a “someone else will deal with it” mentality. “I'm just doing my job.” “The boss makes the decisions.” “I'm just following orders.” But if someone else is responsible for all the important choices in life, then we by definition, are not.

I reminds me of my favorite part of Paul Graham's essay on "Why to not not start a startup":

16. A job is the default

This leads us to the last and probably most powerful reason people get regular jobs: it's the default thing to do. Defaults are enormously powerful, precisely because they operate without any conscious choice.

To almost everyone except criminals, it seems an axiom that if you need money, you should get a job. Actually this tradition is not much more than a hundred years old. Before that, the default way to make a living was by farming. It's a bad plan to treat something only a hundred years old as an axiom. By historical standards, that's something that's changing pretty rapidly.

...

Now we look back on medieval peasants and wonder how they stood it. How grim it must have been to till the same fields your whole life with no hope of anything better, under the thumb of lords and priests you had to give all your surplus to and acknowledge as your masters. I wouldn't be surprised if one day people look back on what we consider a normal job in the same way. How grim it would be to commute every day to a cubicle in some soulless office complex, and be told what to do by someone you had to acknowledge as a boss—someone who could call you into their office and say "take a seat," and you'd sit! Imagine having to ask permission to release software to users. Imagine being sad on Sunday afternoons because the weekend was almost over, and tomorrow you'd have to get up and go to work. How did they stand it?

I have a plan to fix this, but it's a little difficult to describe because people have a hard time thinking outside of the surf mindset, the assumptions that underlie the "job culture". It will take some time.

Three types of ideas - bad ones are often the best

2007-06-22T18:33:00.000-07:00

Product ideas can be divided into three categories:

Obviously good ideas that are very difficult to implement. Efficient cold-fusion, flying cars, and a lot of other sci-fi ideas fall into this group.
Obviously "good" ideas that seem possible but haven't happened yet. Video phones and HDTV were in this category for a long time. I think this happens when people get excited about technology and overestimate the benefits (and possibly underestimate the cost). I just don't care that much about having a video phone.
"Bad" ideas. Many of these ideas are truly bad, but some of them will in hindsight turn out to have been very good ideas. I put them in the same category because they are difficult to distinguish without the benefit of hindsight. Some examples are the personal computer ("why would anyone need a computer?"), Google ("there are already too many search engines, and besides, search engines don't make money"), and Blogger ("can't you just use Geocities, and besides, are there really that many people with something worth saying?"). More recent (and still controversial) examples include Facebook and Twitter.

I expect that a lot of people will argue with the fact that I've grouped "truly bad" ideas together with "seemed bad but were actually very good" ideas. Everyone thinks that they can tell which ideas are the good ones, but observation suggests that they can't. I do believe that some people are better at picking winners than others, but at best they have maybe 50% accuracy. I've seen a lot of very smart people (such as at Google) be very wrong about these things.

For example, I remember when the "upload videos and put them on the web" incarnation of Google video was first being developed. Nearly everyone inside of Google, including me, was very skeptical that anything worthwhile would ever be uploaded. The common prediction was that it would all be movies and porn. Of course there was some of that, but the skeptics were completely wrong about the lack of worthwhile content. Uploaded video is one of the most important developments of the past several years. Unfortunately Google Video was burdened with an incredibly bad upload process (it included installing a windows client to do the upload!), and YouTube, which was started AFTER Google Video launched, took over. I believe that mistake was partially due to the negative expectations.

Here's my point: The best product ideas are often found in the "bad ideas" category!

If you're a super-genius researcher looking to dedicate your life to a really important problem that you may never solve, then the category one "cold fusion" ideas may be a good option. If you've found some clever way to dramatically reduce the cost of implementing a category two idea (like Skype did to the video phone), then that might be a good option. However, the real "low hanging fruit" is likely to be found with the category 3 "bad ideas". You'll have to deal with annoying skeptics and haters who say that you are wasting your time, but sometimes you'll create something hugely important, or at least moderately successful (and they won't).

Realizing that good ideas and bad ideas are often nearly indistinguishable, there are a few more lessons to be learned:

Instead of endlessly debating whether an idea is good or not, we should find faster and cheaper ways of testing them. This is one of the reasons why open systems such as the Internet or market economies develop faster than closed systems, such as communism or big companies -- individuals and small groups are able to build new things without getting approval from anyone.
Your product idea is generally worthless, because to most people it's indistinguishable other bad ideas. Very few people will want to buy (or steal, or take for free) your idea because they already have their own bad ideas which they like better. (There are other reasons as well, but this is one of them. The way to make your idea valuable is to demonstrate that it's not bad by building a real product -- this also demonstrates your ability to execute.)

Obviously there is a lot more to say about these two points, and I haven't done enough to explain or justify them right now. Maybe in another post...

Not getting health insurance is a really bad idea

2007-06-22T17:44:00.000-07:00

Seriously. If you can possibly afford insurance, then there's no excuse for not having it. I'm especially thinking of young people doing a startup and not buying insurance in order to "save money". That's a really dumb optimization, and suggests that you aren't very good at making trade-offs. A quick search shows that in California you can get a decent plan for $77/month. If you can't find $77/month, then maybe you aren't ready to do a startup.

I'm reminded of this issue right now because I just passed someone in SF laying face down on the pavement, not moving. His bicycle was nearby and I'm guessing that he had just been hit by a car. I hope that he does ok, but it didn't look good. (people were already helping him, and the police were just a few cars back, so I didn't stop)

Even if you are young and healthy, bad things can still happen. My brother was perfectly healthy until he got cancer at age 33. When my daughter was in the hospital, it cost $12,000/day -- she was there for almost 3 months.

The simple fact is that the health care system in the US isn't setup for people who don't have insurance. If something bad happens, you don't want to be one of those people. I'm not looking to debate our system, so please don't bother. Right now, the system is the way it is.

My point is simple: Do what you can to get insurance. Now.

Quick: Unintended consequences

2007-06-16T00:59:00.000-07:00

For some reason, I like reading about unintended consequences. It seems that reality always has more depth and complexity than we would like to admit. This is part of the reason why people who claim to have the answers, generally don't.

This comes to mind because I was just reading a little about the effects of pesticides on plants:

Pesticide residues in plants are regulated to protect human health. They are measured when plants enter the market. However, this procedure does not consider that pesticides modulate secondary metabolism in plants when they are applied. The question arises whether it is conceivable that pesticide-induced changes in the chemical composition of plants influence human health. The examples resveratrol, flavonoids, and furanocoumarins indicate that plant phenolics may have subtle effects on physiologic processes that are relevant to human health. These effects may be beneficial, e.g., due to the inhibition of the oxidation of macromolecules and platelet aggregation or by their pharmacologic properties. Depending on concentration and specific chemical composition, however, plant phenolics may also be toxic, mutagenic, or cancerogenic. The consequences of a modulation of plant phenolics on human health are complex and cannot be predicted with certainty. It may be that the modulation of plant phenolics at the time of application and not the usually low level of pesticide residues at the time of consumption is critical for human health.

Put another way, pesticides can actually alter the nutritional properties of plants by removing environmental stress, stress which would have caused the plants to produce nutrients that may be important to humans. I hadn't considered that.

Quick: Big screen development

2007-06-15T01:30:00.000-07:00

This is a good setup for working together, and the price of 1080p televisions is dropping.

Quick: DBE

2007-06-14T13:37:00.001-07:00

People sometimes ask what "Don't Be Evil" really means. Eventually I'll write down my thoughts, but for now this excellent post from Seth Godin explains a good part of it:

Let me be really clear, just in case. If you think that the world would be a better place if everyone owned a handgun, then yes, market handguns as hard as you can. If you honestly believe that kids are well served by drinking a dozen spoonfuls of sugar every morning before school, then I may believe you're wrong, but you should go ahead and market your artificially-sweetened juice product. My point is that you have no right to market things you know are harmful or that lead to bad outcomes, regardless of how much you need that job.

Along the way, “just doing my job,” has become a mantra for blind marketers who are making short-term mistakes in order to avoid a conflict with the client or the boss. As marketing becomes every more powerful, this is just untenable. It’s unacceptable.

If you get asked to market something, you’re responsible. You’re responsible for the impacts, the costs, the side affects and the damage. You killed that kid. You poisoned that river. You led to that fight. If you can’t put your name on it, I hope you’ll walk away. If only 10% of us did that, imagine the changes. Imagine how proud you’d be of your work.

There are a lot of worthwhile things in the world -- why waste your life working on something evil?

The dogmatic programmer - when software becomes religion

2007-06-13T13:09:00.000-07:00

Do you believe that there is only one "right way" to do things? Do you ignore evidence that other solutions may also work? Do you dismiss the possibility that there are reasonable trade-offs to be made between the "right way" and some other way? (do you leave angry comments on my blog?)

Do you think I'm kidding? Read these comments about GET vs POST. There's no question that POST is the "right" way and generally safer, but sometimes it's annoying. Even though GET isn't "supposed" to work, it often can be made to. Don't believe me? Google does billions of dollars a year in GET based transactions in the form of CPC ad clicks (which can cost over $50/click). A lot of other things work this way too, including most non-ajax webmail (for changing read/unread state, at the very least).

If you're writing missile guidance software or something then, please, please, please do things the right way -- it's better safe than sorry, right?

However, if you're writing little web apps, then there's a good chance that you can bend a lot of the rules and produce a better product sooner. I'd rather have something imperfect but useful and popular than something "perfect" but unfinished and unused. Sometimes, it's better sorry than safe.

When is it safe to bend the rules? Just ask yourself, "What's the worst that could happen?" If the worst case isn't too bad, then you're probably ok. Minor bad things will happen no matter what, so it's often better to put your energy into general problem recovery, rather than imagining that you can simply avoid all problems. (this is especially true on the web)

Quick: Vibe

2007-06-13T12:25:00.000-07:00

Seth Godin on "the vibe":

Have you ever been at a banquet or in a boutique or at a concert or a meeting or a company where the vibe was incredibly positive?
...
If vibe is so important, why does it sound flaky to worry about it? Who's in charge of the vibe at your place? Could it be better? A lot better?

This is a very real and under-appreciated part of work. It was one of the first things I noticed when I started at Google after working at Intel -- the place just felt so much more alive. Now it's something that I watch for when visiting startups -- my guess is that upbeat and energetic companies will outperform the ones that feel oppressive and hopeless.

Quick: Wolves

2007-06-11T14:09:00.000-07:00

A great story from Steve Yegge. It sounds like Steve recently realized that there are wolves at Google. Can you see the wolves in your organization? How are you protecting yourself? (or are you being eaten alive?)

"Have you ever noticed that all our folklore about wolves has certain themes and patterns to it? I've never met a wolf, but I give them fair credit for being wily creatures. I'm a big man, not one to fear a big dog's bite, but I would give a wolf a wide berth, because they're reputed to be so crafty. And crafty, friends, I am not. A craftsman, yes, but I realize now that I am more sheep than wolf.

"The landowner in my tale is a good, affable fellow; I could see it within minutes of meeting him, and I stand by him today still. He's a man with a clear vision and a good heart, and he hired crew members who, for all our talents, know nothing of wolves or wizards.

"The problem with setting up an untended, untrained flock of sheep is that there are wolves aplenty, and as sheep are not adequately prepared to recognize or deal with wolves, they will all eventually be eaten. This seems obvious to us. Nearly as obvious is that this idea applies equally to government, with the minor change that the wolves are savvy politicians and the sheep are incompetent ones. Eventually the sheep are all eaten, so our assumption is generally that the government is filled entirely with wolves — an assumption that has proven to be essentially axiomatic, across time and nation.

"What's far less obvious is that the idea applies outside flocks and governments, with the same consequences. If you set up any organization of saintly do-gooders, with no politics anywhere in sight, then the presence of a single master politician can go unrecognized for long enough for every single sheep to be eaten and replaced with a wolf. At some point the shepherd looks down and perceives that he now has a flock of wolves, and he wonders how it happened. But a shepherd rarely witnesses the process while it's in motion, because the wolves are so sneaky.

"I was being eaten alive on my mansion project, and yet I was utterly oblivious to anything but the need to survive, which due to various odd circumstances required working without sleep and eating marshmallows without respite, apparently until either I was dead or forever happened, whichever came first.

Email blackmail is unnecessary

2007-06-09T18:30:00.000-07:00

From Slashdot:

Under the guise of fighting spam, five of the largest Internet service providers in the U.S. plan to start charging businesses for guaranteed delivery of their e-mails. In other words, with regular service we may or may not deliver your email. If you want it delivered, you will have to pay deluxe.

It's tempting to think that such schemes, although distasteful, are necessary to fight spam. They are not.

A good reputation system can be used to provide reliable delivery to high-volume, non-spamming senders, and it can be done without punishing non-profits, mailing lists, startups, or others who can't afford to pay extra delivery fees.

How does that work? Brad Taylor of Google published a paper about the Gmail reputation system, which is quite effective (pdf or html).

... spammy domains get nicely clustered around a reputation of zero. We know these domains are spammy because if our users disagreed with us, they would unmark spam on them and the reputation would no longer be zero. Nonspam domains are more loosely clustered in the 90-100 range. There ends up being a smattering of domains in be- tween. Many of the messages in the in-between category are from legitimate bulk senders, and the lower than 90 percent reputation is a reflection of less-than-ideal sending practices. But not all bulk senders are the same. Some have very high SPF and DomainKey reputations. For example, eBay's DomainKey reputation is 98.2, enough to be whitelisted by Gmail. This is the "holy grail" of bulk sending, and it is all done without requiring any payment or extra effort on the part of the sender other than just having good mail hygiene.

Unfortunately Google does not currently publish this reputation information, though I'm hopeful that they will eventually. (but I know that the team there is already hard at work on a number of other important projects)

Optimizing everything: some details matter a lot, most don't

2007-06-08T00:01:00.000-07:00

Optimization is something that all engineers and product designers (all people, really) should understand. However, their words and behavior suggest that most either don't understand it, or haven't really internalized the lessons of optimization. (and by the way, I'm talking about CS-style "make it better" optimization, not "make it optimal" optimization)

Like everything, optimization is a complex topic, so I'm going to simplify it into three easy steps:

Find the problems ("profiling")
Sort the problems according to benefit-of-fixing/cost-of-fixing ("ROI")
Fix them, in the order of highest benefit/cost to lowest

Most programmers probably think of optimization primarily in terms of CPU profiling, but it really applies to everything. (and in fact, this post is about the general process of optimizing, especially for things other than code -- for specific cases of hardcore game engine optimization or whatever it might be totally wrong)

For example, people who continually "pinch pennies" (perhaps driving around town to save a few cents on gas) but then waste thousands of dollars on something stupid like a fancy car or credit card interest are probably not optimizing their spending very well. They are putting most of their effort into low reward activities (saving a few cents on gas) while losing much more money in other places. Big, dumb, Dilbertesque companies are also notorious for this kind of thing, cutting back on pencils while running super-bowl commercials, or something.

This is also the kind of activity that I was talking about in my post titled "Wasting time on things that really don't matter". It's not that there is zero difference between two slightly different ways of testing for the empty string, it's just that your product probably has 100,000 more serious problems, and so time spent worrying about the empty string style is time NOT spent fixing those more important problems. Put another way, fix all of the other problems first, then debate the empty string.

Of course you may enjoy debating the empty string (and in fact there are some interesting arguments), and that's fine. The zeroth step in optimization is to decide what you are optimizing for. If you're looking to maximize the amount of time spent on endless debates, or perhaps code "beauty", then maybe the empty string debate really is the highest benefit/cost activity (because the benefit is so high). If, on the other hand, you are trying to maximize the success of your product, then the empty string debate probably has negative value and should be strongly avoided.

The empty string comparison is obviously a small detail, and it's tempting to think that we should not bother with such small details, but that too is a huge mistake. Another important lesson of optimization is that some small fraction of the details will determine the majority of the outcome. This is commonly referred to as the 80/20 or 90/10 rule, stating for example that "90% of the time is spent in 10% of the code", but it could just as easily be 99/1 or something like that (and for code that has already been well optimized, this 80/20 rule may not hold -- it's just a heuristic). In code it may be just a few lines in the "inner loop", or perhaps it's the interest rate on your mortgage ("what's a few percent?"). I once profiled a server at Google and discovered that something like 90% of the time was being spent on a single line of code: a call to sscanf (repeated millions of times) that someone had snuck into the scoring code (that's the 99999/1 rule).

Determining which details matter a lot (and which don't) is also the key to designing successful products. In the classic slashdot iPod post from 2001, they summarized the newly released iPod thusly: "No wireless. Less space than a nomad. Lame."

History suggests that those weren't the important details, since it's 2007 and there still isn't an iPod with wireless (not until the 29th, at least), and most iPods have less than the original 5GB of storage. Apple did, however, get a few very important details right: it's easy to get your music onto an iPod, it holds "enough" music (unlike the early flash players), and it fits easily into your pocket (unlike the early disk-based players).

So how do we determine which details are important? Observation. (and some judgment, but I think that's mostly formed via observation) If you're optimizing an application, observe which resources are being exhausted or causing the delay (it's not always CPU -- it's often disk seeks), and then observe where they are being consumed. If you're trying to optimize money, first observe where most of the money goes to and comes from. To optimize a product design, observe which things are most "useful", and which are most painful or annoying (waiting or thinking, usually).

If you get all of the important details right, you'll be so far ahead of everyone else that the other details probably won't matter. But remember, your time is finite, so time spent on unimportant details is time not spent on important details! Most engineers are very detail oriented people, and this is good because they can really focus on the important details, and it is bad because they can really focus on the unimportant details! I suspect that learning to tell the difference will be worth your time.

By the way, to get even more details right, decrease the cost side of the benefit/cost equation so that you can do more things in less time (by finding easier solutions, avoiding endless debates, etc).

Java running faster than C

2007-06-03T04:05:00.000-07:00

Note: A lot of people seem to be taking this post to be the "Ultimate C vs Java shootout". It's not. Performance is a very complex topic. My only real point is this: Java (which used to be slow) has reached the class of "fast languages". For the majority of applications, speed is no longer a valid excuse for using C++ instead of Java.

I just saw this page comparing the performance of several languages on a simple Mandelbrot set generator. His numbers show Java being over twice as slow as C, but then I noticed that he's using an older version of java and only running the test once, which doesn't really give the JVM a chance to show off.

I quickly hacked up the code to run 100 iterations 3 times and then used my standard "go fast" flags (there may be better flags, but I'm lazy). Here are my results:


$ java -server -XX:CompileThreshold=1 Mandelbrot 2>/dev/null
Java Elapsed 2.994
Java Elapsed 1.926
Java Elapsed 1.955

$ gcc -O8 mandelbrot.c
$ ./a.out 2>/dev/null
C Elapsed 2.03
C Elapsed 2.04
C Elapsed 2.05

C still wins on the first iteration, but Java is actually slightly faster on subsequent iterations!

Obviously the results will be different with different code and different machines, but it's clear that the JVM is getting quite fast.

This test was run with Java 1.6.0-b105 and gcc 4.1.2 under Linux 2.6.17 under Parallels on my 2.33GHz Core 2 Duo MacBook Pro. Here is the hacked up code: Java and C.

For extra fun, I also tried running the JS test using the Rhino compiler:


$ java -cp rhino1_6R5/js.jar -server -XX:CompileThreshold=1 org.mozilla.javascript.tools.shell.Main -O 9 mandelbrot.js 2>/dev/null
JavaScript Elapsed 21.95
JavaScript Elapsed 17.039
JavaScript Elapsed 17.466
JavaScript Elapsed 17.147

Compiled JS is about 9x slower than C on this test. If CPU speed doubles every 18 months, then JS in 2007 performs like C in 2002.

Update: A few more cpp flags have been suggested. -march=pentium4 helps a little, but it's still slower than Java.


$ gcc -O9 -march=pentium4 mandelbrot2.c
$ ./a.out 2>/dev/null
C Elapsed 1.99
C Elapsed 1.99
C Elapsed 1.99

Adding -ffast-math puts C in the lead, but I'm not sure what the downside is. The gcc man page says, "This option should never be turned on by any -O option since it can result in incorrect output for programs which depend on an exact implementation of IEEE or ISO rules/specifications for math functions." That sounds like an optimization that Java might not use.


$ gcc -ffast-math -O9 -march=pentium4 mandelbrot2.c
$ ./a.out 2>/dev/null
C Elapsed 1.66
C Elapsed 1.67
C Elapsed 1.67

Update: Several people have claimed that the performance difference is due to fputs (including the top rated comment on reddit, aumusingly). That is not correct, at least not on my computer. I tried replacing the print calls with a trivial function (but with side-effects), and it actually helped the Java more than the C:


C Elapsed 1.88
Java Elapsed 1.554

Many people have pointed out that '-O8' is more than enough 'O' levels. I know, and I don't care -- it's just as good as '-O3' or whatever.

Wasting time on things that really don't matter

2007-06-01T03:26:00.000-07:00

It's easy to waste a lot of time debating things that aren't very important. Maybe that's fun if you don't have anything better to do, but when you're actually trying to accomplish something it can be deadly.

This post on "coding horror" is a great example of this phenomenon, or more specifically, the comments are a great example. At least half of the comments are from people who are debating which of these two lines of code are better:


if (s == String.Empty)
if (s == "")

Here's my answer: It's not important!

It's easy to get dragged into these debates because we all have opinions (and the one form really is better!), and of course the more time is spent talking about the issue the more important it seems.

Here's my trick for killing these stupid debates: Let's list all the issues and problems that we are facing right now. Now, where does this issue rank on that list? Is it in the top 10? The top 100? The top 1000? Are we spending more time talking about this one minor issue than we are spending working on any of our top ten issues? (You don't necessarily have to make the list -- just suggest that it exists and then guess where on the list the issue appears)

Sometimes that works. Sometimes it brings a little perspective.

And yes, this misallocation of time (aka "worrying about dumb stuff") is probably a top 10 productivity killer.

Update: Here's a good link (via news.yc) which offers one explanation for why people often spend the most time on the least important issues:

Parkinson shows how you can go in to the board of directors and get approval for building a multi-million or even billion dollar atomic power plant, but if you want to build a bike shed you will be tangled up in endless discussions.

Parkinson explains that this is because an atomic plant is so vast, so expensive and so complicated that people cannot grasp it, and rather than try, they fall back on the assumption that somebody else checked all the details before it got this far. Richard P. Feynmann gives a couple of interesting, and very much to the point, examples relating to Los Alamos in his books.

A bike shed on the other hand. Anyone can build one of those over a weekend, and still have time to watch the game on TV. So no matter how well prepared, no matter how reasonable you are with your proposal, somebody will seize the chance to show that he is doing his job, that he is paying attention, that he is *here*.

Update Two: Wasting time is fine, and even inane discussions of the best way to compare empty strings can be interesting. My real point is this: if we are actually trying to get something done, then we need to kill this stuff, or it will kill productivity. If, on the other hand, we're ok being in time-wasting-mode, then silly debates are probably harmless enough. I picked on the blog comments above not because there is anything wrong with them per-se (what are blogs for, if not inane debate?), but because it reminded me a lot of the silly debates that I've seen on work mailing lists (where they do cause harm).

Amazingly bad APIs

2007-05-21T00:46:00.000-07:00

I actually don't hate the Java language (at least it's killing off c++), but every time I use it I'm blown away by how hostile the APIs are.

For example, let's say that we want to scale an image file. That's a fairly common thing to do, so it's probably only one line of code, right? Wrong. It turns out that they haven't even bothered to include a reasonable API for doing this. Instead, you have to take the awkward APIs that they do provide, and then write about 50 lines of code to do what you really want. This article that explains what the "right" way is.

Once the image is scaled, you'll want to save the smaller version too. Luckily, that's only a few lines of code. Of course that's only if you're happy with the default settings -- what if you want to use a different JPEG quality level? Here are the docs -- see if you can figure it out (the solution involves 4 additional classes).

For comparison, here is some Python code which loads an image, scales it, and then saves it at a non-default quality level:

from PIL import Image

i = Image.open("/tmp/c.jpg")
i.thumbnail([220, 133], Image.ANTIALIAS)
i.save('/tmp/c-thumb.jpg', quality=90)

Doing the same thing in Java requires closer to 100 lines, not because the language is bad, but because the APIs are terrible.

How did this happen? My theory is that the people who made the Java APIs just sat in a room making stuff up (and possibly drawing UML diagrams). Meanwhile, the Python people were actually writing software and creating libraries to make their life easier.

There is a lesson in this: If you are building a platform, you should also be writing applications for that platform, and the platform should be designed to make life very simple for those apps. It's my understanding that Ruby on Rails came out of a process such as this (they extracted the common tasks out of their web apps). If you just sit around staring at the clouds while writing your APIs, you'll probably end up with something bad.

I also have a second theory about what happened to Java: it attracted people who like to write piles of code filled with endless abstractions. When I worked at Google, one of the engineers on our Java mailing list suggested that we ban the keyword 'new'! I was puzzled by this suggestion, but after a little research I learned that one of the hot fashions in Java is to use factories instead of 'new' (not that factories are bad, but "sometimes good" does not imply "always best"). Better yet, you don't reference the factories directly (not enough abstraction), instead you have some framework that injects the factories into your classes according to what is written in an XML configuration file. In this way, you can make your code perfectly unreadable, but more importantly, it is very abstract.

Update: Several people have interpreted this post to mean that I'm opposed to the factory pattern. That is, of course, silly. Factories, when the situation calls for them, are fine. For example, the python code above is using a factory. What harms code is the blanket application of the factory pattern in every possible situation. To put this another way, Tabasco sauce is delicious on eggs, but I still don't put it on ice cream. (but don't think I haven't tried!)

Update Two: It's not about Java, really! My point here is that your APIs should emerge from the needs of real applications, and that they should make common tasks super-easy. Bad APIs are common in many languages, Java just happens to be the language that I'm using right now.

Why didn't I simply hunt down+install some third-party image library? Because that's a hassle too (probably more work than copying code off of some web page), and all I want is to generate nice looking thumbnails.

Flash, Silverlight, and JavaFX all look nice, but what I really want is better file upload

2007-05-11T02:09:00.000-07:00

One of the most annoying limitations when writing web apps comes from the <input type="file"> form element, which is the standard way of uploading files.

Here are a few of the problems with <input type="file">:

You can only select a single file per input element, and there is no way to select entire folders.
Web browsers just pop-open the standard file open dialog, which is often not very good. For example, the dialog probably just shows a list of files, not thumbnails. This can make it difficult to pick the correct file if they all have names like IMG_3485.JPG, and it's therefore easy to upload the wrong file.
There is no way to actually read the data locally without first sending it to the server and then having the server send it back.
There is no way to do any local processing on the data, such as scaling images to make them smaller.

Flash apparently solves the multiple file problem (kind of), but it's my understanding that it still has all of the other problems (I'd love to be wrong about this though).

Java applets can actually do most of these things if they are signed and the user agrees to allow the applet access to their filesystem. Unfortunately, Java applets don't actually work, and after 11 years of not working, I'm not expecting that to change. Facebook has a nice image uploader applet that lets you browse your photos locally and then scales them before uploading to make the whole thing go fast. The only problem, other than the annoying "do you want to trust this applet" prompt, is that it crashed my browser the second time I tried it.

To me, the great thing about flash is that it fills in basic capabilities missing from the browser, such is video, sound, camera and microphone access, and they did it with a reasonably small and stable browser plugin that I'm not afraid to allow onto my computer. A few months ago, I accidentally upgraded one of my computers to Flash 9 -- I don't remember how exactly, I just remember that it was simple and non-eventful.

Here's my idea: I want to create a small, reliable browser plugin that does for file uploads what flash does for audio and video. It should work with all the modern browser/os combinations, and install with minimal effort and fuss (browser restarts avoided if possible). It would be free and open source, but hosted in a common location for easy install.

Popular Ajax libraries such as jQuery, YUI, and Dojo can add support in such a way that people who don't have the plugin just get the regular browser upload control (and maybe a link to install the plugin), while those who do have the plugin get a better ui. Clearly, getting people to install a new plugin isn't easy, but once any site convinces them, their experience automatically gets better on all other sites that support the plugin. There isn't a chicken&egg problem because the plugin simply provides an enhancement to the already existing (but very poor) browser functionality, so sites can easily add support without fear of losing people who don't have the plugin (unlike building a whole app in silverlight, or whatever).

I'd like to hear from anyone with experience writing browser plugins. How difficult is the technology side of this? (making it small, reliable, and stable -- crashing browsers is not ok) Also, are you interested it writing this plugin? I'm serious about this, and willing to pay quite a bit of money to make it happen.

Here are a few more details on how I imagine the plugin working:

It would not have any in-browser UI -- it would simply be an object that can be called from javascript and that would pop open a nice file browser with thumbnails and all that (similar to windows explorer, perhaps). It would also provide upload-related utility functions, such as image resizing. Since the file access is read-only and limited to what the user selects, security shouldn't be a big problem, assuming of course that the plugin implementation is secure (and it must be).

Here is some JS illustrating how I imagine using this plugin (UploadHelper is the object provided by the plugin):

// Have the user choose files or directories to upload.
// Returns undefined if user clicks Cancel. Doing this with a
// callback instead of a return value would also be ok.
var files = UploadHelper.chooseFiles(
              {filter:'*.jpg,*.gif,*.png', allowDirectories:true});
if (files) {
  var s = 'You selected: ';
  for (var f in files) {
    var file = files[f];
    s += (file.name + " (" + 
          file.imageinfo().width + "x" +
          file.imageinfo().height + ", original size = " + file.size);

    files[f] = file.scaleImage({maxwidth:100, maxheight:100, quality:5});
    s += " scaled size = " + files[f].size + ") ";

    // If I could somehow insert this image on the current web page
    // too, that would be really awsome.
  }
  alert(s);
  
  // using jquery XMLHTTP wrapper
  $.ajax({
    type: "POST",
    url: "/upload",
    data: UploadHelper.encode(files, 'multipart/form-data'),
    contentType: 'multipart/form-data'
  });
}

var files = UploadHelper.chooseFiles({filter:'*.txt'});
if (files) {
  // We can also read the content of the selected file
  // The ability to get chunks of a file also means that it's possible
  // to restart failed uploads, and to chunk large files into multiple
  // POSTs.
  for (var f in files) {
    var file = files[f];
    var d = file.getData(0, 50); // get the first 50 bytes
    if (file.size > 50) d += "...";
    alert(files[f].name + ": " + d);
  }
}

Update:

Several people have suggested that I should instead try to get this feature added to web browsers or (better yet) the web browser standards. That's a fine long-term solution, but realistically it will take years to happen. I would rather have an immediate (months, not years) fix in the form of a plugin, and then let the web browsers add it as a standard feature on their own schedule. This is somewhat similar to the path followed by XMLHTTP.

Sriram Krishnan of Microsoft makes a few good points:

The standard file open dialog in windows already has a thumbnail view. Unfortunately, this feature is hidden under a completely unremarkable icon and despite having used windows for many years, I've never seen it before today. This is a great example of why simply adding a feature isn't good enough -- it also needs a good ui or else it may as well not even exist. Furthermore, not existing seems to be the status of the thumbnail view on OSX -- hopefully that will be fixed in 10.5.
This feature request could be implemented in Silverlight. That's great news! MS should write and release a little Silverlight control that provides these functions to JS. I would be very hesitant to write (or rewrite) my apps in Silverlight, but if I could simply embed a little Silverlight module the same way that I would a flash mp3 player, then I'd definitely use that. This would be a very effective way for MS to get people started installing and using Silverlight (start with something small and optional).

Silverlight sounds pretty nice. Google or Adobe had better have an open alternative on the way, or else MS may be able to recapture the application platform.

Update Two:
I believe that within a few months we will have a few reasonable solutions to this problem!

2007-05-09T01:46:00.000-07:00

When people don't know how to change something, they often start searching for a way to justify failure, rather than thinking about how they could try doing something different to make it work.
...
If we pretend that you can teach anyone anything, we'll find out where it's not (yet) true. But if we think that when someone isn't learning it means they can't be taught, no one will even try.
...
If you take the attitude that anything is possible, you'll find that a lot of things that were previously thought impossible actually do become possible.

Page 125

Beliefs, intelligence, and failure

2007-05-03T21:40:00.000-07:00

From the book "Hard Facts, Dangerous Half-Truths, and Total Nonsense" (via this blog, via an email from a friend):

A series of studies by Columbia University's Carol Dweck shows .... that when people believe they are born with natural and unchangeable smarts ... [they] learn less over time. They don't bother to keep learning new things.

People who believe that intelligence is malleable keep getting smarter and more skilled ... and are willing to do new things.

That's a pretty important fact. It's also an excellent example of how your success is determined by your beliefs/mindset/frame. If you believe that you can't get smarter, you won't!

I searched around a bit and found another interesting quote:

...beliefs about the nature of intelligence have a significant impact on the way they approach challenging intellectual tasks: Students who view their intelligence as an unchangeable internal characteristic tend to shy away from academic challenges, whereas students who believe that their intelligence can be increased through effort and persistence seek them out.

Students who hold an "entity" theory of intelligence agree with statements such as "Your intelligence is something about you that you can't change very much." Since they believe their intelligence is fixed, these students place high value on success. They worry that failure-or even having to work very hard at something-will be perceived as evidence of their low intelligence. Therefore, they make academic choices that maximize the possibility that they will perform well. For example, a student may opt to take a lower-level course because it will be easier to earn an A. In contrast, students who have an "incremental" theory of intelligence are not threatened by failure. Because they believe that their intelligence can be increased through effort and persistence, these students set mastery goals and seek academic challenges that they believe will help them to grow intellectually.

Not only does a belief in fixed intelligence prevent learning, it also makes people more risk averse! It's easy to see how some cultures or micro-cultures make people successful, while others keep them mired in failure.

I expect that this same basic principle applies to most things, whether it's skills with computers or skills with people. Obviously "nature" plays some role, but beliefs, knowledge, and determination are usually more important. Furthermore, I suspect that you can learn to enjoy these activities, and that finding the fun is the key to mastering the skill.

And you thought all the good domain names were taken?

2007-04-30T02:25:00.000-07:00

Using only the magnetic letters stuck to my refrigerator, we were able to discover these promising, yet unregistered domain names!

spitbomz.com
xombiz.com
lisp.tv (imagine the media empire that could be built around that!)
xzombi.com
QykDr.com (quick doctor)
nevrqyk.com
WakiDr.com
FuziXombi.com
phuzix.com
mowbul.com
t-mubal.com
WakyXombieFujPHD.tv

And venturing beyond the refrigerator, we also found these gems:
un4tun8.com (I went ahead and registered this one)
4nic8.net
t-mobull.com
grrrrrrrr8.com (that has eight 'r's)

There's a goldmine in replacing 'for' with '4' and 'ate' with 8.

Don't limit yourself to good ideas...

Whose reality are you living in? Whose reality would you rather live in?

2007-04-28T03:47:00.000-07:00

Mental frames are the biased and limited way in which information is perceived or understood. Because the human brain is inherently biased and limited, we are always in some mental frame. That frame determines how we relate to and understand reality. As far as you can tell, that frame is reality.

This is why the world seems better when we are happy, and worse when we are sad. It is also a reason that people on "the other side" of an issue seem so stupid, misinformed, and out of touch with reality. Not surprisingly, mental frames are often discussed with respect to politics. However, their scope and importance goes well beyond that -- they ultimately determine not only our perceptions, but our whole mindset about what is valuable, practical, or dangerous, and what behaviors are responsible and acceptable.

If someone pursues their passions, are they boldly living life to the fullest, or are they simply being frivolous and irresponsible? The answer, of course, depends on your frame.

Perhaps you might prefer to be objective about reality and escape these limiting frames. Unfortunately, that's not really an option, at least not while we are stuck with these monkey brains inside our heads. Although it is nice to attempt objectivity, we must accept that we are human, and therefore limited. To deny that and claim true objectivity is to deny the truth and be stuck in a very limiting and annoying frame.

Mental frames help to explain why some people needlessly stay in bad jobs, bad relationships, or other bad situations -- in their reality, it makes sense. Clearly, some frames are better than others (from my perspective :).

Here's the good news: You can switch frames!

Many influences can shift your frame, such as reading books or taking a walk in the park, but one of the most powerful influences is the people around us. We tend to synchronize frames with our friends, family, co-workers, and anyone else we encounter (including the people on tv), though obviously some of those people are more influential than others.

Therefore, if you want to change your reality, change your surroundings. Find people with a more attractive reality, and live with them. This is very important. When you spend a lot of time with people, their reality becomes your reality.

For example, if you are interested in startups, but work in a big company, you are in danger. If you stay there too long, you will be drawn into the big company frame shared by the other "lifers". Startups will all seem too risky, frivolous, or impractical, and you'll spend the rest of your life in that big company (and posting bitter comments on TechCrunch).

Similarly, if you dream of pursuing some other career or lifestyle that is not shared by the people around you, then you either need to accept that it's not going to happen, or you need to change your situation.

Please note: I'm not suggesting that you completely cut off contact with people outside of your desired frame (that's how cults operate, btw). To the contrary, it's good to keep in contact with a wide variety of people -- that will help provide perspective and keep you from becoming trapped in your new frame (which might not be as great as it seemed from the outside). What I am suggesting is that your target frame should have significant representation in your life (like 50%), or at the very least, some minimal representation (5%) so that it doesn't completely fade from your reality.

Update: To better understand frames, think of a time when you were excited about an idea or possibility (or anything else), but when you shared that thought with someone else, they somehow ridiculed, doubted, or otherwise criticized it. How did that make you feel about the idea? Were you a little less excited, more doubtful, or perhaps even somewhat embarrassed about it? If so, you've just entered further into their frame -- their reality is becoming your reality. If you want to nurture your dreams, it's better to share them with people whose frame is compatible with the dream.

Read my previous post on reality. What am I doing to your frame?

Update Two: One of the commenters on news.yc suggests that if everyone really were in their own frame/reality, then it would be impossible to communicate or build products for other people. This is a good point, and it would be true if our frames were completely disjoint. Fortunately, they are not -- we always have something in common. However, the more our frames differ, the more trouble we have communicating. This is why it can be so difficult to communicate with a broad audience, such as on a blog, and why I've decided to write primarily for those with similar frames (because it's easier).

For more thoughts on how your mental frame affects your life, there are some interesting posts on Steve Olson's blog (though he uses the term 'belief system').

The secret to making things easy: avoid hard problems

2007-04-24T11:25:00.000-07:00

That may seem obvious, but in my experience most engineers prefer to focus on the hard problems. Working on hard problems is impressive to other engineers, but it's not a great way to build successful products. In fact, this is one of several reasons why YouTube beat Google Video: Google spent a lot of time solving technically challenging problems, while YouTube built a product that people actually used (using PHP and MySQL, I think, which is not at all technically impressive).

For me, the most effective method of getting things done quickly is to cheat (technically), take a lot of shortcuts, and find an easier way around the problem (and before anyone jumps in with some comment about security or bank transactions, there are obviously a few exceptions). You only need to think ahead enough to avoid painting yourself into a corner, or have a plausible plan for escaping the corner. There's always an easier way -- work lazier, not harder. Note that this doesn't preclude doing things that SEEM difficult -- easy solutions to important problems that LOOK really hard are the best.

I was reminded of this while replying to the comments on news.yc in response to my post on disks and databases. Whenever anyone mentions the possibility of not using a conventional database, a lot of people will immediately reply that databases solve a lot of very difficult problems, and that you shouldn't put a lot of work into reinventing the wheel. These people are, of course, correct.

The thing is, a lot of those difficult problems are irrelevant for 99% of products. For example, "real" databases can handle transactions that are too large to fit in memory. That was probably a really important feature in 1980. Today, you can buy a computer with 32GB of memory for around $5000. How many GB transactions do you suppose Twitter performs? My guess is zero -- I suspect that their average transaction size is closer to 0.0000002 GB (messages are limited to 140 characters).

I want to be perfectly clear about one thing though: I'm not advising you to ditch your database! If your database is plenty fast, then the easiest thing to do is probably "nothing", and that's what I advise you do. If, however, your db is getting slow or overloaded, then you need to do two things:

Understand the problem
Fix the problem

The correct solution to your problem will depend on your situation. For example, if you have some data that's very important but doesn't change very often (username and password), and some data that gets updated continually but doesn't have to be correct (last active time or hit counters), then a simple solution would be to leave the important data in your database and move the less important data into something really simple but less reliable.

Want an example of "simple but less reliable"? Here's one (in one or two easy steps):

All updates go in to memcached, but not the database
(optional) A background process occasionally copies entries from memcached to the db. Without this, the values will be completely lost when memcached restarts.

The problem with conventional databases

2007-04-23T13:00:00.000-07:00

They were designed for old computers, and lot has changed in the last 10 or 20 years.

Here are stats on some random hard disk from 1991, the Seagate ST-2106N:
Capacity: 106 MB
Average seek: 18 ms
Max full seek: 35 ms

And here are some numbers from a newer drive, the Seagate ST3500830NS:
Capacity: 500,000 MB
Random write seek: less than 10 ms

In the past 15 years, hard disk capacity has increased about 5000x. However, the ability to seek has only increased 3.5x.

In those same 15 years, DRAM capacity has also increased dramatically, probably more than 5000x. My friend just bought a computer with 32 GB of RAM -- that's 320x the capacity of the 1991 hard disk!

Here's another important stat:
1991: strlen("Paul Buchheit") = 13
2007: strlen("Paul Buchheit") = 13

Surprised? Probably not. My point? Our databases are still storing a lot of the same things they were in 1991 (or 1970).

The problem? Most databases are still storing my name using the same techniques that they did 15 or more years ago -- they seek the disk head to some specific location and then read or write my name to that location (this seek can be deferred with write ahead logging, but it will need to happen eventually). These databases rely on the one operation that DID NOT dramatically improve in the past 15 years! They still perform like it's 1991.

Here's another way of looking at what has happened:

Relative to storage capacity, seeks/sec is approaching zero. Disks should no longer be thought of as a "random access" devices.

Some people try to beat that exponential curve by simply buying more disks. Here are a few more stats to consider (actual measurements from my computer):
Disk - sequential read/write: 52 MB/sec
Disk - random seeks: 100/sec
DRAM - sequential read/write: 3237 MB/sec
DRAM - random read/write: 10,000,000/sec

For sequential access, DRAM is about 62x the speed of disk -- disk is way slower, but only by a few orders of magnitude.

However, for random access, DRAM is 100,000 times the speed of disk! Buying and maintaining 100,000 disks would certainly be a hassle -- I don't recommend it.

What can you do instead of buying 100,000 disks? Keep your data in memory, or at least all of the small items, such as names, tags, etc. Fortunately, there are some easy tools available for doing that, such as memcached. If your db is rarely updated, that may be enough. However, if you also have frequent updates, then your database will be back to thrashing the disk around updating its b-trees, and you will be back to 1991 performance. To fix that, you may need to switch to a different method of storing data, one which is log based (meaning that the db updates are all written to sequential locations instead of random locations). Maybe I'll address that in another post.

Finally, one more interesting stat: 8 GB of flash memory cost about $80

Flash has some weird performance characteristics, but those can be overcome with smarter controllers. I expect that flash will replace disk for all applications other than large object storage (such as video streams) and backup.

Update: I'm not suggesting that everyone should get rid of their databases and replace them with some kind of custom data storage. That would be a big mistake. If your database is working fine, then you shouldn't waste much time worrying about these issues. The intent of this post is to help you understand the performance challenges faced by databases that rely on disk. If your database is having performance problems, the correct solution will depend on your situation, and may be as simple as tuning the db configuration.

I also neglected to mention something very important, which is that most databases have a configurable buffer cache. Increasing the size of that cache may be one of the easiest and most effective ways to improve db performance, since it can reduce the number of disk reads.

What if I could not only change the way things are, but the way things have always been?

2007-04-21T20:42:00.000-07:00

I could make the sky blue and pumpkins orange. I could make silly words, like 'snorkel', and silly species, like monkeys. I could make flowers beautiful, and babies cute. I could make you read these words, right now.

Of course it would be very difficult to prove this to anyone, including myself. You see, once I make the sky blue, it has always been that way. Even if I made something really crazy or unlikely, that would just be the way things are. You could challenge me, saying, "Change the calendar, so that one of the months has 28 days," but once I do, it was already like that (and of course there would be some reason), and the challenge would be something else.

It's tempting to think, "There's no way to prove it one way or the other, so it doesn't matter," but that would be saying that the truth does not matter. It would mean that we are closing off all aspects of reality that our mind can not understand or measure.

I'm not telling you to abandon measurement or believe in these abilities -- that would be foolish. Instead, simply ponder the possibilities. This is not about belief -- it's about disbelief. Reality is larger than we can possibly comprehend.

Are you certain of something? If so, is it possible that you aren't seeing the big picture? Perhaps you would change your mind if your understanding were a little broader.

As you consider this, you may begin to sense gaps in your reality. If certainty is gone, then nothing is definitely impossible. Maybe invention is a simple matter of observing what has always existed, and change happens when you notice parts of your self that were there all along. Maybe big ideas are only impractical for those who lack vision and imagination.

And there it is. Microsoft complains about Google being anti-competitive.

2007-04-16T07:52:00.000-07:00

Things can change quickly. What was a crazy prediction a few years ago, just happened. Which of today's little startups will be the target of anti-trust complaints in a few years?

From today's nyt:

Microsoft, a veteran defendant of epic antitrust battles in the United States and Europe, is urging regulators to consider scuttling Google's plan to buy DoubleClick, an online advertising company.

Microsoft contends that the $3.1 billion deal, announced on Friday, would hurt competition in the fast-growing market for advertising on the Web and raises questions about how much personal information would be collected by Google, already a dominant player in online advertising.

Bradford L. Smith, Microsoft's general counsel, said in an interview yesterday that Google's purchase of DoubleClick would combine the two largest online advertising distributors and thus "substantially reduce competition in the advertising market on the Web."

1 - 2 + 3 - 4 + 5 - 6 + ... made easy

2007-04-15T03:08:00.000-07:00

One of the top links on reddit right now is about the infinite series 1 - 2 + 3 - 4 + 5 -6 + ...

The wikiepedia entry is interesting, but it involves some math that I don't quite understand. Apparently the other reddit readers don't understand it either, since one of the top comments there says, "Really, there's no sum (the sum is undefined). 1/4 is just some number that mathematicians can use to compare sequences like that."

The answer 1/4 is a little less arbitrary than that!

Instead, try simply doing the arithmetic:
1 = 1
1 - 2 = -1
1 - 2 + 3 = 2
1 - 2 + 3 - 4 = -2
1 - 2 + 3 - 4 + 5 = 3
1 - 2 + 3 - 4 + 5 - 6 = -3

See the pattern? It's 1, -1, 2, -2, 3, -3, 4, -4, 5, -5...

What happens if we plot those points on a graph and then draw separate lines for the positive and negative numbers? (first point at position [0,1], the second at [1,-1], third at [2,2], fourth at [3,-2], etc)

The line of positive numbers is defined by the equation y = x/2 + 1
The line of negative numbers is defined by the equation y = -x/2 - 0.5

Notice that the two lines intersect at y = 0.25, which also happens to define the line midway between the other two lines (since they have slopes of +1/2 and -1/2).

It seems that if we "average" the two equations, like so:
(y + y) / 2 = ((x/2 + 1) + (-x/2 - 0.5)) / 2
simplified:
y = (x/2 - x/2 + 1 - 0.5) / 2
simplified:
y = 1/4

Yay!

I don't know if a real mathematician would approve of my method, but at least it's easy to understand.

Google buys DoubleClick, for a double-dose of the advertisin'

2007-04-14T13:53:00.000-07:00

I just wanted to say that. ("His name is Upgrayedd, spelled with two D's, for a double dose of the pimpin'... You see gentlemen, a pimp's love is very different from that of a square." Makes for an interesting analogy, eh?)

Here is the new order:

Microsoft is the new IBM (which is to say, mostly irrelevant)
Google is the new Microsoft (but I like the new one better)
Facebook is the new Google

And Yahoo? They are the new Lotus. They'll continue their slide into total irrelevance and will eventually be bought by the new IBM (Microsoft).

Back when Google was tiny and Microsoft was terrifying, I used to tell people that eventually Microsoft would be making anti-trust complaints against Google. Most people felt that I had a very weak grasp on reality. Now I'm hearing grumblings. After all, it's not like Microsoft has a real shot at direct competition -- they are behind and continue to lose ground.

Please note: I do NOT work for Google, or anyone else for that matter.

Update: Some people seem to be taking my "G is the new MS", "FB is the new G", etc statements a little too literally. I'm not suggesting that they are literally becoming those other companies, but rather that they are taking their place in the tech ecosystem, in the sense of X is the new Y.

"Perfect" is the enemy of "good enough", and

2007-04-14T12:50:00.000-07:00

I hope that you already know that. Perfectionism is a disease. It stops progress and drives us crazy. Perfect is unreachable.

But a while back I had a strange dream, and when I awoke I realized something else: "Good enough" is the enemy of "At all"!

Now I'm not suggesting that quality doesn't matter -- sometimes "good enough" or even "nearly perfect" is very important (brain surgery comes to mind). But more often, "good enough" isn't actually necessary and gets in our way. "Good enough" stops us from ever getting started in the first place.

Forget about your lack of talent, skills, knowledge, time, resources, or whatever else you need to be "good enough". Start an inane blog, take bad photographs, upload boring videos to YouTube, write bad software, create useless products, play bad music, and make ugly art. Forget "good enough", and then simply indulge in the joy of creation.

What did I experience in my dream? Passive consumption is the boring old form of entertainment. Joyous creation is the future of entertainment!

And who knows, maybe after we break through the static friction of quality, we'll discover that some of our work really is good enough, or maybe even great. But remember, although quality is nice, it's not the point.

The self-styled curmudgeons will continue to complain about all this senseless creation, but don't mind them -- they are simply flinging feces through the bars of their monkey cages. They are annoying but irrelevant -- they are not the ones who create the future.

I like xkcd.

Webserver in bash

2007-04-13T17:53:00.000-07:00

And not using perl or any of that fancy stuff. It's the inane things that keep me awake at night.

Getting nc to behave turned out to be the most difficult part. It won't exit until both ends of the connection are closed. Correction, making blogger not mangle this code was the most difficult part.

#!/bin/bash
# web.sh -- http://localhost:9000/hello?world

RESP=/tmp/webresp
[ -p $RESP ] || mkfifo $RESP

while true ; do
( cat $RESP ) | nc -l -p 9000 | (
  REQ=`while read L && [ " " "<" "$L" ] ; do echo "$L" ; done`
  echo "[`date '+%Y-%m-%d %H:%M:%S'`] $REQ" | head -1
  cat >$RESP <<EOF
HTTP/1.0 200 OK
Cache-Control: private
Content-Type: text/plain
Server: bash/2.0
Connection: Close
Content-Length: ${#REQ}

$REQ
EOF
)
done

Update: Fixed script so that it also work in Linux, where tr lacks the -u option.

Did anyone else notice that TechStars and Y-Combinator have the same application?

2007-03-26T18:35:00.000-07:00

They aren't exactly the same of course, but they are surprisingly similar.

Y-Combinator says:

If by August your startup seems to have a significant (say 20%) chance of making you rich, which of the founders would commit to working on it full-time for the next several years?

TechStars says:

If after the summer the business appears to be viable, which of the founders will commit to working on the business full time for several years?

The other questions are similarly similar. I guess that should make life easier for anyone planning to apply for both programs. I wonder how many teams will do that?

Personally, I'm very curious to see how successful TechStars is in cloning the Y-Combinator model. Maybe we will see programs like this springing up around the world. I believe that we are getting an early glimpse of the future of education and investing. The programs will need to evolve of course, but perhaps companies like Y-Combinator will come to form something like an alternate university system. Is that what Steve Yegge was thinking in his (somewhat cryptic) "Wizard School" post?

Update: In case it's not already obvious, these questions originated with Y-Combinator (which has been around a lot longer than TechStars). Archive.org has the older versions of the YC application, such as this one from March 2005. As they say, imitation is the highest form of flattery.

How to be right 90% of the time, and why I'd rather be wrong.

2007-03-22T12:31:00.000-07:00

It's easy -- every time you hear about a new idea or business, just say that it won't work. Say that it's a bad idea, the wrong thing, bubble thinking, a hobby and not a business, hopelessly naive, not innovative enough, too different, that it's been tried before, that it won't appeal to regular people, or any of hundreds of other criticisms seen in the comments on reddit, techcrunch, or elsewhere.

Are these people wrong? Only about 10% of the time. For one reason or another, most new things don't work. That's why it's easy to be right 90% of the time (though the naysayers are likely to misidentify the cause of failure).

What if instead of dismissing everything new or daring, we acknowledge that the future is uncertain, and pursue some of these new ideas despite the risk? History has shown that no one can reliably pick the winners, but what if we were clever enough to pick the right ones 20% of the time? That would mean that were are still wrong 80% of the time.

Is it better to be right 90% of the time, or wrong 80% of the time? It depends on the risks and rewards. If we look at it as an investment where the losers go to zero and the winners have a 10x return, then we can see that it's potentially better to be wrong 80% of the time:
.20 * 10 + .80 * 0 = 2

If we can be right 20% of the time about something with a 10x return, then we should be able to double our money! This is roughly the bet that startup investors are making. They hope that all of their investments will succeed, but realistically they expect that most will not. Of course there are no guarantees -- it's quite possible that due to bad luck or bad bets, none of their investments will succeed.

Aside from possible investment returns, I simply like new things. I like new ideas, new businesses, and the possibility of creating something that will change how people live and work. This is why I would rather be wrong 80% of the time. Not everything will work out the way I would like, but at least I tried.

And now, a fun quote from the earlier days at Google -- it's from a "letter to the editor" at the Wall Street Journal, in response to their "Boom Town" column about Google:

J. Claude Tenday writes: Old habits die hard. For Sergey Brin to expect to grow a $100 billion/year business from an obscure $60 million/year niche player in an already mature Web-search market shows that Mr. Brin is still California dreamin'. And for the Wall Street Journal to take him seriously shows reporters may still be prey to dot-com hype!

Of course Google still isn't a $100 billion/year business, but I'd say there's about a 20% chance that they will become one :)

Recommended reading: Fooled by Randomness. Taleb does a good job of exploring risk and explaining how most people have a very poor understanding of it. He makes his money being wrong most of the time, buying options that usually expire worthless.

Looking for a co-founder? Try attending the REAL startup school.

2007-03-20T14:31:00.000-07:00

I've noticed a number of people saying things like, "Where can I find a co-founder for my startup?", and perhaps, "I also need funding... and a good idea." Not you, of course -- you already have a great idea, so you just need a co-founder and funding, right? Do you also have intelligent and experienced people who you trust to offer reasonable advice, to help you refine your "great idea" into something that will actually work, to help you out when things go wrong?

Do you think those problems will be solved by attending Startup School? Do you think that you'll meet all the right people, find that co-founder that will perfectly complement you, and learn all the inside information that you need to make your startup succeed? I hope that's not what you think. Startup School is a really good idea. It will give you a nice overview of things to think about, create a sense of possibility, and hopefully convince you to make the jump. Also, if you're there at 9:30, you'll get to hear a talk by a very sleepy me. As great as all that is, it's still just a one day event -- it's unlikely that you'll suddenly know all the right people and be ready to create your own startup.

So where is the REAL startup school, the one where you can find a good co-founder and all that? In my opinion, it's going on right now inside of the thousands of startups that ALREADY EXIST. You don't need to find a co-founder and funding and an idea, you just need to find an interesting startup and get them to hire you. I hinted at this in my previous post, "My startup path", but I'll say it more explicitly now: If you don't already know what you're doing, just join one of the many startups that already exist!

Your new startup job should give you the opportunity to:

Learn about startups.
Meet people who would be good future co-founders or employees.
Witness success or failure -- you can learn a lot from either.
Do things that you aren't qualified to do. I certainly wasn't qualified to build Gmail. This is another great way to learn.
Make a lot of money if the startup happens to be a big success, and you don't necessarily need to own a big slice for this to happen.

Of course not all startup jobs will offer these benefits, which is why you should look for one that's run by smart and ambitious people.

This startup school will take more than a day, and in fact it could go on for years if the startup fails to go out of business and you decide to stick around. However, my prediction is that you'll learn a lot more from the experience than you did during the years spent memorizing trivia in school, and you'll be in a much better position if you then decide to found your own startup.

Who are these great startups that you should be applying to? At some point I'll try to make a list, but for now, here is one suggestion: Xobni. Adam and Matt are definitely smart and ambitious, and they also have some exciting news, which I can't yet tell you about.

Update: In case it wasn't clear enough, I'm recommending both the YC Startup School and the real world startup school (i.e., joining a real startup).

Update two: Some people have gotten the impression that I'm telling you to join a startup and then steal their employees. This is not what I'm suggesting. What I am suggesting is that in a year or two the startup will either be a success (which is good!), or it will not be a success, in which case your co-workers will be looking for something new. Even if it is a success, it could be acquired by (or become) a big company, and the startup types may want to move on. Furthermore, in all cases you have learned a lot more about startups, which was my main point.

Equity math for startups

2007-03-19T13:17:00.000-07:00

Many people don't understand how equity works in a startup. It's a little counterintuitive and I didn't quite internalize exactly what's going on until recently. I think this misunderstanding may also be part of the reason that some people think Y-Combinator is getting too much when they take a 1%-10% stake (they aren't -- if anything, they should get more).

The first thing to understand is that when shares are given to an employee or investor, they aren't taken from someone else -- they are newly created shares -- the pie gets bigger. Of course, your slice of this newly enlarged pie is now a smaller percentage of the total. That's dilution. A lot of people understand that. What they often don't understand is deeper in the details.

I'll explain using a simple example of a company raising a few rounds of investment.

To begin, let's say that the founders decide to join Y-Combinator, and YC asks for 6% equity and a 20% option pool (please note: these numbers are just an example -- every company is different). For simplicity, let's start with 100,000 shares.

Founders	74,000	74.0%
YC	6,000	6.0%
Option Pool	20,000	20.0%
	100,000	100.0%

Next, the startup raises a little money from an angel investor: $100,000 at a $2,000,000 pre-money valuation. The "pre-money" is simply the value of the company before the new investment. Since there are 100,000 shares and the company is worth $2,000,000, we calculate the share price to be $2,000,000 / 100,000 = $20. Since the angel is investing $100,000 at $20/share, he will get 5000 shares. (Sometimes these early angel investments are done with a convertible note instead, but for this example we're just going to issue new stock)

Founders	74,000	70.5%
YC	6,000	5.7%
Option Pool	20,000	19.0%
Angel investor	5,000	4.8%
	105,000	100.0%

That was easy. Now let's get some VC money! What's the company worth? Surprisingly, the answer seems to depend a lot on how much money they are raising. For some reason, a common pattern is that the first major VC round will take about 1/3 of the company. Let's do $5 million at a $10 million pre-money. Also, the option pool should be increased to 25%. That seems like no big deal since it started out at 20%, right? Wrong. They want it to be 25% of the newly financed company -- dilution is making that old option pool too small percentage-wise. In fact, the number of shares in the option pool is going to increase by 225%!

Founders	74,000	36.3%
YC	6,000	2.9%
Option Pool	51,000	25.0%
Angel investor	5,000	2.5%
VC	68,000	33.3%
	204,000	100.0%

The company started with 100,000 shares and by now there are 204,000. Surprisingly, although the VC only took 1/3, everyone else's stake was cut in half. Also, notice that while the option pool is unused, the the VC owns (68,000 / (204,000 - 51,000)) = 44% of the outstanding shares.

What happens if the company decides to raise more money? That will depend a lot on how successful their business is. If it's a huge success and they barely even need the money (something like Facebook), then they can raise money at a very high valuation and have very little dilution. On the other hand, if things aren't going so great and they're desperate for cash, they may end up being forced to raise money at a valuation closer to $0 (I believe this is known as a "cram down"). That will cause massive dilution and basically wipe-out all of the previous owners. Not good. Employees may be offered additional options to keep them from leaving, but for any investor not participating in the new round, it will be a near-total loss.

One last detail: the investors will almost certainly hold "preferred stock" which gives them a number of special rights. For some reason, those rights seem to be slightly different for every deal, but expect there to be a liquidation preference guaranteeing that the investors get their money back (and perhaps a profit too) before anyone else gets anything. If the company in the above example were acquired for $5 million, that money would all go to the investors, and possibly only the VC investor if they included terms giving themselves priority over the earlier investors.

I hope this is helpful. Please let me know if I've made any errors in my examples or explanation. My goal for this post was to make the numbers behind startup equity a little more familiar and intuitive.

The question that I haven't addressed: Is it a good idea to give away 64% (or more) of your company? The answer: it depends. 36% (or 10% or 1% or 0.1% or 0.01%) of something big is worth a whole lot more than 100% of nothing, which is what you'll probably end up with if you are overly focused on retaining ownership. Your primary concern needs to be building a great product and a great company. If you can significantly improve your odds of doing that by giving partial ownership to investors, advisers, and employees, then that's what you should do. Of course you don't want excessive dilution, that's not good for anyone, but there is a healthy middle-ground.

Another fun example: In 1999 Google raised $25 million from Sequoia and KP at a $75 million pre-money valuation, meaning that the VCs together got 25% of the company (about 12.5% each -- I think there may have been a few smaller investors in there too though). That was exceptionally good. Supposedly, John Doerr was heard to say, "I have never paid more money for so little a stake in a startup." (source) By the time of the Google IPO filing, Larry and Sergey each owned about 15.6% of the company (from the S1). Sequoia and KP each had 9.7%. Considering that Google went public with a valuation around $30 billion, and has since risen as high as $150 billion, that was pretty much the best-case scenario for everyone involved. At today's price of $445/share, a 0.1% slice of Google would be worth $138 million!

Update: On news.YC, chandrab pointed out this nice VC dictionary, which explains liquidation preferences and the like.

Welcome to Costco. I love you.

2007-03-17T15:34:00.000-07:00

Idiocracy is a flawed movie. The whole is less than the sum of the parts. However, a lot of the parts are really good. Most movies are completely forgettable. This one isn't. I keep wanting to use some reference from Idiocracy the way I would with Office Space or The Matrix. Unfortunately, nobody has seen it yet, so the references wouldn't convey any meaning. Please watch this movie -- it's a funny and worthwhile use of 84 minutes. The movie is available on DVD, or you can download it.

My startup path

2007-03-16T00:16:00.000-07:00

For some reason, I've always liked the idea of starting an important new company or building something really cool that everyone will use. In college, I became interested in founding or joining a software startup, but had no idea how to do that. I had read a few random books, such as Startup: A Silicon Valley Adventure and Steve Jobs & the Next Big Thing, but I still had no idea how to go about starting a company or even how to find a good one to join. Really, all I knew was that the cool startups seemed to be located in Silicon Valley, and so the only plan I could think of was to find out where this "Silicon Valley" was located and move there (I went to school in Ohio). Luckily, my friend Chad had recently taken a job with Intel in Santa Clara, and he reported that Santa Clara was in fact a part of "Silicon Valley", so I joined Intel too.

That was in 1998 -- I kind of assumed that the streets of Silicon Valley would be "paved with startups", and that I would probably run into them all over the place. As it turns out, that wasn't exactly true. After almost a year spent in the cubicles of Intel, I was ready for something new. I still didn't know much about startups, and the ones that I read about in "Red Herring" magazine all sounded really boring. Fortunately, working at Intel meant that I had plenty of time to read Slashdot, and so rather than do some kind of intelligent research, I simply sent my resume to the handful of local startups I had read about that seemed technically interesting.

I never heard back from most of those startups, but one actually offered me a job: Google. At the time, they were just a small team of people in a little office on University Ave in Palo Alto. Their "business model" seemed pretty unlikely to me, but I liked the product, the people seemed smart, the work sounded very interesting, and the office had a bright and high energy feel, a nice change from the two shades of gray at Intel. I didn't understand how they could possibly beat Altavista, which had more users, more money, more engineers, and more everything. If Google did start to take off, wouldn't Altavista simply match their product and kill them? So I pretty much assumed that Google wouldn't make it, but it seemed like a fun job and a great opportunity to meet new people and learn more about startups. Maybe I could find a better startup the next time around. As it turns out, I was only half-right: Google was a great opportunity to meet people and learn about startups (and technology, and products, and more), but of course the whole "Altavista crushes Google" thing didn't happen.

What's my point? You don't need to have it all figured out right now. The important thing is to keep moving forward. Seek opportunities to learn or try something new, something with uncertain outcomes. If you're interested in startups, don't sit around waiting for the perfect opportunity, just go find one that sounds interesting and get a job there. The startup will probably fail, but you will succeed because you have learned a lot more than you otherwise would have. (This isn't to say that you can't learn things in a big company -- you can. For example, you can learn how to be a worker bee.)

Where does my path lead now? Last year I decided that life at Google had become too predictable, and too typical. I quit. It was time to move forward to something new, something uncertain.

Are you my audience?

2007-03-14T23:15:00.000-07:00

Every so often I have an idea or thought that might be worth sharing. But then I think of all the ways in which it could be misinterpreted or disputed, and then I think about how to better explain or justify myself, and then I'm just tired of all that thinking and so I don't actually write anything.

In an attempt to avoid that trap, and to clarify my own thinking, I've decided to write for a very limited audience: me, and people who might be similar to me.

My fear is that I will get drawn into some dumb debate or end up arguing with fools, which is no way to live. To avoid this, I'll probably just delete all comments that seem trollish, smell anything like spam, or are just hopelessly annoying.

Also, I'd like to preface everything I'll ever say with this: I don't actually know anything. (and neither do you)

Hello

2007-03-13T15:31:00.000-07:00