Quick: Read this if you ever store password data

Thomas Ptacek has a good post explaining why your password storage scheme is probably bad.

His post has a lot of words, so I'll summarize:

• Storing passwords in plaintext is obviously bad (most developers already know this)
  
• Using a salted-hash such as MD5 or SHA-1 isn't much better. (too easy to brute force)
  
• Your other clever password storage ideas are probably bad too.
  
• Use bcrypt instead. It was created by real cryptographers for just this kind of thing.
  

Here's bcrypt for Java. It has a nice simple API:

// Hash a password for the first time
String hashed = BCrypt.hashpw(password, BCrypt.gensalt());

// Check that an unencrypted password matches one that has
// previously been hashed
if (BCrypt.checkpw(candidate, hashed))
    System.out.println("It matches");
else
    System.out.println("It does not match");

I'm sure other popular languages have similar libraries.

16GB iPhone "SLR" includes high-quality 8MP camera

And it still fits in my pocket:

Reality, page 191

...replaces all objects of belief with one single thing: reality itself. We believe only in this universe. We don't believe in the afterlife. We don't believe in the sovereignty of nations. We don't believe in money or power or fame. We don't believe in our idols. We don't believe in our positions or our possessions. We don't believe we can be insulted, or that our honor or the honor of our family, our nation or our faith can be offended...

We just believe in reality. Just this.

...doesn't ask you to believe in anything you cannot confirm for yourself. It does not ask you to memorize any sacred words. It doesn't require you to worship any particular thing or revere any particular person. It doesn't offer any rules to obey. It doesn't give you any hierarchy of learned men whose profound teachings you must follow to the letter. It doesn't ask you to conform to any code of dress. It doesn't ask you to allow anyone else to choose what is right for you and what is wrong.

...is the complete absence of belief. ...is the complete lack of authority. ...tears away every false refuge in which you might hide from the truth and forces you to sit naked before what is real. That's real refuge.

Reality will announce itself to you in utterly unmistakable ways once you learn to listen. Learning to listen to reality, though, ain't so easy. You're so used to shouting reality down, drowning it out completely with your own opinions and views, that you might not even be able to recognize reality's voice anymore. It's a funny thing, though, because reality is the single most glaringly obvious thing there is... Yet we've forgotten how to recognize it.

What does it mean to own a "right"? Was Gandhi a thief?

From the Wikipedia entry on Gandhi's 1930 Salt March:

The British monopoly on the salt trade in India dictated that the sale or production of salt by anyone but the British government was a criminal offense punishable by law. Salt was readily accessible to coastal area dwellers, but instead of being allowed to collect and use it themselves for free, they were instead forced to purchase it from the colonial government.

In other words, the British government owned the right to produce and sell salt in India.

And another page says:

On April 5, 1930 Gandhi and his satyagrahis reached the coast. After prayers were offered, Gandhi spoke to the large crowd. He picked up a tiny lump of salt, breaking the law. Within moments, the satyagrahis followed Gandhi's passive defiance, picking up salt everywhere along the coast. A month later, Gandhi was arrested and thrown into prison, already full with fellow protesters.

Was Gandhi stealing salt from the British government, or was he simply breaking the law which gave the British the exclusive right to produce salt?

Since Gandhi wasn't actually taking the salt away from the British, and the ocean contained a practically infinite supply of salt (so it can't be claimed that he was taking it from anyone else), I'd argue that he was not stealing anything.

Did Gandhi steal the legal "right" to produce salt? Not really. A right has no substance or physical reality, so it is difficult to outright steal one (you could perhaps trick someone into signing a contract transferring the right, but clearly that's not the case here).

Gandhi's actions DID decrease the value of the British right to produce salt, but many actions could have that effect. For example, he could have convinced the people of India to switch to a lower salt diet, thereby decreasing the demand for salt and the value of the British monopoly. I don't think that anyone would call that stealing.

In fact, I've never seen anyone claim that Gandhi was a thief. It seems like a somewhat silly argument.

So what's my point? Owning a "right", which is a form of "Imaginary Property", is not the same as owning real property. If someone violates your exclusive rights, they may be breaking the law, but they are not stealing. To claim otherwise is silly and dishonest.

Update: Some people have interpreted this post as meaning that I'm opposed to Imaginary Property, or that infringing on other's legal rights is ok. That is not the case at all -- IP can serve an important and beneficial role in society. My only point is that Imaginary Property is not the same as Real Property, and that infringing on someone's legal rights isn't the same as theft. Pretending that they are the same has caused a lot of unnecessary trouble.

The first thing that you need to understand about humans

Humans aren't rational -- they rationalize. And I don't just mean "some of them" or "other people". I'm talking about everyone. We have a "logic engine" in our brains, but for the most part, it's not the one in the driver's seat -- instead it operates after the fact, generating rationalizations and excuses for our behavior.

I first realized this fact several years ago when I read something about "split brain" people. These people have had their corpus collosum, the structure which connects the left and right brain hemispheres, cut or damaged, leaving the two halves of the brain largely unable to communicate with each other. I don't recall where I read that, but a quick Google turned up this similar article:

"A split-brain patient shown a photograph of Hitler only in the right hemisphere, for example, might exhibit facial expressions indicating anger or disgust. But when asked to explain those emotions, the patient will often invent an answer, such as 'I was thinking about a time when someone made me angry.'" (Newberg and D'Aquili 2001, p. 23)

Kenneth Heilman offers another, more concrete example, writing about the research of Dr. Michael Gazzaniga and his colleagues. In one experiment, they showed sexually suggestive pictures to a woman with callosal disconnection, flashing them only on the left half of a screen so only her right hemisphere could perceive them. The woman giggled and blushed, but when asked why she was doing so, she replied that she was thinking of something embarrassing (Heilman 2002, p. 129).

Are these people lying? In one sense of the word, perhaps; but it seems clear that there is no conscious intent to deceive. Rather, researchers have concluded, what is happening is that the right hemisphere, upon seeing an image with strong emotional connotations, generates the appropriate response. However, due to the callosal disconnection, it cannot transmit the associated sensory data to the left hemisphere and its language centers. The left hemisphere perceives a change in the body's state, but does not know why - and so it "fills in" the missing details, fabricating a logical reason for the emotional reaction. This happens at a subconscious level, so that the person genuinely believes the verbal explanation they provide. In the language of psychology, this filling-in process of unconscious invention is called confabulation.

The important realization is that this process of "confabulation" is not limited to people with brain damage -- everyone does it -- people with "split brain" are just a little more obvious.

It's tempting to think that "most people" have this problem, but that you or I are different, to think that our actions are all logical. That is a mistake -- denying the truth is irrational and dangerous. By accepting that people are fundamentally irrational, we can deal with ourselves and others in a more rational and effective manner. We can learn to manage our irrational selves (somewhat). If, however, we insist that all of our actions and feelings are rational, then we will never be able to deal with them honestly, and are more likely to cling to irrational beliefs and limitations. If we are going to be honest, then we must admit the possibility that everything we know and believe is, in fact, incorrect.

The fascinating (and easy to read) book "Influence: Science and Practice" has some great examples and explanations of how people actually work. Here's a great quote about the effect of attractiveness, and how unaware people are of their true decision making process:

A study of the 1974 Canadian federal elections found that attractive candidates received more than two and a half times as many votes as unattractive candidates (Efran & Patterson, 1976). Despite such evidence of favoritism toward handsome politicians, follow-up research demonstrated that voters did not realize their bias. In fact, 73 percent of Canadian voters surveyed denied in the strongest possible terms that their votes had been influenced by physical appearance; only 14 percent even allowed for the possibility of such influence (Efran & Patterson, 1976). Voters can deny the impact of attractiveness on electability all they want, but evidence has continued to confirm its troubling presence (Budesheim & DePaola, 1994).

A similar effect has been found in hiring situations. In one study, good grooming of applicants in a simulated employment interview accounted for more favorable hiring decisions than did job qualifications—this, even though the interviewers claimed that appearance played a small role in their choices (Mack & Rainey, 1990). The advantage given to attractive workers extends past hiring day to payday. Economists examining U.S. and Canadian samples have found that attractive individuals get paid an average of 12-14 percent more than their unattractive coworkers (Hammermesh & Biddle, 1994).

Equally unsettling research indicates that our judicial process is similarly susceptible to the influences of body dimensions and bone structure. It now appears that good looking people are likely to receive highly favorable treatment in the legal system (see Castellow, Wuensch, & Moore, 1991; and Downs & Lyons, 1990, for reviews). For example, in a Pennsylvania study (Stewart, 1980), researchers rated the physical attractiveness of 74 separate male defendants at the start of their criminal trials. When, much later, the researchers checked court records for the results of these cases, they found that the handsome men had received significantly lighter sentences. In fact, attractive defendants were twice as likely to avoid jail as unattractive defendants. In another study —- this one on the damages awarded in a staged negligence trial -- a defendant who was better looking than his victim was assessed an average amount of $5,623; but when the victim was more attractive of the two, the average compensation was $10,051. What's more, both male and female jurors exhibited the attractiveness-based favoritism (Kulka & Kessler, 1978).

Other experiments have demonstrated that attractive people are more likely to obtain help when in need (Benson, Karabenic, & Lerner, 1976) and are more persuasive in changing the opinions of an audience (Chaiken, 1979).

Understanding our irrational nature is also critical to product design. How can you expect to make something for humans if you don't understand how they think or make decisions?

Another book, "The Culture Code", has some great examples based on the author's experience helping companies improve the design and marketing of their products.

The people at Chrysler had indeed asked hundreds of questions; they just hadn't asked the right ones. They kept listening to what people said. This is always a mistake. As a result, they had theories about moving the Wrangler in multiple directions (more luxurious, more like a traditional car, without removable doors, enclosed rather than convertible, and so on) with no clear path to follow.
...
When I put groups of consumers together, I asked them different questions. I didn't ask them what they wanted in a Jeep; I asked them to tell me about their earliest memories of Jeeps.
...
I returned to those wary Chrysler executives and told them that the Code for Jeep in America is HORSE. Their notion of turning the Wrangler into just another SUV was ill advised.
...
The executives weren't particularly moved. After all, they had vast research that told them consumers said they wanted something else. ... I asked them to test my theory by making a relatively minor adjustment to the car's design: replacing the square headlights with round ones. Why? Because horses have round eyes, not square ones.
...
They tested the new design and the response was instantly positive. Wrangler sales rose and the new "face" of the Wrangler became its most prominent and marketable feature. In fact, the car's logo has incorporated its grille and round headlights ever since.

Using surveys and focus groups to design your product is a great way to produce boring and mediocre products that are the same as everything else.

Another great quote from "The Culture Code" connects us back to the "split brain" example that I started with:

What do Americans look for in a car? I've heard many answers when I've asked this question. The answers include excellent safety ratings, great gas mileage, handling, and cornering ability, among others. I don't believe any of these. That's because the first principle of the Culture Code is that the only effective way to understand what people truly mean is to ignore what they say. This is not to suggest that people intentionally lie or misrepresent themselves. What it means is that, when asked direct questions about their interests and preferences, people tend to give answers they believe the questioner wants to hear. Again, this is not because they intend to mislead. It is because people respond to these questions with their cortexes, the parts of their brains that control intelligence rather than emotion or instinct. They ponder a question, they process a question, and when they deliver an answer, it is the product of deliberation. They believe they are telling the truth. A lie detector would confirm this. In most cases, however, they aren't saying what they mean.

The reason for this is simple: most people don't know why they do the things they do. In a classic study, the nineteenth-century scientist Jean-Martin Charcot hypnotized a female patient, handed her an umbrella, and asked her to open it. After this, he slowly brought the woman out of her hypnotic state. When she came to, she was surprised by the object she held in her hand. Charcot then asked her why she was carrying an open umbrella indoors. The woman was utterly confused by the question. She of course had no idea of what she had been through and no memories of Charcot's instructions. Baffled, she looked at the ceiling. Then she looked back at Charcot and said, "It was raining."
...
Even the most self-examining of us are rarely in close contact with our subconscious. ... Therefore, we give answers to questions that sound logical and are even what the questioner expected, but which don't reveal the unconscious forced that precondition our feelings. This is why polls and surveys are so often misleading and useless.

Of course this understanding is important to more than just product design. It's critical to anything involving humans, which is everything involving us. How does it apply to your life and work? Are you still making decisions with the assumption that we are rational?

Quick: All actions should have 'undo'

I just slipped and added a misspelled word to the Firefox dictionary. Guess what the process is for undoing that mistake?

Some apps "fix" this problem by prompting with an "are you sure" type dialog before performing the action. That's very annoying, and it doesn't always work since I might hit space or enter or something and accidentally "ok" it. (this happens to me a lot when the dialog pops up while I'm typing)

I really like the solution used in Gmail:


After an action is performed, state what the action was and give an opportunity to undo, but in a non-intrusive way. More people should copy this approach, including Gmail, which needs it on "Send" (this will require adding a short delivery delay, like 10 sec, but it's worth it).

Quick: Maybe you can hack computers

But do you know how to hack people? It's only code.

Quick: Job culture

Here's a good link about the "Job culture":

The cultural assumption of a true free enterprise system would be: “Individuals are responsible for their own lives and labors. They trade as equals, but are beholdin' to nobody.”

Free enterprise isn't anything like big-corporate capitalism. We've been told the two are equivalent, but that's just another bit of cultural brainwashing.

Think about it. Job holders by definition aren't capitalists. Job holders, no matter how well paid they might be, function merely as the servants of capitalists, just as medieval serfs functioned as the servants of lords. They are beholdin'. They function in a climate of diminished responsibility, diminished risk, and diminished reward. A climate of institutional dependency.

...

The daily act of surrendering individual sovereignty – the act of becoming a mere interchangeable cog in a machine – an act we have been conditioned to accept and to call a part of “capitalism” and “free enterprise” when it is not – is the key reason why the present Job Culture is a disaster for freedom.

James Madison, the father of the Bill of Rights, wrote:
“The class of citizens who provide at once their own food and their own raiment, may be viewed as the most truly independent and happy. They are more: They are the best basis of public liberty, and the strongest bulwark of public safety. It follows, that the greater the proportion of this class to the whole society, the more free, the more independent, and the more happy must be the society itself.”

Madison was speaking specifically about independent farmers, but he was also a believer in the independent entrepreneur – and for the same reasons.

Madison (and his like-minded friend Jefferson) knew that people who are self-sufficient in life's basics, who make their own decisions, whose livelihood relies on their own choices rather than someone else's, are less likely to march in lockstep. Independent enterprisers are far more likely to think for themselves, and far more capable of independent action than those whose first aim is to appease institutional gods.

Living in the Job Culture, on the other hand, has conditioned us to take a “someone else will deal with it” mentality. “I'm just doing my job.” “The boss makes the decisions.” “I'm just following orders.” But if someone else is responsible for all the important choices in life, then we by definition, are not.

I reminds me of my favorite part of Paul Graham's essay on "Why to not not start a startup":

16. A job is the default

This leads us to the last and probably most powerful reason people get regular jobs: it's the default thing to do. Defaults are enormously powerful, precisely because they operate without any conscious choice.

To almost everyone except criminals, it seems an axiom that if you need money, you should get a job. Actually this tradition is not much more than a hundred years old. Before that, the default way to make a living was by farming. It's a bad plan to treat something only a hundred years old as an axiom. By historical standards, that's something that's changing pretty rapidly.

...

Now we look back on medieval peasants and wonder how they stood it. How grim it must have been to till the same fields your whole life with no hope of anything better, under the thumb of lords and priests you had to give all your surplus to and acknowledge as your masters. I wouldn't be surprised if one day people look back on what we consider a normal job in the same way. How grim it would be to commute every day to a cubicle in some soulless office complex, and be told what to do by someone you had to acknowledge as a boss—someone who could call you into their office and say "take a seat," and you'd sit! Imagine having to ask permission to release software to users. Imagine being sad on Sunday afternoons because the weekend was almost over, and tomorrow you'd have to get up and go to work. How did they stand it?

I have a plan to fix this, but it's a little difficult to describe because people have a hard time thinking outside of the surf mindset, the assumptions that underlie the "job culture". It will take some time.

Three types of ideas - bad ones are often the best

Product ideas can be divided into three categories:

1. Obviously good ideas that are very difficult to implement. Efficient cold-fusion, flying cars, and a lot of other sci-fi ideas fall into this group.
   
2. Obviously "good" ideas that seem possible but haven't happened yet. Video phones and HDTV were in this category for a long time. I think this happens when people get excited about technology and overestimate the benefits (and possibly underestimate the cost). I just don't care that much about having a video phone.
   
3. "Bad" ideas. Many of these ideas are truly bad, but some of them will in hindsight turn out to have been very good ideas. I put them in the same category because they are difficult to distinguish without the benefit of hindsight. Some examples are the personal computer ("why would anyone need a computer?"), Google ("there are already too many search engines, and besides, search engines don't make money"), and Blogger ("can't you just use Geocities, and besides, are there really that many people with something worth saying?"). More recent (and still controversial) examples include Facebook and Twitter.
   

I expect that a lot of people will argue with the fact that I've grouped "truly bad" ideas together with "seemed bad but were actually very good" ideas. Everyone thinks that they can tell which ideas are the good ones, but observation suggests that they can't. I do believe that some people are better at picking winners than others, but at best they have maybe 50% accuracy. I've seen a lot of very smart people (such as at Google) be very wrong about these things.

For example, I remember when the "upload videos and put them on the web" incarnation of Google video was first being developed. Nearly everyone inside of Google, including me, was very skeptical that anything worthwhile would ever be uploaded. The common prediction was that it would all be movies and porn. Of course there was some of that, but the skeptics were completely wrong about the lack of worthwhile content. Uploaded video is one of the most important developments of the past several years. Unfortunately Google Video was burdened with an incredibly bad upload process (it included installing a windows client to do the upload!), and YouTube, which was started AFTER Google Video launched, took over. I believe that mistake was partially due to the negative expectations.

Here's my point: The best product ideas are often found in the "bad ideas" category!

If you're a super-genius researcher looking to dedicate your life to a really important problem that you may never solve, then the category one "cold fusion" ideas may be a good option. If you've found some clever way to dramatically reduce the cost of implementing a category two idea (like Skype did to the video phone), then that might be a good option. However, the real "low hanging fruit" is likely to be found with the category 3 "bad ideas". You'll have to deal with annoying skeptics and haters who say that you are wasting your time, but sometimes you'll create something hugely important, or at least moderately successful (and they won't).

Realizing that good ideas and bad ideas are often nearly indistinguishable, there are a few more lessons to be learned:

• Instead of endlessly debating whether an idea is good or not, we should find faster and cheaper ways of testing them. This is one of the reasons why open systems such as the Internet or market economies develop faster than closed systems, such as communism or big companies -- individuals and small groups are able to build new things without getting approval from anyone.
  
• Your product idea is generally worthless, because to most people it's indistinguishable other bad ideas. Very few people will want to buy (or steal, or take for free) your idea because they already have their own bad ideas which they like better. (There are other reasons as well, but this is one of them. The way to make your idea valuable is to demonstrate that it's not bad by building a real product -- this also demonstrates your ability to execute.)
  

Obviously there is a lot more to say about these two points, and I haven't done enough to explain or justify them right now. Maybe in another post...

Not getting health insurance is a really bad idea

Seriously. If you can possibly afford insurance, then there's no excuse for not having it. I'm especially thinking of young people doing a startup and not buying insurance in order to "save money". That's a really dumb optimization, and suggests that you aren't very good at making trade-offs. A quick search shows that in California you can get a decent plan for $77/month. If you can't find $77/month, then maybe you aren't ready to do a startup.

I'm reminded of this issue right now because I just passed someone in SF laying face down on the pavement, not moving. His bicycle was nearby and I'm guessing that he had just been hit by a car. I hope that he does ok, but it didn't look good. (people were already helping him, and the police were just a few cars back, so I didn't stop)

Even if you are young and healthy, bad things can still happen. My brother was perfectly healthy until he got cancer at age 33. When my daughter was in the hospital, it cost $12,000/day -- she was there for almost 3 months.

The simple fact is that the health care system in the US isn't setup for people who don't have insurance. If something bad happens, you don't want to be one of those people. I'm not looking to debate our system, so please don't bother. Right now, the system is the way it is.

My point is simple: Do what you can to get insurance. Now.

Quick: Unintended consequences

For some reason, I like reading about unintended consequences. It seems that reality always has more depth and complexity than we would like to admit. This is part of the reason why people who claim to have the answers, generally don't.

This comes to mind because I was just reading a little about the effects of pesticides on plants:

Pesticide residues in plants are regulated to protect human health. They are measured when plants enter the market. However, this procedure does not consider that pesticides modulate secondary metabolism in plants when they are applied. The question arises whether it is conceivable that pesticide-induced changes in the chemical composition of plants influence human health. The examples resveratrol, flavonoids, and furanocoumarins indicate that plant phenolics may have subtle effects on physiologic processes that are relevant to human health. These effects may be beneficial, e.g., due to the inhibition of the oxidation of macromolecules and platelet aggregation or by their pharmacologic properties. Depending on concentration and specific chemical composition, however, plant phenolics may also be toxic, mutagenic, or cancerogenic. The consequences of a modulation of plant phenolics on human health are complex and cannot be predicted with certainty. It may be that the modulation of plant phenolics at the time of application and not the usually low level of pesticide residues at the time of consumption is critical for human health.

Put another way, pesticides can actually alter the nutritional properties of plants by removing environmental stress, stress which would have caused the plants to produce nutrients that may be important to humans. I hadn't considered that.

Quick: Big screen development

This is a good setup for working together, and the price of 1080p televisions is dropping.

Quick: DBE

People sometimes ask what "Don't Be Evil" really means. Eventually I'll write down my thoughts, but for now this excellent post from Seth Godin explains a good part of it:

Let me be really clear, just in case. If you think that the world would be a better place if everyone owned a handgun, then yes, market handguns as hard as you can. If you honestly believe that kids are well served by drinking a dozen spoonfuls of sugar every morning before school, then I may believe you're wrong, but you should go ahead and market your artificially-sweetened juice product. My point is that you have no right to market things you know are harmful or that lead to bad outcomes, regardless of how much you need that job.

Along the way, “just doing my job,” has become a mantra for blind marketers who are making short-term mistakes in order to avoid a conflict with the client or the boss. As marketing becomes every more powerful, this is just untenable. It’s unacceptable.

If you get asked to market something, you’re responsible. You’re responsible for the impacts, the costs, the side affects and the damage. You killed that kid. You poisoned that river. You led to that fight. If you can’t put your name on it, I hope you’ll walk away. If only 10% of us did that, imagine the changes. Imagine how proud you’d be of your work.

There are a lot of worthwhile things in the world -- why waste your life working on something evil?

The dogmatic programmer - when software becomes religion

Do you believe that there is only one "right way" to do things? Do you ignore evidence that other solutions may also work? Do you dismiss the possibility that there are reasonable trade-offs to be made between the "right way" and some other way? (do you leave angry comments on my blog?)

Do you think I'm kidding? Read these comments about GET vs POST. There's no question that POST is the "right" way and generally safer, but sometimes it's annoying. Even though GET isn't "supposed" to work, it often can be made to. Don't believe me? Google does billions of dollars a year in GET based transactions in the form of CPC ad clicks (which can cost over $50/click). A lot of other things work this way too, including most non-ajax webmail (for changing read/unread state, at the very least).

If you're writing missile guidance software or something then, please, please, please do things the right way -- it's better safe than sorry, right?

However, if you're writing little web apps, then there's a good chance that you can bend a lot of the rules and produce a better product sooner. I'd rather have something imperfect but useful and popular than something "perfect" but unfinished and unused. Sometimes, it's better sorry than safe.

When is it safe to bend the rules? Just ask yourself, "What's the worst that could happen?" If the worst case isn't too bad, then you're probably ok. Minor bad things will happen no matter what, so it's often better to put your energy into general problem recovery, rather than imagining that you can simply avoid all problems. (this is especially true on the web)

Quick: Vibe

Seth Godin on "the vibe":

Have you ever been at a banquet or in a boutique or at a concert or a meeting or a company where the vibe was incredibly positive?
...
If vibe is so important, why does it sound flaky to worry about it? Who's in charge of the vibe at your place? Could it be better? A lot better?

This is a very real and under-appreciated part of work. It was one of the first things I noticed when I started at Google after working at Intel -- the place just felt so much more alive. Now it's something that I watch for when visiting startups -- my guess is that upbeat and energetic companies will outperform the ones that feel oppressive and hopeless.