Have you ever used a product that is so difficult you asked yourself, “How did they ship this steaming pile?” Doesn’t anyone evaluate usability?

Many companies make mistakes conducting usability testing, but I want to discuss people who do it right–and still get it wrong.  In other words, when can results mislead you?

Usability testing typically involves showing people products and asking them to try several tasks. This gives you a good assessment of how easy your product is to learn, but could miss (or create) problems in other areas of usability.

For example, a designer once told me testing showed people couldn’t find a given feature. So they decided to display a dialog offering the feature every time you used the app. Problem solved. Unfortunately, other  problem created, as in, I can’t believe you show me this stupid screen every time. And follow-up testing testing might not detect such problems since people are more patient in “tests” than in real life (e.g., they may actually read wizard text rather than impatiently clicking “Next”). To cite the Heisenberg principle (albeit inaccurately), the act of measuring usability can affect the results.

Also, usability testing captures the first experience, not learning curves.  As novelty wears off, swearing may ensue.  Or conversely, some features are difficult to discover but delight customers anyway (classic example: ejecting Mac floppy disks by dragging to the trash).   A test subject once explained this paradox to me by saying, “it’s intuitive once you figure it out.” (Tip: when testing, repeat troublesome tasks to test retention.)

Most important, over time the value of intuitiveness declines, but the value of efficiency increases. You stop learning new features, but extra steps for the ones you use often become annoying. (For the theoretical extreme of “the most intuitive product ever,” see the Onion spoof of a Mac with no keyboard but a giant iPod wheel.  “Everything is just a few hundred clicks away.”)

In other words, myopic reliance on process can create a false sense of success.  Understand the limitations of usability testing, and augment it with real-world evaluation over time.