peterme.com

Is Lab Usability Dead?

I would love it if we could simply put a stake into the practice of lab usability. It's run its course, and it's simply not well suited to truly measuring the effectiveness of designs in the modern era.

Usability engineering was born of a simpler time. Before everyone was networked. When people used computers for Calculation (basic math, spreadsheets, etc.) and Creation (word processors, graphics programs), not Consumption and Communication. When someone could be expected to focus on a single task at hand for many minutes, if not hours on end. When people had one "program" running at a time.

And now we're in a world people are interrupted an average of every 11 minutes. Where they have multiple applications, and multiple windows within those applications, open. Where people are storing passwords and bookmarks on their machines. Where people are managing multiple devices, from their computer to their cell phone to their iPod.

And what have we done? We still practice "lab usability," where we invite someone into a sterile neutral environment (either an observation facility, or a conference room), and ask them to use a strange computer to engage in a series of tasks (whether scripted or self-motivated, it doesn't matter). And so while we might understand how well someone can use our tool in this exceedingly artificial world, it falls short of helping us appreciate how this tool will fit into their ever crazier contexts.

We need to practice methods that factor the complexity and messiness of reality. We need to get out of our labs, and into our users' homes and offices. Where they can use their machines, with their myriad applications and devices, and learn what works for them.

Case in point: I recently finished a project where we were going to develop a product to help people remember the important dates and events in their lives. A fairly standard process would have involved prototyping of this product, and then bringing people in to "test" the prototype. What we did, however, was field research. We went into 12 homes, and saw how people currently managed their stuff. And, believe me, it's messy and complex. One participant used: a church address book, a week-at-a-glance, a Palm-style PDA, a simple address-storing-PDA, and an Access database to manage this task. Had we brought her in to test our prototype, we could have found out all kinds of stuff about how she used this prototype in isolation and away from her tools. But we would have learned nothing about how this tool could possibly have integrated itself into his complex web.

We can't have people come to us. We need to go to them. I can't ask someone to come into my lab and use my neutral computer. They have bookmarks, files, passwords stored, and all manner of things that get them through their lives. This is becoming increasingly true in our ever-more Web-2.0 environment, as monolithic solutions give way to smaller point solutions. As people store valuable information in a variety of places (local machine, website/hosted application, paper print-outs filed away, etc.).

At Adaptive Path, we've become a lot more inclined to use remote usability. We've been playing around with a tool called Ethnio that serves as a Webex-like screen sharing environment, optimized for user research. We're in our offices; the users are in their own environments. And we virtually "look over their shoulder" as they attempt to get things done. We see the instant messages coming in, the interruptions of email; we hear the phone calls or the people walking by their cube. And we do so for less cost (no renting a facility, no video cameras or video tapes, lower incentives for recruitment since it's less a burden on the user).

Or, we're going into people's homes and environments and observing them _in situ_. And we're realizing all kinds of insights that we wouldn't otherwise get.

For our DC workshop, I worked up a table for comparing costs of conducting research with 12 participants, for 1 hour each):

So, let's get out of our labs and conference rooms and into our users' environments. It's becoming clear that it's the only way to truly understand, and meet, their needs.

Technorati Tags: fieldresearch, usability, userresearch