October 16, 2005

Is Lab Usability Dead?

I would love it if we could simply put a stake into the practice of lab usability. It's run its course, and it's simply not well suited to truly measuring the effectiveness of designs in the modern era.

Usability engineering was born of a simpler time. Before everyone was networked. When people used computers for Calculation (basic math, spreadsheets, etc.) and Creation (word processors, graphics programs), not Consumption and Communication. When someone could be expected to focus on a single task at hand for many minutes, if not hours on end. When people had one "program" running at a time.

And now we're in a world people are interrupted an average of every 11 minutes. Where they have multiple applications, and multiple windows within those applications, open. Where people are storing passwords and bookmarks on their machines. Where people are managing multiple devices, from their computer to their cell phone to their iPod.

And what have we done? We still practice "lab usability," where we invite someone into a sterile neutral environment (either an observation facility, or a conference room), and ask them to use a strange computer to engage in a series of tasks (whether scripted or self-motivated, it doesn't matter). And so while we might understand how well someone can use our tool in this exceedingly artificial world, it falls short of helping us appreciate how this tool will fit into their ever crazier contexts.

We need to practice methods that factor the complexity and messiness of reality. We need to get out of our labs, and into our users' homes and offices. Where they can use their machines, with their myriad applications and devices, and learn what works for them.

Case in point: I recently finished a project where we were going to develop a product to help people remember the important dates and events in their lives. A fairly standard process would have involved prototyping of this product, and then bringing people in to "test" the prototype. What we did, however, was field research. We went into 12 homes, and saw how people currently managed their stuff. And, believe me, it's messy and complex. One participant used: a church address book, a week-at-a-glance, a Palm-style PDA, a simple address-storing-PDA, and an Access database to manage this task. Had we brought her in to test our prototype, we could have found out all kinds of stuff about how she used this prototype in isolation and away from her tools. But we would have learned nothing about how this tool could possibly have integrated itself into his complex web.

We can't have people come to us. We need to go to them. I can't ask someone to come into my lab and use my neutral computer. They have bookmarks, files, passwords stored, and all manner of things that get them through their lives. This is becoming increasingly true in our ever-more Web-2.0 environment, as monolithic solutions give way to smaller point solutions. As people store valuable information in a variety of places (local machine, website/hosted application, paper print-outs filed away, etc.).

At Adaptive Path, we've become a lot more inclined to use remote usability. We've been playing around with a tool called Ethnio that serves as a Webex-like screen sharing environment, optimized for user research. We're in our offices; the users are in their own environments. And we virtually "look over their shoulder" as they attempt to get things done. We see the instant messages coming in, the interruptions of email; we hear the phone calls or the people walking by their cube. And we do so for less cost (no renting a facility, no video cameras or video tapes, lower incentives for recruitment since it's less a burden on the user).

Or, we're going into people's homes and environments and observing them _in situ_. And we're realizing all kinds of insights that we wouldn't otherwise get.

For our DC workshop, I worked up a table for comparing costs of conducting research with 12 participants, for 1 hour each):

 Lab UsabilityRemote UsabilityField Research
Preparation (Plan, Protocol) 1-2 weeks1-2 weeks1-2 weeks
Recruiting1-2 weeks1-2 weeks1-2 weeks
Conducting Interviews3 days3 days4 days (5-6 with travel)
Analysis1-2 weeks1-2 weeks1-2 weeks
Totals4-7 weeks4-7 weeks4-7 weeks
Equipment(Camera, tapes) $400-600$50 (audio tapes)$400-600
Travel$0$0$0 - $3000 (Air, ground, lodging, meals)
Facilities$0 - $1,000/day$0$0
Phone charges$0$200 $0
Totals$1000-4800$850 - 1450$1600 - 5400

So, let's get out of our labs and conference rooms and into our users' environments. It's becoming clear that it's the only way to truly understand, and meet, their needs.

Technorati Tags: , ,

Posted by peterme at 02:31 PM | Comments (0) | TrackBack


See Me Travel
October 2005
September 2005
August 2005
July 2005
June 2005
May 2005
April 2005
March 2005
February 2005
January 2005
December 2004
November 2004
October 2004
September 2004
August 2004
July 2004
June 2004
May 2004
April 2004
March 2004
February 2004
January 2004
December 2003
November 2003
October 2003
September 2003
August 2003
July 2003
June 2003
May 2003
April 2003
March 2003
Archives from June 13, 2001 to January 2003
Archives from before June 13, 2001
Recent Entries
Is Lab Usability Dead?
Subscribe to my feed:
Powered by
Movable Type 3.2