peterme.com

Thank you Peter. In addition to - the from a simplier time problem - existing usability practices were developed during a time when people most likely _didn't_ have access to a personal computer, let alone an webmail account or an Amazon history. 63% of customers now have experience with all these tools or at least a solution in place to solve them (as you describe). This reminds me of the 'usability not usable' rant I wrote a while back. Thanks for the refresher.

Hi, I wanted to reply to this comment.

Developing methodology is an art. We truly need to understand the limits of each of the methods and tactics.

In my opinion the lab will always have its place and use.

Using a controlled environment in order to execute usability testing depends on the goal of the test. It is obvious that the controlled environment should be chosen when and only when we need to evaluate if the artificial grouping (information architecture) and if the user interface elements correspond to the mental mapping and needs of the intended users. That means that the exterior elements to the task aren't being and don't need to be considered. We aren't evaluating the multi-tasking factor here.

If data related to the exterior elements (such as nose, multi-tasking factor, the physical potion of the users,...) needs to be captured and processed, we often will opt for contextual analysis (in order to capture those data) and contextual testing (testing in real environment).

Is Lab Usability Dead? My answer is absolutely not.

Peter, I believe you are comparing a research activity (field research) with an evaluation activity (lab testing). The two happen in very distinct phases of a design cycle. Sure, an evaluation will help you understand your users better so your next design meets their requirements more closely, but conceptually they happen in different phases of a design iteration.

By the way: If you would have changed the title to "Lab Usability is Dead" (which you do seem to imply in the article), you would have sounded very Jakob-like. Just a warning, you know...

Another value of lab-based testing is that it allows observers and it allows for the creation of video clips (not impossible in the field, but harder to gather) which can then be used in driving home points about the product. When you work in an environment where you have a lot of control over the design decisions and relatively few masters to convince/cajole having observers and making highlight videos is less important but there are circumstances where there is persuasive power in observing and/or creating video--persuasion that can mean the difference between an iterative design phase in a project or an iterative design methodology for a department being *funded*. Your point is well taken that field research is going to drive better results from a pure design perspective (esp early in the design cycle) but lab usability still has a place--this is a case where the baby could be thrown out with the bath water.

Peter, you know that I know that. I have two responses to it:

1) People *do* use lab usability as a research activity. Which is foolish, for the reasons above.

but, more interestingly, I think
2) lab usability is failing as an evaluation activity, because of the forces I discuss. It made a perfectly fine evaluation activity in a simpler world. Now it provides a false sense of performance, because no one uses tools in such isolated fashions.

Samantha, you can get everything you ask for from remote usability testing.

I agree that field research and remote usability testing are under-used and undervalued compared to lab usability testing. This should change.

A few caveats, though:
1) A lot of usability managers (Dennis Wixon being the most prominent) advocate highly iterative design processes with many closely-coupled test-revise-test cycles. The more testing you're doing, the greater the value of an efficient recruiting and lab setup, and the harder it is to do field tests. Remote usability testing has promise here--but it seems it would be harder to scale than lab testing. In my very limited experience with remote usability testing, it was a lot of hassle to coordinate, set up, and make sure everything ran smoothly.

2) Sometimes you want to test the task, not the interruption. There has to be a balance between naturalistic exploration of users' lives, information use, etc. and answering the question 'Can people really use this when they sit down to do it?' Particularly for applications that people use regularly for significant periods of time, optimizing the task flow is really important.

For the most part i agree, but i think your opinion is biased toward the domain which you design for which is ICT's. There's still a lot of design being done for high-risk work environments with time pressure (aviation, shipping, medicine) for which lab usability / research can give you specific feedback that you can't get in an ethnographic study. An example would be recreating a flight simulator and bringing in pilots to test a new cockpit interface to see if it matches their mental models, directs attention to critical information at specific times... in short is usable. Although design within these more industrial domains has definitely warmed up to the idea of ethnograpic research, lab usability research is still considered critical.

So Peter, thinking about your reply to PeterB, maybe you should focus on the real issue:

Usability testing is not user research.

You have mixed them up as much as the people you are having a go at. I know you know the difference, but you haven't made it clear here ;)

Lab testing as user research is ridiculous. Lab testing as evaluation can be useful as long as it is considered as a confirmation of a design process, not a research or design activity. Sometimes it is just far too hard to get your new site/service into someone's normal environment, but is easier to get them into yours.

Anyone else getting tired of reading articles titled "... is dead"?

A few thoughts...
David, you're right that lab usability is appropriate for high risk environments with time pressure (aviation, shipping, medicine)... because those are the environments the method was developed for! But those aren't the environments in which 99% of lab usability is being done.

Donna, your missing my point. I am coming out and saying that Lab Testing As Evaluation should be done away with. At least, lab testing for evaluating web sites. With remote user testing, you get *all* the methodological benefit of lab testing, AND you let that person be in their normal environment.

We're using remote testing more and more for all the reasons stated above. For a tool, take a look at Macromedia Breeze Live, a screen-sharing tool based on Flash, which means minimal setup for participants. We're quite happy with it.

If it's dead then why does your company still offer Usability Testing as a research service? :)

I have experience with remote usability, enthographic studies, and formal lab studies...I think it just boils down to applying the right method for the project?

Abe and Steve, the difficulty scaling and having participants join sessions will not be a problem in the very near future. Breeze is very cool but still requires a 2mb install as flash object for the participant. You can find more information and tools at the remote usability Wiki here: http://remoteusability.com

Hi Peter -- I appreciate your points about lab usability, but I cannot agree with what I perceive to be a comparison + substitute methodology discussion. I don't agree that remote UT can get the same results as lab UT because the methodologies and environments differ. Echoing some other comments, UT is not user research. Having someone come to a lab to evaluate some thing is not the same methodology as user research of the ethnographic type/methodologies you've discussed. Each has their place and, in my opinion (which is based on my experience as a practicioner and observer), one is not "better" or "worse" than the other - it hinges on what is being evaluated.

Hi
Curious as how you would perform a test on an early HTML prototype that cannot be loaded externally? Since it's best to test early in the design cycle surely this example alone is enough to keep lab testing alive?