July 29, 2003

The Most Important Usability Article I've Read This Year

The latest issue of interactions magazine, contains Dennis Wixon's "Evaluating Usability Methods: Why the Current Literature Fails the Practitioner,". (You have to pay to read it online, unfortunately.) Dennis is annoyed with the current debate within the usability field about "how many users?" This debate recurs every professional generation, and, as he points out, generates a lot of heat with little light.

(The debate has lately been stirred up by Jared Spool's group, with "Eight Is Not Enough".)

Dennis, in a remarkably exacting fashion, demonstrates how this debate is pointless, because it assumes (among other things) that the single most important criterion of usability testing is finding the greatest number of problems. He reminds us that our practices derive from engineering, not scientific method. And as such, making smart trade-offs is essential.

From the essay:

[The criterion] is short sighted in that it ignores that problems should be fixed and not just found. If we considered instead a more relevant criterion—namely, how much can we improve the product in the shortest time with the least effort?—we probably would not have asked this question, or we would have asked it in a different form, such as, "What is the best way of deploying the usability resources we have available for this development cycle in order to maximize our beneficial impact on the product?"

Dennis works with Microsoft's Game Testing User Research, which has published a number of articles that are freely available (if, annoying, in Word .doc format).

Posted by peterme at 09:12 PM | Comments (19) | TrackBack

The Slings of Boxes and Arrows -- Persecuting Jakob

One of the best things I've ever (co-)written is now up at Boxes and Arrows, a review of the Nielsen/Norman Group report Usability Return on Investment

We aimed for a New Yorker-style review -- to use the subject as a jumping off point for discussing the underlying issues. So, while about half the review is an evisceration of the reports remarkably flawed methodology and lack of usefulness, the other half suggests steps that user experience professionals can take to begin to appropriately value their contribution.

Believe me when I say that when we set out to write the review, it was not as an excuse to engage in Nielsen-bashing. That emerged only with a close reading (and re-reading) of the report, where it became clear that their approach was so broken that you couldn't take a single aggregate finding seriously.

The reports findings are predicated on case studies. The sampling for those studies was wholly self-selected: people who submitted cases to Jakob's site. People are unlikely to submit a failing case, which obviously skews the findings. Anyone with a financial sense would see through this, and the report's thesis will thusly be discredited.

What I found quite revealing was a small detail that spoke volumes. Each case study features a "return on investment metric" which states the percentage improvement of a key metric (sales, traffic, downloads, etc.) The problem is, we never learn the cost necessary for achieving this improvement. And you can't calculate a "return" if you don't figure the initial investment. All we know is how much it improved, not what it cost to get there.

And we're supposed to take to heart this report's findings on "Usability Return on Investment"?

I'm frustrated with this report mostly because the user experience profession needs research and analysis that demonstrates its value, and the Nielsen/Norman Group's prominence means many people outside our profession will look to them, and when they read this misleading report, they might dismiss outright the contribution of user experience professionals.

Enough griping. I hope you enjoy the review for the contribution it makes, particularly in the last third or so, with suggestions for steps we can take to better understand our value. It was a great experience writing this with Scott, a newly minted MBA from Haas, who has the ability to frame business arcana in such a way as to make it digestible to mere mortals such as myself. This was one of those experiences where either of us could have written a decent article, but only together could we have written one this good.

Posted by peterme at 08:59 AM | Comments (13) | TrackBack


See Me Travel
December 2006
November 2006
October 2006
September 2006
August 2006
July 2006
June 2006
May 2006
April 2006
March 2006
February 2006
January 2006
December 2005
November 2005
October 2005
September 2005
August 2005
July 2005
June 2005
May 2005
April 2005
March 2005
February 2005
January 2005
December 2004
November 2004
October 2004
September 2004
August 2004
July 2004
June 2004
May 2004
April 2004
March 2004
February 2004
January 2004
December 2003
November 2003
October 2003
September 2003
August 2003
July 2003
June 2003
May 2003
April 2003
March 2003
Archives from June 13, 2001 to January 2003
Archives from before June 13, 2001
Recent Entries
The Most Important Usability Article I've Read This Year
The Slings of Boxes and Arrows -- Persecuting Jakob
Subscribe to my feed:
Powered by
Movable Type 3.2