peterme.com

Today we posted Scott Hirsch's essay on ROI Is Not a Silver Bullet: Five Actionable Steps for Valuing User Experience Design.

This coincides with the release of Adaptive Path's first serious research report, Leveraging Business Value: How ROI Changes User Experience, written by Scott, Janice, and Sara Beckman from Haas School of Business. This report marks a significant turning point in how we user experience types recognize our value to the enterprises in which we work.

Up until now, the single best source for valuing user experience is the book Cost-Justifying Usability. Considering it was published over 10 years ago, that's either a testament to the book's brillance, or the remarkable paucity in thought and discourse on the subject in the intervening years. (I suspect a combination of both.) When we've discussed the book with others, particularly those in bigger companies, we realized that there's vast skepticism toward its daisy-chaining-numbers-approach, where benefits are figured out by saying that if you can reduce user's time on a screen by 5 seconds, and that screen is seen X times by Y people being paid Z dollars per year, then you multiply everything and voila, there's your expected benefit. Perhaps in certain highly controlled processes (like repetitive data entry) this kind of calculation is possible and meaningful, but typically this just leads to hypothetical results that aren't verifiable. And if you try to bring such reasoning to your finance folks, they'll roll their eyes.

What we've seen that works are more macro, bolder, simpler, and more realistic claims. Something direct like, "This will increase registration by 40%." In one of our client companies, they bought a CMS. In trying to get the money for it, they soon found out that a cost-justifying-usability argument about productivity (users will spend 10 fewer minutes every day finding information, saving $n!) meant nothing. But when they could say, "This CMS will eliminate the need for printing and distributing documentation," that resonated, because "printing and distributing documentation" was a clear line item on a budget, and the financial person could immediately realize the benefit of crossing that line out.

This report is different from other attempts at addressing the ROI of UX question in that it doesn't make specious industry-wide claims about UX investments, nor does it try to "value" UX methodologies. In fact, for the report, design methods are a "black box" - it's impossible to say that card sorting returns x%, or that iterative prototyping returns y%. The foundation of the report are 5 in-depth case studies (Bank of America, KQED, Cathay Pacific Airways, Belkin, and ESPN.com) of how companies are actually valuing user experience design. The results have less to do with specific returns on investments, and a lot more to do with organizational maturity, and processes that ensure that UX teams are appropriately valued. In the end, this report feels much richer, and much truer, than what I've seen that has gone before.

Here are some of my favorite quotes from the report:

Generalizations such as this [10% investment in usability leads to 168% ROI] fail to take into account the project, business model, industry, and other factors that are important to the financial accounting people who are used to making decisions based on ROI. Bottom line, ROI is an internal tool for comparing investments, not a proscriptive guideline — generalizations are meaningless and unconvincing.

This, of course, rings of the critique Scott and I leveled at the Nielsen/Norman Group's Usability ROI report. Providing industry-wide "ROI" numbers is pernicious -- they're the Holy Grail of this research, but in reality are practically impossible to derive. One of the best aspects of this report is to set aside the search for this grail and focus on things that are addressable.

Based on our observation, establishing well understood selection criteria [in determining projects] and ensuring transparency in the final decision are vital. If project sponsors do not understand the rationale behind selection decisions, they are less likely to propose “fundable” project opportunities. Similarly, if project selection seems arbitrary, they are less likely to view user experience as a strategic resource available for their use.

For me, perhaps the single most resonant finding of this report involves project ideation, prioritization, and selection. So often, with our clients, with our colleagues, with anyone we talk to, the projects they work on seem to have been chosen by dart-throwing. Or executive fiat (we had one project that started because a senior executive wanted "fewer links on the home page.") And nothing makes a project team feel like their twisting in the wind than arbitrary decision making processes that end up with them expending time and money for an unclear gain.

One subject firm in the report, Cathay Pacific Airways, has a remarkable project prioritization and selection process that's worth emulating. A boatload of explicit project attributes, each of which are assigned a value, leads to an obvious and transparent decision.

Control of pipeline. The Web team controls its own development pipeline by managing a transparent process for selecting projects that is well understood by internal clients.

This is a bullet point from a larger list, dealing with what needs to happen to make Web teams effective. They need some measure of independence, so they're not subject to the whim of others. They need to control their pipeline. Such control should be okay, if they make it explicit to others why they've opted to work on some projects and not on others. If it's explicit, those others can't really complain if they're project isn't chosen, because the reasons should be clear. Without this control, the web team becomes pawns whose time can be wasted on thoughtless projects.

An additional layer of complexity is added when the analytic process itself has a significant cost. A few firms commented that in some cases they would like to apply more rigorous ROI methodology, but the data to determine business value is too difficult to analyze given their current technology and staffing constraints. However, these limitations do not negate the use of ROI methodology. In most cases, some linkage between user behavior and business value can still be made, even if the quality or granularity of the data is not ideal. Instead, teams can establish user experience metrics that are likely to indicate business impact. Such hypotheses regarding the quality of indicators can then be tested.

Getting good metrics is so difficult that many companies just throw their hands up at the prospect. Or because they don't have the metrics now, they feel they can't start measuring value. These obstacles are more mental than real, though. The reality is you've got to start somewhere. If you're not collecting these metrics, that's okay. Just start doing it now. If you can't get all the metrics you think you need, that's okay -- find a few that seem relevant. Start small and take it from there. The point, though, is to *do it*. Don't let some idea that you can't do it "right" mean that you don't do it at all. It's a matter of degrees, not right and wrong.

Again, an ROI calculation is always going to be a guess. The value of the analytic process is not diminished by the difficulty in drawing statistically significant correlation between user experience and business impact — the greatest value comes over time, as the ability to make more-accurate estimates increases.

This is kind of a corollary to the previous quote. But that first sentence is important. Folks like me assume that the business/finance people require certainty, and that puts us in a panic, since, well, it's hard to be certain about potential effects of user experience. The thing I've learned, though, is that the business/finance people are okay with some squishiness. They want an educated guess, but they don't need precision to 2 decimal places.

To get more of a taste, Adaptive Path has made the report's executive summary freely available. It captures just how the report is reframing the issues, away from specious statistical claims towards a richer, more interesting take on the reality of organization and how they evolve. If that first taste tantalizes you, use the coupon code BVDEAL and get 15% off the cost of the report.

Posted by peterme at July 21, 2004 09:12 AM

Comments

Posted by: Rani at July 25, 2004 09:05 AM

Posted by: christina at July 27, 2004 08:52 AM

Posted by: Klaus at September 1, 2004 04:40 PM

Posted by: sexy lingerie at October 11, 2004 10:25 AM

Posted by: stasio at October 26, 2004 03:52 AM

Posted by: stasio at October 26, 2004 05:28 AM

Posted by: 5t453k at October 26, 2004 08:19 AM

Posted by: dianne at March 31, 2006 12:16 AM