USABILITY ISSUES IN DECISION SUPPORT TOOLS: PART II
by Ed Colet

Decision-support tools such as data mining applications are complex solutions designed to address important business issues. Last week in part I, I looked at some of the issues in development that can contribute to these systems being hard to use. In Part II, I address how to evaluate and correct usability problems before products or solutions are released.

User centered design (UCD): An effective design philosophy:

As the term suggests, the underlying philosophy behind UCD is the notion of the user as central to the design process. UCD can also extend beyond the design phase through the duration of the user's interaction with the product after release or purchase. The philosophy is to make the design/system fit the user, as opposed to expecting the user to adapt themselves to the design.

In addition to the principle of a consistent and early focus on the needs of the end-user, UCD involves 2 other general principles. One is an emphasis on explicit ways to gather behavioral measures that indicate ease of learning and ease of use. These might be "time to completion" or "number of time 'Help' is requested." The other principle is the use of a highly iterative process of design, modification and testing. This involves a willingness to re-design and shape the product both during the early phases and throughout the design process. UCD is a concern with usability that pervades the entire process, rather than merely an afterthought late in the development cycle. If the only opportunity for changes and revisions occur late in the development cycle, then changes tend to be minimal and cosmetic rather than fundamentally contributing to a better and more usable product.

Implementing UCD throughout the development cycle involves a variety of techniques briefly described here. (More details can be found in Norman & Draper's, "User Centered System Design: New Perspectives on Human-Computer Interaction", and several of the research papers of John Gould). The order in which these techniques are presented correspond to the stages of development prior to a product's official release.

Participatory design:

UCD suggests putting one or more representative end-users on the design team itself. For decision support or data-mining software, this would mean putting a trained statistical analyst on the design team or an business executive if he/she were the intended end-user. The former is fairly typical today because many of the same people developing new algorithms and analytical techniques are also involved in developing data mining software. But as some end-users are expected to be the managerial decision-makers, it would be important to involve them on the design team as much as is feasibly possible.

Focus group research:

It's important that the concepts and ideas put forth by the representative end-user on the design team be evaluated by a peer end-user group, and the use of focus groups to collect detailed feedback is used for this purpose. Less detailed feedback but from a larger sample can be gathered through use of surveys. Note that in terms of software development, no code need be written yet as concepts and diagrams are what are discussed and evaluated. Programming and evaluating a prototype occurs in the next stage.

Design walk-through:

This occurs once there is an early prototype available to be explored. Sample tasks or scenarios are presented, and the end-user's progress and steps to complete the task(s), and their concerns are recorded and evaluated. These may involve business case scenarios, sample analysis of available data, and other tasks that users walk through.

Expert evaluation:

This is a formal review of the product by a usability or human factors expert who has not been involved in the project prior to this point. The purpose is to evaluate the system in terms of general usability principles, and adherence with accepted and common norms established in literature or research. It is similar to a usability audit, in which the product's design is compared to a checklist of standards for usability criteria established in the literature and/or the organization.

Usability testing:

This involves techniques to collect data while observing representative endusers doing a variety of representative tasks. These tests can be formally designed experiments to test specific hypotheses, or a less formal approach of structured but open-ended observations of users. The goal is to expose usability deficiencies (e.g. difficulties users may have in trying to run a particular analyses, or generating certain types of analytical reports) and refine the product further before official release.

Each of the above techniques are used at the various points of the design process prior to official release of the product/tool. And even after the product is released it is still important to collect feedback and further refine the product for subsequent versions. Careful analysis of results of each technique, and the process of iterative design will contribute to a usable and useful product for the intended end-users. In the data mining arena the consequences will be better decision-support tools and better decision-making.

Ed Colet is the Acting Director of Research at Virtual Gold Inc., responsible for developing analytical methods for data mining and for investigating human factors and usability issues of business intelligence systems. At present, he is in the final stage of completing a doctoral dissertation in the Cognition and Perception program at New York University's Department of Psychology. Ed has also worked for IBM Research at the T.J. Watson Research Center. At IBM, Ed was a member of the group that developed Advanced Scout, the data mining application for NBA teams. His research interests focus on statistical methods and human factors.

For more information, see www.virtualgold.com.