• Informal ---------------------------------------------------------------------------------------- Formal
• More interaction ------------------------------------------------------------------- Less interaction
• Series of quick pointed studies ------------------------------- Experimental methodology

Informal, iterative approach to testing with experimental rigor at its core

Purpose of usability testing is not to formulate and test specific hypotheses but rather to improve products.

- Rubin, Handbook of Usability Testing, 1994

• Elements of Usability Testing
  

• Development of test objectives or problem statements
• Representative sample of end users
• Some representation of actual work environment
• Observations of end users in using or reviewing product as well as questioning, probing participants view
• Collection of quantitative, qualitative and preference measures

Types of Tests

Exploratory:

• early in development cycle
• examine or explore effectiveness of preliminary design concept
• explore high-level design issues
• verify your assumptions about users
• involves extensive interaction with participants using preliminary versions of interface, materials, prototypes, mock-ups
• only need to show enough functionality to address particular test objective
• determine "horizontal" or "vertical" representation of functions for testing
• user performs representative tasks
• exploration of users' thought processes through think alouds
• seek why user performs as he or she does
• emphasis on discussion and examination of high-level concepts and thought processes

Assessment:

• early or mid-way in development cycle, after high-level design or organization is established
• examine or investigate how effectively the concept has been implemented
• determine how well a user can actually perform realistic tasks
• identify specific usability deficiencies
• user performs tasks with less interaction by monitor
• less emphasis on thought process and more on behaviors
• focus on quantiative data

Validation:

• late in development cycle closer to release of product
• certify product's usability
• compare product to some usability standard (performance, competitor, historical)
• evaluate how all components of a product may work together (documentation, software, hardware, etc.)
• attempt to anticipate major deficiencies to prepare for or fix prior to release
• little or no interaciton with test monitor
• collection of quantitative data with reasons for substandard performance

Comparison:

• can be used in any point in the development cycle to compare interface styles, compare effectiveness of elements, compare products against existing products
• side-by-side comparison of two or more alternative designs
• range of informal exploratory to experimental-control group designs varying along a single dimension
• observational, qualitative study, alternatives may vary on several dimensions
• best designs turn out to be combination of alternatives with the best aspects of each design used to form a hybrid design
• for best results, use dramatically different alternatives rather than similar alternatives
• allows for revisiting fundamental premises about an interface, resulting in design that redefines and improves product
• participant is forced to consider and contemplate why one design is better than anther

• Usability Criteria
  Usability objectives usually stated in performance criteria - such as speed and accuracy, how well or how fast user can perform various tasks and oeprations. Or objectives may be stated in terms of preference criteria, achieving a particular ranking or preference criteria.

• The Test Plan
  Provides a blueprint that describes exactly how you will go about testing the product. Serves as communication vehicle among team. delineates exactly what will happen and when, provides a focal point for systematic testing

Purpose - what are the reasons for performing the test at this time

Problem statements/test objectives - focuses the research, need to be precise, accurate, clear, and measurable (observable) as possible

User profile - determine target population that you will draw from and selected participants

Method - how the test session will unfold with participants, enables others to understand and visualize what will happen, allows several monitors to conduct the test in a similar manner

 

Ensuring Rigor

• Less formal usability test - four to five participants will expose 80 percent of usability deficiencies of a product and most major problems
• Consistency in sessions -using scripts, checklists, etc.
• Confirm characteristics of audience - questionnaire to verify correct characteristics
• Note problems - note what occurred and decide later if results are applicable
• Specific goals - search for specific deficiencies, flaws or problems
• Conduct a pilot test - test your processes
• Keep it simple - attain meaningful results from a smaller, simpler study
• Testing environment as realistic as possible - try to mimic the actual working environment