WG 4: Review on the Computational and Definitional Approaches in Usability Evaluation

Coordinator: Christian Stary - University of Linz (AT)

Interested partners: Jean Vanderdonckt, Marta Lárusdóttir, Quentin Limbourg, Dominique Scapin, Philippe Palanque, Christian Märtin/Jürgen Engel, Peter Eberle, Effie Law, Marco Winckler, Niels Ole Bernsen


The primary objective of this Activity is to review systematically the existing models and procedures for estimating certain key usability test parameters and traditionally defined usability quality metrics.

Based on the insights gained from the comprehensive review, we aspire to find out how to compute accurately the problem-discovery rate (p), inter-/intra- evaluator concordance, and appropriate sample size of test participants. Alternative mathematical/ statistical modelling and algorithms are developed to compensate for any drawbacks thus identified. The results of Activity 2 are expected to shed some light on this issue. Furthermore, approaches to evaluating the three classic usability metrics will be examined. Specifically, “effectiveness” and “efficiency” as the function of (unassisted) task completion times and “satisfaction” as the function of users’ subjective and often retrospective ratings may not be enough to capture the multifaceted features of usability. Hence, these definitions will be ameliorated and augmented to accommodate the increasing complexity of user interface designs. In this Activity, we will also study the relationships between internal, external, and in-use metrics that pertain to usability with the use of sophisticated statistical models or other computational techniques (cf. results of Activity 2).

Expected Outcomes: With the improved models and algorithms, more accurate estimations of key usability test parameters can be obtained. These results are particularly important for practitioners to optimally arrange usability evaluation (e.g., iterative or one-shot, how many users/evaluators to be involved, etc.) with allocated budget. Besides, the three revised usability metrics can serve as better indicators about system usability, especially for evaluating complex user interfaces. Furthermore, usability specialists and software engineers are enabled to collaborate more effectively if the relationships among the quality attributes upheld by each of these two professional groups can be concretely represented with a robust statistical model.

Note that how wide and deep individual topics mentioned in the foregoing descriptions will be dealt with is contingent upon the ongoing progress of the Action. Foci of the R&D works will be adjusted when necessary so as to divert efforts to meet more urgent needs and to exploit more promising areas.