About the COST Action 294: MAUSE

Executive Summary by Effie Lai-Chong Law, ETH Zürich(Chair) and Ebba Thora Hvannberg, University of Iceland (Vice Chair)
Last updated on: February 19, 2005

Mission

The ultimate goal of MAUSE is to bring more science to bear on Usability Evaluation Methods (UEM) development, evaluation, and comparison, aiming for results that can be transferred to industry and educators, thus leading to increased competitiveness of European industry and benefit to the public.

Objectives

The main objective of the COST 294 is threefold:

  • to deepen the understanding about the inherent strengths and weaknesses of individual Usability Evaluation Methods (UEMs);
  • to identify reliable and valid methods to compare different UEMs in terms of their effectiveness, efficiency as well as scope of applicability;
  • to develop efficacious strategies for extracting useful information from the results of UEMs to improve the system tested;
Background

We have identified opportunities for research that we think is currently lacking in the field of usability evaluation, including:

  • a sound theoretical framework to explain the phenomena observed;
  • a set of empirically based and widely accepted criteria for defining usability problems;
  • a standard approach to estimating values of key usability test parameters;
  • effective strategies to manage systematically the user/evaluator effect;
  • a thoroughly validated defect classification system for analysing usability problems;
  • widely applicable guidelines for selecting tasks for a scenario-based usability evaluation;
  • a sophisticated statistical model to represent the relationships between usability and other quality attributes;
  • a clear understanding about the role of culture in usability evaluation;

Clearly, the above list is not exhaustive, and the endeavour of a single research team cannot address all the issues identified. Instead, extensive co-operations among a community of usability professionals and international project teams are indispensable. The diverse backgrounds and different foci of their works are complementary and beneficial to the fulfilment of the Objectives of the current initiative.

Scientifc Program

Four major R&D activities will be implemented by of four Working Groups (WGs):

WG 1: Critical Review and Analysis of Individual UEMs
The primary objective of this Activity is to build a refined, substantiated and consolidated knowledge-pool about usability evaluation, based on the expertise, experiences, and research works of the participating project partners. Each of the UEMs of interest will be analysed in terms of a selected set of relevant attributes.

WG 2: Comparing UEMs: Strategies and Implementation
The primary objective of this Activity is to identify effective strategies to compare UEMs. Besides, it is relevant to conduct close observations on how evaluators build consensus when they are confronted with discrepant results. Note that empirical, analytic and model-based UEMs will be investigated.

WG 3: Refining and Validating Classification Schemes for Usability Problems
The primary objective of this Activity is to improve and substantiate the tools for analysing usability problems. Specifically, the recently developed scheme known as Classification of Usability Problems (CUP) will be refined and validated.

WG 4: Review on the Computational and Definitional Approaches in Usability Evaluation
The primary objective of this Activity is to review systematically the existing models and procedures for estimating certain key usability test parameters and traditionally defined usability quality metrics. Specifically, we aspire to develop alternative means to compute accurately the problem-discovery rate (p), inter-/intra- evaluator concordance as well as appropriate sample size of test participants, and to examine the relationships between internal, external and in-use metrics that pertain to usability with the use of sophisticated statistical models.