WG 1: Critical Review and Analysis of Individual UEMs

Coordinator: Dominique Scapin, INRIA-Rocquencourt (FR)

Interested partners: Oscar Pastor, Asbjørn Følstad, Reinhard-Sefelin (V. Seibert Giller), Erik Frøkjær, Macin Sikorski, Jiri Kubalik, Simos Retalis, Marta Lárusdóttir, Jan Gulliksen, Timo Jokela, Leena Norros, Gilbert Cockton, Quentin Limbourg, Dominique Scapin, Philippe Palanque, Christian Jetter, Effie Law, Marco Winckler, Laila Dybkjær

The primary objective of this Activity is to build a refined, substantiated and consolidated knowledge-pool about usability evaluation, based on the expertise, experiences, and research work of the participating project partners. The UEMs of interest are subjected to critical review and analysis on the following aspects:
  • underlying theoretical background, if any; alternative theoretical frameworks that can better explain the effects of UEMs may be proposed;
  • scope of application: as indicated by the range of conditions and contexts, and the development phases in which individual UEMs have been deployed;
  • cost-effectiveness:
    • costs are primarily defined in terms of person-hours involved;
    • definition of effect per se is a problem to be researched; individual project teams may define this attribute in their specific way; the various definitions thus collected can serve as reference point to initiate the discussion (see also Activity 2);
  • user/evaluator effect: Its extent can be measured with the deployment of existing methods (e.g., Monte Carlo simulation, any-two agreement measure, etc.). As a corollary, individual project teams can evaluate the validity of these methods and even develop alternative ones (see also Activity 4);
  • limitations of the techniques/tools involved; individual UEMs rely on a specific technique to realize their functions, for instance, thinking aloud in case of user tests, a set of principles in case of heuristic evaluation, etc. However, there are inherent limitations in these techniques/tools, and the individual project teams are to identify such limitations and even remedial strategies;
  • level of acceptance: To assess with existing technology acceptance models whether individual project team members tend to adopt the UEMs under investigation in their future works.
  • possible extension: To examine whether individual UEMs can be extended to increase its scope of applicability and effectiveness.

Expected Outcomes: A database is created to record the findings of individual project teams involved in this Action with regard to the attributes of different UEMs as described above. These data serve as important reference points from which insights into the future development of UEMs can be gained.

Tool support for Method Analysis [ login is required ]