TwinTide Newsletter N.4, May 2013
The Number 4 of the TWINTIDE Newsletter is available for download at here. The current issue is dedicated to the TwinTide Training School TUTOREM which was organized in Autumn 2012, in Bled, Slovenia.
Short Term Scientific Mission (STSM)

STSMs are exchange visits of researchers to the institutes of the community members to perform some specific studies or experiments, thereby enabling mutual enhancement of expertise and experience.

Procedure for applicatons

Are eligible applicants for STMS postgraduate students, post-doc fellows and faculty members of a partner institution of the Action having signed the MoU.

Any partner institution of the Action having signed the MoU a non-COST country (exceptional cases).

STMS usually are supported from 1 week (5 working days) to 3 months.

How to Apply

Before to apply for a STMS, please contact the Action coordinator Effie Law by sending your CV, the work plan describing the goals of the STMS and budget requested.

Please check the STMS Guidelines from COST official Web site for further information.


Past Short Term Scientific Mission

Marianna Obrist, Marie Curie Fellow at the University of Newcastle, United Kingdom -
- -

Topic: Narrative Approaches for User Experience Research

Host: University of Copenhagen, Denmark, hosted by Prof. Kasper Hornbæk and Dr. Alexandre Tuch

Throughout the one week STSM in February 2013 we shared and discussed ideas about different analysis approaches for narratives including linguistic research and the exploration of the structure of narratives, linguistic inquiry and word count approaches including automated tools such as LIWC, and entropy classifier based on a machine learning approach. These quantitative approaches are further extended with approaches from constructivism and interpretative phenomenology, fostering the holistic understanding of experiences.

The STSM included the following activities:

  • Exchange of knowledge on different approaches on narratives including sharing of previous work on quantitative and qualitative research.
  • Elaboration of a joined project idea on exploring users experiences through narratives in a large and small-scale approach, involving quantitative and qualitative data.
  • Planning of future publication activities, especially a joined publication based on the project emerging from (b) for CHI 2014.

Overall I have widely profited from this STSM by strengthening my knowledge on quantitative narrative approaches in user experience research and its relation to qualitative strategies and practices.

Darko Bozhinoski, FON University, Macedonia -

In October 2011 I went to Cyprus University of Technology for a month and a half long STSM at the Laboratory for Human Computer Interaction and Inclusive Design led by Dr. Panayotis Zaphiris.

The general purpose of my visit was to test the usability and learnability of the structure of the information provided in the WORLD project that I was working on. WORLD (WORLD RANDOM LINKED DATA) is a cross-sector prototype of a web based service which provides information using the open data of World Bank and generates random information in the form of sentences with a combination of pictures, videos and maps. Using the World Bank database as well as the United Nations Human Development Index, the service generates intelligent information as a step towards reaching the Millennium Development Goals. As an attempt to obtain further development of the prototype in the direction of its usability across different sectors, and refine the interaction, the current status of the solution needed to be tested in one of its practical implementation of this web service: the Moodle e-learning system (the most used E-Learning platform around the world). It is a Moodle block that uses the data from the web service from areas that are within the scope of Millennium Development Goals. With the collaboration and supervision of Dr. Panayotis Zaphiris in the first week of my visit at Cyprus University of Technology I learned to operate a RED 500 SMI eye tracker. Once I'd learned how to design a controlled experiment and how to collect and analyze data with the eye-tracker, I proceeded creating the experiment. The participants in the experiment were the students from Cyprus University of Technology as the most suitable group of people that had experience using the Moodle system.

The experiment went as expected with no unusual events that would have introduced an error or lost data. During the experiment a large volume of data was collected. The results were displayed and discussed in three different sections. Analyzing the results from the experiment we've seen the negative and positive aspects of the WORLD Block and we've come to the conclusion that this solution presents a good approach for displaying a lot of new information from different areas of interest in a small section, but we should continue working in the structure of the Block to achieve our final goal - making a generator of intelligent information that will:

  • Provide positive user experience
  • Be used by any web application
  • Provide efficiency and effectiveness in gathering and presenting the information

This month and a half spent at Cyprus University of Technology (CUT) was very rich in terms of the knowledge gained from the discussions with researchers whose background was in HCI and also provided a very good opportunity of discussing with many people coming from various backgrounds and horizons and confronting opinions about various topics in HCI.

During the WORLD Usability Day I had the opportunity to participate in the exhibition as a member of the research team at Cyprus University of Technology demonstrating the eye tracker to all attendees and presenting my project to them. It was an honor for me to participate in the whole event that was really well organized by all the researchers at CUT.

Working on the development of cross-sector methodologies of Web interaction using open data as a part of my STSM at Cyprus University of Technology was very valuable for my studies at FON University and I would like to thank COST IC0904 TwinTide for making this possible.

Ioana Gghergulescu, IRCSET Post Graduate Researcher School of Computing, National College of Ireland, Ireland -

Towards Objective User Experience (UX) Evaluation Methodologies in Educational Games

Due to the high success of gaming among young people, educational games started to be used in the learning process. Educational games proved to be excellent learning tools, supporting learning by doing, supporting learners in developing and acquiring new skills. However, as compared to entertainment games, the area of educational games is not mature. Major challenges that still need to be overcome in educational games include reducing the development cost and designing assessment methodologies (e.g. user experience, player motivation, game effectives, etc.). Currently, there is a lack of non-invasive objective evaluation methodologies for educational games.

During the three weeks research visit at University of Leicester in September 2011, I and Dr. Effie Law aimed to approach these challenge by investigating eye-tracking metrics that can contribute toward the development of objective evaluation methodologies in educational games.

For this purpose, data from a previous evaluation experiment of a digital educational game with children aged 10-14, was analysed. Additional tests were performed in order to increase the number of participants. All the studies took place in the university's usability lab. The activities that a participant had to complete were: fill in a background questionnaire, complete a domain-specific learning questionnaire, play the educational game, complete the same learning questionnaire, and fill in a short usability and user-experience questionnaire. During the game play, the participants' eye-movements were captured using a Tobii T60 eye-tracker.

Two approaches for analysing eye-tracking data were considered: Areas Of Interest (AOI)-based Visual Attention Analysis Approach and Grid-based Visual Attention Analysis Approach. A meta-analysis of different fixation metrics (e.g. mean fixation duration, maximum fixation duration, median fixation duration, number of fixations, fixation frequencies), was conducted for each of the two approaches in order to investigate the implication of the different metrics.

Understanding how the visual attention is distributed over the various game UI elements can reveal design issues in educational games. Furthermore, understanding where and how the visual attention was distributed can be used in a predictive manner to determine if the player understood the game play.

This STSM can represent the starting point towards further research needed in order to propose objective evaluation methodologies in educational games. Such evaluation methodologies will help both designers and educators to improve the educational games.

My cooperation with Dr. Effie Law continues beyond my short research visit in Leicester, as we are currently working to publish our results. I want to thank TwinTide and Dr. Effie Law for making possible this great learning and research experience.

Marta Lárusdóttir, Assistant professor, University of Reykjavick, Iceland, School of Computer Science -

The purpose of my STSM was to work actively on analysing data, interpreting the results and writing a journal paper describing the results from a study conducted in the software industry in Sweden. The goal of the study was to examine how user centred evaluation is practiced when using the software development process Scrum for planning the development. Information was gathered thorough interviews with 21 usability professionals, working in 14 different software companies.

The paper will be included in my PhD-thesis at the Royal Institute of Technology. This research study is done in collaboration with my supervisors, Dr. Jan Gulliksen and Dr. Asa Cajander, Uppsala University, which are also members of the TwinTide project.

The research questions examined were:

I) What methods do usability professionals use to evaluate their product?

II) Does the evaluation methods usage vary among the different professional roles?

III) What is the purpose of the evaluation?

It is notable in the study that various user centred evaluations are conducted, and that they particularly occur in phases before the implementation phases as they are often a part of requirements engineering or pre-studies to Scrum projects. Unexpected results are that the user centred evaluations used typically are empirical qualitative and very seldom empirical quantitative. The empirical qualitative user centred evaluations comprise blogs from users, lunch meetings, comments on prototypes and other informal meetings with users. Other significant results are that the analytical evaluations were all qualitative. Only three out of 21 informants conducted quantitative empirical evaluation, and those where either usability experts or Scrum manager. The interaction designers, business analysts and developers did not perform any quantitative evaluation. Hence, user centred evaluation methods as presented in much research publications were seldom used in Scrum practice. Finally, the paper presents the lessons learned and gives suggestions to practitioners and researchers examining user centred evaluations in Scrum practice.

Michail Salampasis, PhD. Associate professor, Alexander TEI of Thessaloniki, Deparment of Informatics and Marie Curie Fellow, Vienna University of Technology Institute of Software Technology and Interactive Systems - /

Study of eye tracking technique and of other evaluation methods which involve physiological measures.

Host: Alex Tuch. Human-Computer Interaction Research Group at the Department of Cognitive Psychology & Methodology at the University of Basel

The purpose of this STSM was to study eye tracking and other techniques which involve other physiological measures, and generally qualitative approaches to interaction design and usability evaluation. Further to this, another more specific aim was to study and examine the potential application of these methods within the context of web-based interactive search systems and other integrated search systems in which users can use multiple search tools and user interfaces to satisfy their information needs.

During the STSM visit the following work has been carried out:

  • Introduction and use of the eye-tracking lab and all the necessary software for analysis of the results. During the introduction examples of eye tracking studies were demonstrated and a discussion about problems, challenges and benefits of such studies were discussed.
  • Visit and introduction to the psychophysiology lab at the University of Basel. A study was carried out with the help of MSc students who volunteered to play the role of subjects during the demonstration about the use of psychophysiology measurements. The discussion which followed the presentation revealed the need and the benefits in combining the results of eye-tracking data with data related to psychophysiology measurements.
  • Presentation of the my work in modelling and analysing the behaviour of blind users using the web. The presentation was made to the MSc colloquium and a discussion was made after the presentation about Web accessibility issues.
  • Talk to MSc/PhD students. Some of the students who attended the presentation expressed their interest to learn more about the methods and the criteria that have been used to evaluate the performance and the overall UX of blind users. Potential areas and research problems which are worthy of further exploration have been identified.
  • I received further information about software and other facilities (e.g. eye-tracking glasses) and how they can be used in various settings other than an eye-tracking lab.

Description of the main results obtained

  • Better understanding from the STSM applicant of the methods (eye-tracking, psychophysiology measurements) which are used in Basel University for usability testing and UX evaluation.
  • Better knowledge about the problems and challenges in using these methods in particular in respect to the evaluation of interactive search systems. ?

Martin Mihajlov, E-business Department, Ss. Cyril and Methodius University, Skopje, Macedonia -

The two-month long STSM held in February-March 2011 at the Laboratory for Open Systems and Networks at the Jozef Stefan Institute is a complementary activity to my PhD thesis where I research usable security of recognition-based graphical authentication mechanisms. In the first week of the STSM I learned how to use an eyetracker to perform eyetracking experiments and subsequently learned how to interpret eyetracking data. I then proceeded to complete three experiments. Two of them were related to my PhD research work and one additional experiment idea.

Experiment 1: Eyetracking graphical authentication. In line with my PhD thesis resarch I needed to evaluate a proposition for a novel graphical authentication mechanism that uses images with single item content. We eyetracked subsequent uses of the graphical authentication mechanism over different periods of time in order to analyze the cognitive differences in the learning effect occurring with continuous system use. We analyzed different areas of the password selection grid to evaluate whether image selection is influenced by image position. We also tested the usability of the generic interface components and gained knowledge on user behavior when dealing with graphical authentication.

Experiment 2: Eyetracking task performance time. With the advent of eyetracking and the increased availability of both hardware and software platforms, eyetrackers have been extensively used in many usability and user experience experiments. Nevertheless, the influence that the eyetracker itself has on the experiment has never been thoroughly studied. Although modern eyetrackers are less obtrusive than strap-on devices from years past, we believe that in some aspects the eyetracking process affects variable analyzed in the eyetracking process. Specifically, this experiment tested the following hypothesis: "Task performance time is significantly affected by the eyetracking process." We repeated popular web-based eyetracking tasks with two separate control groups, the main difference between the groups being the involvement of eyetracking. The eye-tracking group performed the tasks at the E5 laboratory, while the non-eyetracking group completed the exepriment in Macedonia prior to the STSM. As this experiment has just been completed we are in the process of aggregating and analyzing the data. As essentially performance time is the only variable relevant to this study, we have discarded the eyetracking records.

Experiment 3: Ubiquitous eyetracking. With the increased popularity of mobile devices the interest in usability studies of these devices grow accordingly. Current methodologies for evaluating mobile devices and applications are: laboratory experiments, field studies, observational fieldwork, remote tracking and data logging. However, carrying out eyetracking tests on mobile devices and applications is not straightforward as eyetracking requires complex methods to overcome the limitations inherent with ubiquitous device testing. In this experiment we initially plan to test the suitability of different eyetracker set-up propositions that can be used in ubiquitous eyetracking. We will then proceed with mobile application interface usability and design testing on an application for graphical authentication developed as part of my PhD research. As a result of this STSM the cooperation between E5-IJS and the E-business department has intensified. Along with Professor Borka Jerman-Blazic we have submitted two journal papers and two conference papers. We are also preparing a proposal for a joint project.

The Laboratory for Open Systems and Networks at the Jozef Stefan Institute is led by Profesor Borka Jerman-Blazic. The main research areas of the laboratory are: technology-enhanced learning, security and privacy in information systems, next generation networks and usability testing. The laboratory is implementing the "Future internet technologies: concepts, architectures, services and socio-economic issues" research programme. It also coordinates and participates in FP7, Leonardo da Vinci, eContentplus, COST and national projects.

Marta Lárusdóttir

Jan Gulliksen

Marta Lárusdóttir, Assistant professor, University of Reykjavick, Iceland, School of Computer Science -

How are usability activities conducted by software practitioners using the Scrum development process? This is the main question that I explored in my STSM in November 2010 at the Royal Institute of Technology (KTH) in Sweden.

KTH is Sweden's premier university on engineering. The School of Computer Science and Communication has one of Europe's biggest HCI groups (45 people) and conducts research and education on User-centred design, Interaction design, Visualization, elearning, Human Robot Interaction, etc. Our TwinTide member Jan Gulliksen used to head the HCI group but has now handed over to Ann Lantz as he assumed the position of Dean for the school.

During the visit Asa Cajander (also a TwinTide member) and I interviewed 20 people in the Swedish software industry that use Scrum as their development process. The main goal of the interviews was to understand to what extent and how usability activities are integrated into Scrum. The interviewees were from 14 different companies and had 9 different job roles. All the interviews, that took around 1 hour, were recorded and transcribed.

The main subjects that we asked about in the interviews were:

  • how Scrum is used in the company;
  • how usability actitivies are integrated into the Scrum process,
  • how users are involved in the development,
  • how usability requirements are defined and
  • who is responsible for usability activities.

Our preliminary results show that because the focus in Scrum is very much on delivering functionality, it has been hard for the practitioners to find time for usability activities. Many interviewees mentioned that they gather feedback from users after each delivery in Scrum and not during the development of that functionality.

The STSM visit was very valuable for my PhD studies at KTH. I had very inspiring co-operation with both Asa Cajander at Uppsala University and Jan Gulliksen at KTH on this research project. We had many good discussions on the subject and gathered really interesting data from the interviews during my STSM in Sweden. We have a couple of papers planned for publication of this data.

Carmelo Ardito, IVU Lab, University of Bari, Italy -

In June 2010 I went to the University of Patras (GR) for a one month long STSM at the HCI Group lead by Nikos Avouris. The aim of this STSM was to start a collaboration on the theme of designing mobile and ubiquitous systems in the cultural heritage domain. In Italy and Greece there are plenty of archaeological parks and other outdoor historical sites, and both research groups have competence in designing systems supporting visits to such places. the IVU Lab has designed Explore!, an m-learning framework which implements the excursion-game, an experience planned to engage groups of young visitors exploring cultural heritage sites through a pervasive game; the HCI Group has designed MuseumScrabble, a pervasive game for students, aged 10-12 years old, to be played in a museum.

The STSM was focused on studying heuristics for designing or evaluating educational games. The first phase of this joint research was an extensive review of the literature on pervasive games for exploring cultural heritage sites (museums, archaeological parks, historical cities, etc.) and guidelines for game design and evaluation. Then, the work concentrated on the analysis of three case studies representative of pervasive games for visiting an archaeological park, a museum, and an old town center. As a result, a large set of issues relevant to such games were defined. From these issues, preliminary guidelines were proposed: 36 guidelines classified along 5 dimensions. These early results have been reported in a paper presented at the international conference "Social Applications for Life Long Learning" (SALL 2010), while more recent advances and future work are in a paper accepted to HCI International 2011.

During the STSM also dissemination activities were organized. On June 15th at the University of Patras the full day workshop "Games Narratives and their Evaluation" took place. Researchers of the Hellenic Open University and University of Peloponnese participated in this event.

The collaboration between IVU Lab and HCI group that was initiated during this visit is still going on. We are planning a more systematic study in order to involve a wide number of designers in the validation of the proposed guidelines. To this aim a website is being created; it reports the motivation of the research, the adopted methodology, the guidelines and the dimensions identified. In this site the guidelines are presented and concrete examples of their use are included, while visitors can post comments on them. In order to induce people to reflect more deeply on the guidelines, for each guideline a set of simple questions is provided.

In addition, in the frame of the International Symposium on End-User Development (IS-EUD 2011) we are going to organize the workshop "DEG: Involving End Users and Domain Experts in the Design of Educational Games", in which we invite researchers and practitioners involved in designing and evaluating technology supported games to discuss their experience in relation to means for involving the end users in the development process, during and after the product has been completed.

My STSM at the University of Patras was very productive and I would like to thank TwinTide for making this possible. I'm very glad to Nikos and the whole HCI Group for the warmest welcome and pleasant stay in Greece.