Célia Martinie

Les UIDLs (User Interface Description Languages) sont des langages conçus pour faciliter la conception des interfaces utilisateur. Ils permettent de se concentrer sur le développement de l’interface utilisateur sans se préoccuper du reste du programme, tout en offrant une syntaxe adéquate àleur description. Cependant, ces langages sont utilisés dans des domaines critiques, tels que l’aéronautique ou le domaine médical, alors qu’ils ne permettent pas, en l’état, d’apporter les garantiesnécessaires pour ce type d’applications critiques.Dans cette thèse, nous nous questionnons sur les UIDLs spécialisés dansla description des interfaces graphiques et leur utilisation dans les contextes critiques. Notre approche porte sur l’étude de la sémantiquede ces langages et de leur formalisation. Les sémantiques des UIDLs ontpeu été étudiées dans la littérature et pourtant, leur formalisation pourrait permettre de vérifier l’ensemble des interfaces descriptibles. Nous présentons des propriétés communes aux UIDLs pourenfin nous questionner sur la façon de les formaliser. Pour répondre àcette question, nous proposons d’utiliser les bigraphes de Robin Milner,un formalisme mathématique permettant de modéliser un système évoluanten espace et en temps. Nous montrons que la théorie des bigraphes estadéquate pour la formalisation de la sémantique des UIDLs et définissonsun UIDL ayant pour fondement théorique les bigraphes. La définition d’untel UIDL permet son utilisation en tant que langage intermédiaire pour la compilation d’autres UIDLs et, par son intermédiaire, de pouvoir vérifier des interfaces graphiques. Nous essayons notre approche encompilant le langage Smala, un UIDL utilisé dans le domaine de l’aviation, vers l’UIDL défini et en vérifiant certaines propriétés sur des exemples d’interfaces.

The word ``user experience'' encompasses the entire design area. User experience guides design decisions. It informs us which attributes of designs suit and which attributes do not support user needs. While design as a discipline is well studied, the user experience as a discipline got attention only in recent decades. The user experience, just like any scientific field of inquiry, requires systematic methods and studies to gain authentic insights into user needs and preferences. This thesis fills this gap and examines user experience under a significant theory of human actions, scientifically called the theory of affordances. The original contribution to knowledge is the development of the synthesis that, user evaluation process must be based on both Gibson and Norman's approach to affordances. Both approaches to affordances explain user interactions from cognitive and environmental dimensions. Such as affordance derived from cognitive capacities and the affordance that the environment encourages. Further, this synthesis is operationalized in a user evaluation methodology. The methodology involves identifying user affordances through focused conversation-based interviews and further evaluating on Gibson and Normans' approach to affordances. This user experience evaluation method applies to any design domain. As a result, anyone who wants to evaluate a design's desirability can utilize this method. The thesis considers different design artifacts to demonstrate the applicability of the proposed method in various contexts. The thesis examined the developers' affordances with their workspace artifacts and overall experience. Such as the developers' perceptions of the ideal and effective workspace. In another study, the method is applied to examine the requirements elicitation documentation. Which attributes of the documentation were desirable and practical for developers to proceed with the implementation further. The method is also applied for game design analysis and corresponding user experiences. The results revealed players' perceptions about design components and how much the design communicates the play mechanics without the interference of rule books. The thesis asserts and proves how important it is to employ affordance theory to understand user experience. The work operationalizes this theory on various design artifacts, including games, workspace, and documentation. As a result, the thesis application extends beyond classified design domains and lays the groundwork for the discipline of user experience evaluation.

Human-machine interaction (HMI) has recently started to benefit from the rise of neuroergonomics and physiological computing. Indeed, these new research fields provide both a new approach and a new means to evaluate the quality of HMI based on the assessment of the user/operator’s mental state through machine learning applied onto physiological metrics. Although growing at a fast pace, research in this domain is still hindered by several challenges. Amongst those, one can list reaching high detection accuracies in realistic and ecological settings, finding appropriate counter-measures (i.e. system adaptation/modifications) to increase safety and performance, but also better defining mental states of interest. This thesis presents selected work that aim at addressing these challenges. Hence, after the first part - part I- that details my academic resume including my educational and research activities, my communications and supervisees, the second part - part II- is dedicated to detailing two main research tracks I have followed these last 7 years. The first track - in chapter 7- presents cognitive neuroscience contributions for improved operator mental state assessment. This includes work on mental states’ definitions and concepts, on characterizing user/operator engagement level depending on task demands, prolonged operation, cooperation and confinement setting, as well as designing countermeasures for adequate engagement. The second track - in chapter 8- presents methodological and signal processing contributions for improved physiological computing. This includes work on probes and their dedicated processing, cerebral connectivity features, work on peripheral measures such as cardiac ones, as well as work related to actually closing the loop and out-of-the lab progress. This research work is of course not finite and opens new questions. Hence, chapter 9 details research perspectives on physiological computing as a neuroergonomic tool, by focusing on projects that I recently initiated on human-robot interaction in collaborative and mobile settings (incl. studies on electrophysiological inputs for collaborative robotics enhancement and the impact of teleoperators’ orientation and movement for space applications) and on how to go further to cope with EEG non-stationarity (incl. open science, exploring mathematical tools for extracting new EEG features, and large scale projects for collaborative work), as well as reflections on how to maneuver a research practice shift with fundamental and clinical considerations. Lastly, the third part - part III- is comprised of five articles that I selected to present diverse aspects of my work : one article on passive brain-computer interface (BCI) in realistic settings, a second on a comprehensive neuroergonomic approach to inattentional deafness, a third on physiological computing for a hyperscanning application written with my first PhD student, a fourth on a review and perspective of physiological computing benefits to human-robot interaction, and lastly a fifth on a recently organized pBCI competition.

Domain knowledge is a prerequisite to produce software design and implementation tailored to stakeholders’ requirements. One common way to formalize that knowledge is achieved through conceptual models, which are commonly used to describe or simulate a system. Acquiring such expertise requires to discuss with knowledgeable stakeholders and/or to get an access to useful documents, which both might not always be easily accessible. In the same time, more model samples can be gathered from multiple sources, what represents an increasing number of already formalized and accessible knowledge pieces. For example, some companies keep archives of internal model repositories. There also exist numerous open source projects that contain models while some modeling tools even o↵er the possibility to create public projects that are free to browse. Such data sources could be exploited to create domain knowledge that could be provided to software engineers while modeling. To be useful, this knowledge must be of high quality, but must also be well integrated into the software modeling process. The focus of this thesis is to provide a framework to exploit knowledge to assist users of computer-based modeling tools with software modeling assistants. This thesis first introduces our research questions based on a systematic mapping study about software assistants for software engineering, and then focuses on software assistants for modeling. It reports on the design of modeling assistants based on a user-centered approach. We present the conclusions of interviews conducted with experts in modeling, a stage in which requirements are collected. Then, we develop the creation of a prototype modeling knowledge base allowing (i) to create general and specific artificial modeling knowledge, and (ii) to make them available to any software client via recommendations. After introducing the results of an experiment regarding the accuracy of the system, we discuss these preliminary results. Finally, this thesis presents a software modeling assistant implementation integrated to the Papyrus tool, which aims to cognify the UML modeling environment by integrating the previously created knowledge. Our work helps to clarify the need for assistance during software modeling work, presents an initial approach to the design of software assistants for software modeling, and identify research challenges in modeling assistance.

Automation enables systems to execute some functions without outside control and to adapt functions they execute to new contexts and goals. Systems with automation are used more and more to help humans in everyday tasks with, for example, the dishwasher. Systems with automation are also used to help humans in their professional life. For example, in the field of aeronautics, automation has gradually reduced the number from 4 pilots to 2 pilots. Automation was first considered as a way to increase performance and reduce effort by migrating tasks previously allocated to humans to systems. This, in the hypothesis that systems would be better than humans would at performing certain tasks and vice-versa. Paul Fitts proposed MABA-MABA (Machine Are Better At – Man Are Better At), a tasks and functions allocation method based on this hypothesis. In line with this hypothesis, various descriptions of levels of automation have been proposed. The 10 levels of Automation (LoA) by Parasuraman, Sheridan et Wickens describe different tasks and functions allocations between the human and the system. The higher the level of automation, the more tasks migrate from human to system. These approaches have been the subject of criticism. « MABA-MABA or Abracadabra? Progress on Human–Automation Coordination » of Dekker and Woods highlights that automation leads to new tasks allocated to humans to manage this automation. Moreover, they recall that these approaches hide the cooperative aspect of the human-system couple. To characterize the human-system cooperation, the importance of considering, at design time, the allocation of authority, responsibility, control and the initiative to modify these allocations during the activity was demonstrated. However, the existing approaches describe a high-level design of automation and cooperation early in the design and development process. These approaches does not provide support for reasoning about the allocation of resources, control transitions, responsibility and authority throughout the design and development process. The purpose of this thesis is to demonstrate the possibility to analyze and describe at a low-level tasks and functions as well as the cooperation between humans and the system with automation. This analysis and this description enable to characterize tasks, functions and the cooperation in terms of authority, responsibility, resource sharing and control transition initiation. The aim of this work is to provide a framework and a model-based tooled process to analyze and understand automation. In order to show the feasibility of this approach, this thesis presents the results of the application of the proposed process to an industrial case study in the field of aeronautics.

In user-centered approaches, the techniques, methods, and development processes used aim to know and understand the users (analyze their needs, evaluate their ways of using the systems) in order to design and develop usable systems that is in line with their behavior, skills and needs. Among the techniques used to guarantee usability, task modeling makes it possible to describe the objectives and activities of the users. With task models, human factors specialists can analyze and evaluate the effectiveness of interactive applications. This approach of task analysis and modeling has always focused on the explicit representation of the standard behavior of the user. This is because human errors are not part of the users' objectives and are therefore excluded from the job description. This vision of error-free activities, widely followed by the human-machine interaction community, is very different from the Human Factor community vison on user tasks. Since its inception, Human Factor community has been interested in understanding the causes of human error and its impact on performance, but also on major aspects like the reliability of the operation and the reliability of the users and their work. The objective of this thesis is to demonstrate that it is possible to systematically describe, in task models, user errors that may occur during the performance of user tasks. For this demonstration, we propose an approach based on task models associated with a human error description process and supported by a set of tools. This thesis presents the results of the application of the proposed approach to an industrial case study in the application domain of aeronautics.

In a user-centered design process, artifacts evolve in iterative cycles until they meet user requirements and then become the final product. Every cycle gives the opportunity to revise the design and to introduce new requirements which might affect the artifacts that have been set in former development phases. Keeping the consistency of requirements in such artifacts along the development process is a cumbersome and time-consuming activity, especially if it is done manually. Nowadays, some software development frameworks implement Behavior-Driven Development (BDD) and User Stories as a means of automating the test of interactive systems under construction. Automated testing helps to simulate user’s actions on the user interface and therefore check if the system behaves properly and in accordance with the user requirements. However, current tools supporting BDD requires that tests should be written using low-level events and components that only exist when the system is already implemented. As a consequence of such low-level of abstraction, BDD tests can hardly be reused with more abstract artifacts. In order to prevent that tests should be written to every type of artifact, we have investigated the use of ontologies for specifying both requirements and tests once, and then run tests on all artifacts sharing the ontological concepts. The resultant behavior-based ontology we propose herein is therefore aimed at raising the abstraction level while supporting test automation on multiple artifacts. This thesis presents this ontology and an approach based on BDD and User Stories to support the specification and the automated assessment of user requirements on software artifacts along the development process of interactive systems. Two case studies are also presented to validate our approach. The first case study evaluates the understandability of User Stories specifications by a team of Product Owners (POs) from the department in charge of business trips in our institute. With the help of this first case study, we designed a second one to demonstrate how User Stories written using our ontology can be used to assess functional requirements expressed in different artifacts, such as task models, user interface (UI) prototypes, and full-fledged UIs. The results have shown that our approach is able to identify even finegrained inconsistencies in the mentioned artifacts, allowing establishing a reliable compatibility among different user interface design artifacts.

The work of this thesis aims at contributing to the field of the engineering of interactive critical systems. We aim at easing the introduction of new input and output devices (such as touch screens, mid-air gesture recognition ...) allowing multi-user and multimodal interactions in next generation of aircraft’s cockpits. Currently, development process in the aeronautical filed is not compatible with the complexity of multimodal interaction. On the other side development process of wide spread systems cannot meet the requirements of critical systems. We introduce a generic software and hardware architecture model called MIODMIT (Multiple Input Output devices Multiple Interaction Techniques) which aim the introduction of dynamically instantiated devices, allowing multimodal interaction in critical systems. It describes the organization of information flux with a complete and non-ambiguous way. It covers the entire spectrum of multimodal interactive systems, from input devices and their drivers, to the specification of interaction techniques and the core of the application. It also covers the rendering of every software components, dealing with fission and fusion of information. Furthermore, this architecture model ensure the system configuration modifiability (i.e. add or suppress a device in design or operation phase). Furthermore, moralizing a system reveals that an important part of the interactive part is autonomous (i.e. not driven by the user). This kind of behavior is very difficult to understand and to foresee for the users, causing errors called automation surprises. We introduce a model-based process of evaluation of the interaction techniques which decrease significantly this kind of error. Lastly, we exploited ICO (Interactive Cooperative Objects) formalism , to describe completely and unambiguously each of the software components of MIODMIT. This language is available in an IDE (integrated development environment) called Petshop, which can execute globally the interactive application (from input/output devices to the application core). We completed this IDE with an execution platform named ARISSIM (ARINC 653 Standard SIMulator), adding safety mechanisms. More precisely, ARRISIM allows spatial segregation of processes (memory allocution to each executing partition to ensure the confinement of potential errors) and temporal segregation (sequential use of processor). Those adding increase significantly the system reliability during execution. Our work is a base for multidisciplinary teams (more specifically ergonoms, HMI specialist and developers) which will conceive future human machine interaction in the next generation of aircraft cockpits.

The current European Air Traffic Management (ATM) System needs to be improved for coping with the growth in air traffic forecasted for next years. It has been broadly recognised that the future ATM capacity and safety objectives can only be achieved by an intense enhancement of integrated automation support. However, increase of automation might come along with an increase of performance variability of the whole ATM System especially in case of automation degradation. ATM systems are considered complex as they encompass interactions involving humans and machines deeply influenced by environmental aspects (i.e. weather, organizational structure) making them belong to the class of Socio-Technical Systems (STS) (Emery & Trist, 1960). Due to this complexity, the interactions between the STS elements (human, system and organisational) can be partly linear and partly non-linear making its performance evolution complex and hardly predictable. Within such STS, interactive systems have to be usable i.e. enabling users to perform their tasks efficiently and effectively while ensuring a certain level of operator satisfaction. Besides, the STS has to be resilient to adverse events including potential automation degradation issues but also interaction problems between their interactive systems and the operators. These issues may affect several STS aspects such as resources, time in tasks performance, ability to adjust to environment, etc. In order to be able to analyse the impact of these perturbations and to assess the potential performance variability of a STS, dedicated techniques and methods are required. These techniques and methods have to provide support for modelling and analysing in a systematic way usability and resilience of interactive systems featuring partly autonomous behaviours. They also have to provide support for describing and structuring a large amount of information and to be able to address the variability of each of STS elements as well as the variability related to their interrelations. Current techniques, methods and processes do not enable to model a STS as a whole and to analyse both usability and resilience properties. Also, they do not embed all the elements that are required to describe and analyse each part of the STS (such as knowledge of different types which is needed by a user for accomplishing tasks or for interacting with dedicated technologies). Lastly, they do not provide means for analysing task migrations when a new technology is introduced or for analysing performance variability in case of degradation of the newly introduced automation. Such statements are argued in this thesis by a detailed analysis of existing modelling techniques and associated methods highlighting their advantages and limitations. This thesis proposes a multi-models based approach for the modelling and the analysis of partly-autonomous interactive systems for assessing their resilience and usability. The contribution is based on the identification of a set of requirements needed being able to model and analyse each of the STS elements. Some of these requirements were met by existing modelling techniques, others were reachable by extending and refining existing ones. This thesis proposes an approach which integrates 3 modelling techniques: FRAM (focused on organisational functions), HAMSTERS (centred on human goals and activities) and ICO (dedicated to the modelling of interactive systems). The principles of the multi-models approach is illustrated on an example for carefully showing the extensions proposed to the selected modelling techniques and how they integrate together. A more complex case study from the ATM domain is then presented to demonstrate the scalability of the approach. This case study, dealing with aircraft route change due to bad weather conditions, highlights the ability of the integration of models to cope with performance variability of the various parts of the STS.