Prototyping Cyber-Physical Systems using Wayfaring - An Experiment and Insights for Early-Stage Development

Abstract

The two main aims of the thesis are firstly to qualitatively and quantitatively research the effect of using a prototype-driven wayfaring approach for the early-stage development of cyber-physical systems to give normative recommendations to developers and secondly to show and discuss how this kind of research with human subjects can be conducted and improved. Many development methodologies rely on early definition of requirements and design specifications and lack the agility to make fast design adaptations. However, this agility is necessary to cope with unforeseen design problems that arise during the early stages of highly uncertain complex development projects. Prototype-driven approaches, that rely on frequent testing and evaluation of prototypes for advancing the design, offer the necessary agility but research is required to better understand what influences the outcome of such a development process and when it is appropriate to use it. I answer these questions in the context of developing cyber-physical systems by studying two case studies and a controlled quantitative experiment in the context of programming a mobile autonomous robot. The first case study of a small but multidisciplinary project is used to extend the wayfaring approach and shows how prototyping helps to discover and solve unforeseen problems, especially while merging different disciplines. The second case study explains how the wayfaring approach can be applied in highly uncertain large-scale development projects as shown here with the example of the ATLAS detector development at CERN. The controlled experiment quantitatively explores whether early testing of prototypes has an influence on the performance of the design outcome and also explores qualitatively what influences this performance. The research shows that a wayfaring-like approach can be suitable during the explorative concept evaluation phase of solving highly uncertain engineering problems. Testing prototypes and abductively reflecting on the test results helps to discover unforeseen design errors and in finding possible solutions. However, the experiment could not show a statistically significant performance difference between participants who tested their designs early and often compared to participants who could not test their design early. It showed large individual performance differences between participants regardless of their experimental condition. Qualitatively, these individual differences can be explained by variations in error finding ability which not only depends on the participant’s mental ability and programming experience but also on when, how and what the participants test and how the code is written. Furthermore, the results suggest that urged early testing without sufficient questioning of design choices can lead to design fixation supposedly enhanced by the sunk cost of building and testing prototypes. The insights lead to the normative suggestions to use early low-resolution prototyping to keep the sunk costs to a minimum, prototype only the critical functions of several disciplines quasi-simultaneously, prototype different concepts in parallel and prioritize the critical functions whose solutions appear to be least likely to succeed. Well controlled quantitative human subject experiments are necessary to better understand, quantify and compare design methodologies. Such quantitative experiments in design methodology research are rare and often not controlled enough to unambiguously attribute the observed results to the stimulus. This thesis uses the above mentioned robot experiment as a case to show measures like automating the experiment procedures or avoiding direct human-human interaction between the experimenter and the participant to increase repeatability and internal validity of the experiment. The thesis also discusses lessons learned and when using the wayfaring approach is appropriate for developing such experiments.