Prototyping Cyber-Physical Systems using Wayfaring - An Experiment and Insights for Early-Stage Development
MetadataShow full item record
The two main aims of the thesis are firstly to qualitatively and quantitatively research the effect of using a prototype-driven wayfaring approach for the early-stage development of cyber-physical systems to give normative recommendations to developers and secondly to show and discuss how this kind of research with human subjects can be conducted and improved. Many development methodologies rely on early definition of requirements and design specifications and lack the agility to make fast design adaptations. However, this agility is necessary to cope with unforeseen design problems that arise during the early stages of highly uncertain complex development projects. Prototype-driven approaches, that rely on frequent testing and evaluation of prototypes for advancing the design, offer the necessary agility but research is required to better understand what influences the outcome of such a development process and when it is appropriate to use it. I answer these questions in the context of developing cyber-physical systems by studying two case studies and a controlled quantitative experiment in the context of programming a mobile autonomous robot. The first case study of a small but multidisciplinary project is used to extend the wayfaring approach and shows how prototyping helps to discover and solve unforeseen problems, especially while merging different disciplines. The second case study explains how the wayfaring approach can be applied in highly uncertain large-scale development projects as shown here with the example of the ATLAS detector development at CERN. The controlled experiment quantitatively explores whether early testing of prototypes has an influence on the performance of the design outcome and also explores qualitatively what influences this performance. The research shows that a wayfaring-like approach can be suitable during the explorative concept evaluation phase of solving highly uncertain engineering problems. Testing prototypes and abductively reflecting on the test results helps to discover unforeseen design errors and in finding possible solutions. However, the experiment could not show a statistically significant performance difference between participants who tested their designs early and often compared to participants who could not test their design early. It showed large individual performance differences between participants regardless of their experimental condition. Qualitatively, these individual differences can be explained by variations in error finding ability which not only depends on the participant’s mental ability and programming experience but also on when, how and what the participants test and how the code is written. Furthermore, the results suggest that urged early testing without sufficient questioning of design choices can lead to design fixation supposedly enhanced by the sunk cost of building and testing prototypes. The insights lead to the normative suggestions to use early low-resolution prototyping to keep the sunk costs to a minimum, prototype only the critical functions of several disciplines quasi-simultaneously, prototype different concepts in parallel and prioritize the critical functions whose solutions appear to be least likely to succeed. Well controlled quantitative human subject experiments are necessary to better understand, quantify and compare design methodologies. Such quantitative experiments in design methodology research are rare and often not controlled enough to unambiguously attribute the observed results to the stimulus. This thesis uses the above mentioned robot experiment as a case to show measures like automating the experiment procedures or avoiding direct human-human interaction between the experimenter and the participant to increase repeatability and internal validity of the experiment. The thesis also discusses lessons learned and when using the wayfaring approach is appropriate for developing such experiments.
Has partsPaper 1: Kriesi, Carlo; Steinert, Martin; Gerstenberg, Achim; et al. Distributed Experiments in Design Sciences, A Next Step in Design Observation Studies?. I: DS 80-2 Proceedings of the 20th International Conference on Engineering Design (ICED 15) Vol 2: Design Theory and Research Methodology Design Processes, Milan, Italy, 27-30.07.15. The Design Society 2015 ISBN 978-1-904670-65-0. s. 319-328 Not included due to copyright restrictions.
Paper 2: Reime, Thov; Sjöman, Heikki; Gerstenberg, Achim; Abrahamsson, Pekka Kalevi; Steinert, Martin. Bridging Tangible and Virtual Interaction: Rapid Prototyping of a Gaming Idea. I: Entertainment Computing - ICEC 2015 14th International Conference, ICEC 2015, Trondheim, Norway, September 29 - Ocotober 2, 2015, Proceedings. Springer 2015 ISBN 978-3-319-24588-1. s. 523-528
Paper 3: Gerstenberg, Achim; Sjöman, Heikki; Reime, Thov; Abrahamsson, Pekka Kalevi; Steinert, Martin. A Simultaneous, Multidisciplinary Development and Design Journey – Reflections on Prototyping. I: Entertainment Computing - ICEC 2015 14th International Conference, ICEC 2015, Trondheim, Norway, September 29 - Ocotober 2, 2015, Proceedings. Springer 2015 ISBN 978-3-319-24588-1. s. 409-416
Paper 4: Gerstenberg, Achim; Steinert, Martin. Large-scale engineering prototyping-Approaching complex engineering problems cern-style. Proceedings of the International Conference on Engineering Design 2017 ;Volum 2.(87-2) s. 397-406. Not included due to copyright restrictions.
Paper 5: Gerstenberg, Achim; Steinert, Martin. Open ended problems – A robot programming experiment to compare and test different development and design approaches. I: DS 91: Proceedings of NordDesign 2018, Linköping, Sweden, 14th - 17th August 2018 DESIGN IN THE ERA OF DIGITALIZATION. The Design Society 2018 ISBN 978-91-7685-185-2. s. – Not included due to copyright restrictions.
Paper 6: Gerstenberg, Achim; Steinert, Martin. Testing the effect of desirable difficulties on teaching robotics. CERN IdeaSquare Journal of Experimental Innovation, 2020; X(X): X-X
Paper 7: Gerstenberg, Achim; Steinert, Ralf Martin. Development and verification of a simulation for leveraging results of a human subjects programming experiment.
Paper 8: Gerstenberg, Achim; Steinert, Martin. Evaluating and optimizing chaotically behaving mobile robots with a deterministic simulation. Procedia CIRP 2019 ;Volum 84. s. 219-224
Paper 9: Gerstenberg, Achim; Steinert, Martin. The Relevance of Testing in Engineering Product Development - Investigations on a Robot Programming Task. Procedia CIRP 2019 ;Volum 84. s. 586-592
Paper 10: Gerstenberg, Achim; Sjöman, Heikki; Steinert, Martin. Fixation on premature concept choices - a pitfall of early prototyping?. Procedia CIRP 2019 ;Volum 84. s. 33-37
Paper 11: Dybvik, Henrikke; Løland, Martin Poverud; Gerstenberg, Achim; Slåttsveen, Kristoffer Bjørnerud; Steinert, Ralf Martin. A low-cost predictive display for teleoperation: Investigating effects on human performance and workload. International Journal of Human-Computer Studies 2020 ;Volum 145. s. -