dc.contributor.advisor | Giannakos, Michail | |
dc.contributor.advisor | Divitini, Monica | |
dc.contributor.advisor | Sharma, Kshitij | |
dc.contributor.author | Lee-Cultura, Serena Glyn | |
dc.date.accessioned | 2023-08-16T07:02:28Z | |
dc.date.available | 2023-08-16T07:02:28Z | |
dc.date.issued | 2023 | |
dc.identifier.isbn | 978-82-326-7161-8 | |
dc.identifier.issn | 2703-8084 | |
dc.identifier.uri | https://hdl.handle.net/11250/3084285 | |
dc.description.abstract | Including children in the learning process through purposeful motion is a complex, yet promising endeavour. Motion-Based Technologies (MBTs), which depend on sensors to naturally engage children through “touchless” gesture, offer new opportunities to leverage motion, while enriching children’s learning experiences on their pursuit of knowledge acquisition, cognitive skill development, and advancement of executive functions. Due to their capacity to harmonise fun and learning, MBTs have started gaining traction as drivers of children’s interactive learning. Parallel to this, advancements in technology have led to increased accessibility of ubiquitous lightweight sensors (e.g., eye-trackers, wristbands, motion sensors, web cameras), capable of extracting children’s cognitive, physiological, affective, and skeletal data during their motion-based interactions. Moreover, the analysis of MultiModal Data (MMD) from different data sources (i.e., sensors) may hold strong explanatory power with respect to children’s learning experiences centred on MBTs. However, despite the potential of leveraging MMD to advance our understanding of children’s technology-mediated Movement-Based Learning (MBL) experiences, limited research efforts have explored this generative space.
This doctoral research describes a methodological exploration which investigates how MMD can augment our understanding, and facilitate our use, of MBTs in children’s learning. This dissertation presents a corpus of research conducted across four and a half years. The research examines the question of children’s MBT-enhanced learning experiences from various angles in order to construct a comprehensive understanding that can be leveraged by children and their support sphere, teachers and parents, to promote and enrich technology-mediated learning. The context of this doctoral work is framed in the confluence of Child-Computer Interaction, MBL, and MultiModal Learning Analytics (MMLA).
Pursuant to our research goal, this doctoral work adhered to a Design-Based Research (DBR) approach in order to understand the advantages and challenges of integrating MMD into children’s MBT enhanced learning experiences. In particular, we employed Motion-Based Educational Games (MBEGs). This included exploring the potential of MMD to understand the influence of various game settings (e.g., avatar-self representation, interaction modes); investigating synergies between children’s technology-mediated play and problem-solving behaviours, and their cognitive, physiological and behavioural states; discerning how children’s physio-cognitive states progress through different stages of their problem-solving experience; predicting children’s performance outcomes; and lastly, exploring the advantages and challenges of visualising MMD to augment pedagogy. Three DBR cycles, each encompassing a field study, were designed and conducted. In line with different objectives, each DBR cycle included different participant demographics, data collection methods, variables, and analysis approaches. In cycles 1 and 2, children in the concrete operational stage of development played a series of MBEGs. Cycle 1 included children with Special Educational Needs (SEN) and data was collected through the MBEG’s system logs. Cycle 2 involved typically developing children and in addition to system logs and screen capture, we collected their MMD using eye-trackers, wristbands, motion sensors, and web camera. Analyses were both quantitative and mixed-methods. In cycle 3, a teacher dashboard to visualise MMD was designed and developed. Educational experts (i.e., teachers, educational researchers) were interviewed regarding their perceived challenges and advantages of using such a tool in practice. Analysis was strictly qualitative.
The contributions offered by this doctoral research include a systematic literature review of empirical studies, with findings reporting on Embodied Interaction (EI) and spatial skills, in particular, motion sensing and its impact on children’s learning; a new observational scheme for categorizing children’s play and problem-solving behaviours in the context of MBTs for learning, a synthesis of MMD measures, and conceptual models to facilitate researchers’ data analysis and sense-making; a series of feedback provision related design guidelines; and a context-flexible MMD-powered dashboard proof of concept with data-driven notifications to visualises learners’ data in real or replay-time. | en_US |
dc.language.iso | eng | en_US |
dc.publisher | NTNU | en_US |
dc.relation.ispartofseries | Doctoral theses at NTNU;2023:231 | |
dc.relation.haspart | Paper 1: Lee-Cultura, Serena; Giannakos, Michail. Embodied Interaction and Spatial Skills: A Systematic Review of Empirical Studies. Interacting with computers 2020 ;Volum 32.(4) s. 331-366 https://doi.org/10.1093/iwcomp/iwaa023 This article is available under the Creative Commons CC-BY-NC license | en_US |
dc.relation.haspart | Paper 2: Lee-Cultura, Serena; Sharma, Kshitij; Aloizou, Valeria; Retalis, Symeon; Giannakos, Michail. Children's Interaction with Motion-Based Touchless Games: Kinecting Effectiveness and Efficiency. I: Proceedings of the Annual Symposium on Computer-Human Interaction in Play. Association for Computing Machinery (ACM) 2020 ISBN 978-1-4503-8074-4. s. 140-145 https://doi.org/10.1145/3383668.3419937 | en_US |
dc.relation.haspart | Paper 3: Lee-Cultura, Serena; Sharma, Kshitij; Papavlasopoulou, Sofia; Retalis, Symeon; Giannakos, Michail. Using Sensing Technologies to Explain Children’s Self-Representation in Motion-Based Educational Games. I: IDC '20: Proceedings of the Interaction Design and Children Conference. Association for Computing Machinery (ACM) 2020 ISBN 978-1-4503-7981-6. s. 541-555 https://doi.org/10.1145/3392063.3394419 | en_US |
dc.relation.haspart | Paper 4: Lee-Cultura, Serena; Sharma, Kshitij; Papavlasopoulou, Sofia; Giannakos, Michail. Motion-Based Educational Games: Using Multi-Modal Data to Predict Player’s Performance. IEEE Conference on Computatonal Intelligence and Games, CIG 2020 s. 17-24 https://doi.org/10.1109/CoG47356.2020.9231892 | en_US |
dc.relation.haspart | Paper 5: Sharma, Kshitij; Lee-Cultura, Serena; Giannakos, Michail. Keep Calm and Do Not Carry-Forward: Toward Sensor-Data Driven AI Agent to Enhance Human Learning. Frontiers in Artificial Intelligence 2022 ;Volum 4. s. 1-27 https://doi.org/10.3389/frai.2021.713176 This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY) | en_US |
dc.relation.haspart | Paper 6: Lee-Cultura, Serena; Sharma, Kshitij; Cosentino, Giulia; Papavlasopoulou, Sofia; Giannakos, Michail. Children’s Play and Problem Solving in Motion-Based Educational Games: Synergies between Human Annotations and Multi-Modal Data. I: IDC '21: Interaction Design and Children. Association for Computing Machinery (ACM) 2021 ISBN 978-1-4503-8452-0. s. 408-420 https://doi.org/10.1145/3459990.3460702 and https://hdl.handle.net/11250/2990109 | en_US |
dc.relation.haspart | Paper 7: Lee-Cultura, Serena; Sharma, Kshitij; Giannakos, Michail. Children's play and problem-solving in motion-based learning technologies using a multi-modal mixed methods approach. International Journal of Child-Computer Interaction 2022 ;Volum 31. https://doi.org/10.1016/j.ijcci.2021.100355 This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY) | en_US |
dc.relation.haspart | Paper 8: MultiModal Teacher Dashboards: Challenges and Opportunities of Enhancing Teacher Insights through a case study IEEE Transactions on Learning Technologies https://doi.org/10.1109/TLT.2023.3276848 This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY) | en_US |
dc.title | Understanding and augmenting children's learning experiences with motion-based technologies: The role of multimodal analytics | en_US |
dc.type | Doctoral thesis | en_US |
dc.subject.nsi | VDP::Technology: 500::Information and communication technology: 550::Computer technology: 551 | en_US |