Understanding and augmenting children's learning experiences with motion-based technologies: The role of multimodal analytics

Abstract

Including children in the learning process through purposeful motion is a complex, yet promising endeavour. Motion-Based Technologies (MBTs), which depend on sensors to naturally engage children through “touchless” gesture, offer new opportunities to leverage motion, while enriching children’s learning experiences on their pursuit of knowledge acquisition, cognitive skill development, and advancement of executive functions. Due to their capacity to harmonise fun and learning, MBTs have started gaining traction as drivers of children’s interactive learning. Parallel to this, advancements in technology have led to increased accessibility of ubiquitous lightweight sensors (e.g., eye-trackers, wristbands, motion sensors, web cameras), capable of extracting children’s cognitive, physiological, affective, and skeletal data during their motion-based interactions. Moreover, the analysis of MultiModal Data (MMD) from different data sources (i.e., sensors) may hold strong explanatory power with respect to children’s learning experiences centred on MBTs. However, despite the potential of leveraging MMD to advance our understanding of children’s technology-mediated Movement-Based Learning (MBL) experiences, limited research efforts have explored this generative space. This doctoral research describes a methodological exploration which investigates how MMD can augment our understanding, and facilitate our use, of MBTs in children’s learning. This dissertation presents a corpus of research conducted across four and a half years. The research examines the question of children’s MBT-enhanced learning experiences from various angles in order to construct a comprehensive understanding that can be leveraged by children and their support sphere, teachers and parents, to promote and enrich technology-mediated learning. The context of this doctoral work is framed in the confluence of Child-Computer Interaction, MBL, and MultiModal Learning Analytics (MMLA). Pursuant to our research goal, this doctoral work adhered to a Design-Based Research (DBR) approach in order to understand the advantages and challenges of integrating MMD into children’s MBT enhanced learning experiences. In particular, we employed Motion-Based Educational Games (MBEGs). This included exploring the potential of MMD to understand the influence of various game settings (e.g., avatar-self representation, interaction modes); investigating synergies between children’s technology-mediated play and problem-solving behaviours, and their cognitive, physiological and behavioural states; discerning how children’s physio-cognitive states progress through different stages of their problem-solving experience; predicting children’s performance outcomes; and lastly, exploring the advantages and challenges of visualising MMD to augment pedagogy. Three DBR cycles, each encompassing a field study, were designed and conducted. In line with different objectives, each DBR cycle included different participant demographics, data collection methods, variables, and analysis approaches. In cycles 1 and 2, children in the concrete operational stage of development played a series of MBEGs. Cycle 1 included children with Special Educational Needs (SEN) and data was collected through the MBEG’s system logs. Cycle 2 involved typically developing children and in addition to system logs and screen capture, we collected their MMD using eye-trackers, wristbands, motion sensors, and web camera. Analyses were both quantitative and mixed-methods. In cycle 3, a teacher dashboard to visualise MMD was designed and developed. Educational experts (i.e., teachers, educational researchers) were interviewed regarding their perceived challenges and advantages of using such a tool in practice. Analysis was strictly qualitative. The contributions offered by this doctoral research include a systematic literature review of empirical studies, with findings reporting on Embodied Interaction (EI) and spatial skills, in particular, motion sensing and its impact on children’s learning; a new observational scheme for categorizing children’s play and problem-solving behaviours in the context of MBTs for learning, a synthesis of MMD measures, and conceptual models to facilitate researchers’ data analysis and sense-making; a series of feedback provision related design guidelines; and a context-flexible MMD-powered dashboard proof of concept with data-driven notifications to visualises learners’ data in real or replay-time.