Game-based learning design and evaluation: Towards better understanding and improvement

Abstract

In recent years, Game-based learning (GBL) has proliferated rapidly and is widely used in various fields to improve learning. Educational games can actively engage learners and meet individual learning situations, making them effective educational tools for various learning contexts. The rapid increase in educational games makes it necessary to understand and improve the GBL design and evaluation process and analyze the various aspects that influence the GBL experience. The GBL research presents a wide range of diverse criteria for designing and evaluating educational games used inconsistently across literature, hindering a holistic view. It is important to conceptualize the GBL elements for comprehensive design, analysis, and evaluation. Educational game development is a complex process and introduces challenges related to balancing different GBL aspects that require collaboration among the heterogeneous team of experts. However, GBL research lacks operationalizable approaches for integrating GBL design knowledge into the educational game design process to facilitate collaboration and shared understanding among team members. Educational game researchers are interested in investigating the effectiveness of the GBL approach in different educational settings. Especially the recent Syrian war and COVID-19 crisis have emphasized the potential of educational technology such as learning games to support informal and non-formal learning for the marginalized population of refugee children and online education during the pandemic. Despite the growing research in GBL, there is still a considerable need to further and systematically investigate the effectiveness of learning games to provide empirical evidence on their appropriate design. Identifying the evaluation criteria is complex and time-consuming, making GBL evaluation a critical undertaking. Therefore, appropriate techniques, methods, and principles are required to conduct the evaluation research. The research undertaken in this thesis investigates how the GBL design and evaluation process can be facilitated to improve the effectiveness of learning games. The research work considers the need for multi-dimensional focus with various interlinked aspects that need to be balanced to facilitate the learning process in GBL. Consequently, the focus of this doctoral work is to build a more comprehensive understanding of GBL by investigating factors that influence the learning experience with games in different domains and learning contexts. Moreover, the research work focuses on facilitating the GBL design team in collaboratively ideating effective design and facilitating stakeholders (such as designers, educators, researchers) to analyze learning games and plan the GBL evaluation studies. The research work also explored the potential of GBL in different learning contexts focusing on evaluating educational games in formal and online education (amid COVID-19) with students and informal learning setup (focusing refugee crisis) with refugee children. Moreover, since the current Human-Computer Interaction (HCI) and GBL research do not consider the peculiar, situated nature of GBL research with refugee children, this research work also focused on understanding methods and guidelines for conducting GBL research with this user group. To this end, this doctoral research followed the design science research (DSR) methodology to develop a GBL framework and three tools/instruments for the design and evaluation of learning games, answering the identified problems. A mixed methods research design (exploratorytriangulation design) was used in connection with DSR in three cycles focusing on the GBL phenomenon, GBL design process and GBL evaluation process. This research comprises three review studies, the design of artifacts, three case studies, and six empirical studies (GBL evaluation studies and GBL design workshops). The research work focused on two primary activities of DSR, building and evaluating. First, the GBL framework was constructed by combining the existing knowledge then GBL evaluation studies were conducted to understand and validate the framework constructs. The GBL framework is put into practice by utilizing the framework components to design and implement three instruments/tools to design, analyze, and evaluate learning games. The evaluation results from design workshops and GBL evaluation studies (quasi-experiments) along with case studies were used to revise the tools/instruments, validate the framework concepts, and provide further implications/guidelines for improvement. The resulting contributions provide substantive findings and identify challenges and issues regarding current GBL design and evaluation practices. A holistic conceptual GBL framework has been devised to bootstrap the design and evaluation process. The framework provides the key GBL elements in a detailed hierarchy focusing on a multi-dimensional approach. The research also investigates the GBL learning process and the contributing factors by employing educational games in different learning contexts. Moreover, this doctoral work includes developing and implementing a card-based toolkit for ideating educational game design, facilitating collaboration and completeness in the GBL ideation process. It also provides the knowledge for transforming a theoretical framework into a design toolkit. Furthermore, the GBL evaluation process is facilitated by devising an analysis instrument and an integrated evaluation approach to guide stakeholders. Finally, guidelines are presented for evaluating and designing effective learning games in general and particularly for refugee children. The particular emphasis on this special group is based on empirical evidence from user studies and lessons learned from the author's practical experience from the EduAPP4Syria project. The GBL community of educators, researchers, and practitioners can learn from these guidelines for more inclusive design focusing on user characteristics.