Publications

My 124 research publications include 49 journal articles (23 JCR Q1s, 19 JCR Q2s, 3 JCR Q3s, 1 JCR Q4), 4 editorials, 39 conference papers, and 32 book chapters.

Filter by topic:

Journal Articles

2024

• López-Pernas S., Saqr M. (2024). How the dynamics of engagement explain the momentum of achievement and the inertia of disengagement: A complex systems theory approach. Computers in Human Behavior, vol. 153, art. no. 108126. doi: 10.1016/j.chb.2023.108126. (JCR Q1).

  
  Engagement can be understood as a complex dynamic process that unfolds over time and interacts with variables within the student, school, and environment. Most of the research on the dynamics of engagement comes from classroom settings and it is so far inconclusive how and why engagement and disengagement evolve over time. Using person-centered methods, sequence, transition, and covariate analysis, we examined a large dataset of 18 consecutive courses of 245 students over a full program. We identified three engagement states (active, average, and disengaged), as well as three distinct longitudinal engagement trajectories (engaged, fluctuating, and mostly disengaged). Taken together, our results showed that engagement trajectories are rather stable over time conforming to the universal dynamics of complex systems. Engaged students were driven by course materials, their achievement, and their previous engaged states (momentum). Most importantly, our results offer a novel theoretical grounding for the understanding of disengagement which has so far remained unexplained. According to our results, disengagement follows the dynamics of a complex system where stability does not require a hard-wired causal mechanism but rather, it is an attractor state that pulls the system to settle in (inertia). Thus, disengagement becomes an equilibrium state for those students that is hard to change (or a stuck state).

  learning analyticsengagementmarkov modelingsequence analysislongitudinal studycomplex dynamic systems

  
  
• López-Pernas S., Saqr M., Conde M.Á., Apiola M., Tedre M. (2024). Mapping computer engineering education research: A topic analysis. Computer (in-press). doi: 10.1109/MC.2024.3349913. (JCR Q3).

  
  The field of computer engineering has evolved into a separate entity from electrical engineering and computer science. Emerging technologies such as IoT and cloud computing have made their way into computer engineering programs and, as a result, computer engineering education has become increasingly relevant. To ensure that computer engineering students receive a comprehensive education, research is necessary to identify the key areas of focus and determine the current state of the field. In this article, we applied structural topic modeling to identify the themes of computer engineering education research using bibliometric metadata. We analyze the trends in research and the relationships between themes. Our findings reveal that research mainly focuses on subjects that are not unique to computer engineering education (e.g., mathematics and programming). Furthermore, pedagogy and teaching practices play an increasingly central role in connecting research themes. Lastly, there is increasing attention to learning analytics and psychological research.

  structural topic modelingcomputer engineeringengineering educationbibliometrics

  
  
• Mertala P., López-Pernas S., Vartiainen H., Saqr M., Tedre M. (2024). Digital natives in the scientific literature: A topic modeling approach. Computers in Human Behavior, vol. 152, art. no. 108076. doi: 10.1016/j.chb.2023.108076. (JCR Q1).

  
  The term “digital natives” was introduced in 2001 to describe a generation that has grown up surrounded by technology and the internet. The accompanying claims of a new way of thinking among digital natives were influential in shaping educational policy. Still, they were challenged by research that found no evidence of generation-wide cognitive changes in learners. Yet, the digital natives narrative persists in popular media and the education discourse. This study set out to investigate the reasons for the persistence of the digital native myth. It analyzed the metadata from 1886 articles related to the term between 2001 and 2022 using bibliometric methods and structural topic modeling. The results show that the concept of “digital native” is still both warmly embraced and fiercely criticized by scholars mostly from western and high income countries, and the volume of research on the topic is growing. However, the results suggest that what appears as the persistence of the idea is actually evolution and complete reinvention: The way the “digital native” concept is operationalized has shifted over time through a series of (metaphorical) mutations. The concept of digital native is one (albeit a highly successful) mutation of the generational gap discourse dating back to the early 1900s. While the initial digital native literature relied on Prensky’s unvalidated claims and waned upon facing empirical challenges, subsequent versions have sought more nuanced interpretations. Notably, a burgeoning third mutation now co-opts the “digital native” terminology for diverse purposes, often completely decoupled from the foundational literature and its critiques. This study explains the concept’s persistence as dynamic evolution of the digital native discourse in contemporary academic and public spheres.

  digital nativesbibliometricsstructured topic modelingdigital immigrants

  
  
• Bobrowicz K., López-Pernas S., Teuber Z., Saqr M., Greiff S. (2024). Prospects in the field of learning and individual differences: Examining the past to forecast the future using bibliometrics. Learning and Individual Differences, vol. 109, art. no. 102399. doi: 10.1016/j.lindif.2023.102399. (JCR Q1).

  
  Over two centuries, research has delved into individual differences in learning across educational and professional contexts. This commentary conducts a bibliometric analysis of 6556 articles, identifying key research keywords, topics and themes, and their historical evolution. The findings revealed a longstanding emphasis on educational psychology, particularly motivation and achievement, rather than cross-curricular competencies and learner’s well-being and socio-economic background. Notably, self-regulated learning (SRL) emerged as an overarching research subject in terms of motivation and achievement, but, surprisingly, not (meta)cognition. Prospects for the field build on cross-disciplinary research, theoretical refinement, and methodological advances. Further, the field is expected to maintain academic rigor, address diversity among learners, foster global collaboration, and focus on underprivileged populations.

  bibliometric analysislearning and individual differenceseducation trends

  
  
• Saqr M., López-Pernas S. (2024). Mapping the self in self-regulation using complex dynamic systems and idiographic methods. British Journal of Educational Technology (in-press). (JCR Q1).

  
  Complex dynamic systems offer a rich platform for understanding the individual or the person-specific mechanisms. Yet, in learning analytics research and education at large, a complex dynamic system has rarely been framed, developed, or used to understand the individual student where the learning process takes place. Individual (or person-specific) methods can accurately and precisely model the individual person, create person-specific models, and devise unique parameters for each individual. Our study used the latest advances in complex systems dynamics to study the differences between group-based and individual self-regulated learning (SRL) dynamics. The findings show that SRL is a complex, dynamic system where different sub-processes influence each other resulting in the emergence of non-trivial patterns that vary across individuals and time scales, and as such far from the uniform picture commonly theorized. We found that the average SRL process does not reflect the individual SRL processes of different people. Therefore, interventions derived from the group-based SRL insights are unlikely to be effective in personalization. We posit that, if personalized interventions are needed, modeling the person with person-specific methods should be the guiding principle. Our study offered a reliable solution to model the person-specific self-regulation processes which can serve as a ground for understanding and improving individual learning and open the door for precision education.

  
  
• Barra E., López-Pernas S., Gordillo A., Pozo Huertas A., Munoz-Arcentales A., Conde J. (2024). Empowering Database Learning through Remote Educational Escape Rooms. IEEE Internet Computing (in-press). doi: 10.1109/MIC.2023.3333199. (JCR Q2).

  
  Learning databases is indispensable for individuals studying software engineering, computer science, or involved in the IT industry. We have analyzed a remote educational escape room for teaching databases in four different higher education courses in two consecutive academic years. We employed three instruments for evaluation: a pre- and post-test to assess the escape room’s learning effectiveness, a questionnaire to gather students’ perceptions, and a web platform that unobtrusively records students’ interactions and performance. We show novel evidence that educational escape rooms conducted remotely can be engaging as well as effective for teaching databases.

  
  
• Santamaría-Urbieta A., López-Pernas S. (2024). Hint Strategies in Educational Escape Rooms: A Process Mining Approach. Revista de Educación (in-press). (JCR Q4).

  
  Educational escape rooms have become a useful tool for teachers who want to engage their students and attract their attention to the content taught. Also, they have been found to be valuable in improving learner outcomes, perceptions, and engagement in higher education. Although much attention has been placed on students’ opinions when playing educational escape rooms, not much attention has been placed on the importance of the design process, and little research has investigated the effectiveness of hint strategies in optimizing participant experiences and learning outcomes. In this study, and through a process mining approach, the hints strategies of four online educational escape rooms at the university level are determined. The games were designed with the software Escapp, which allows researchers to collect students’ trace log data during the escape rooms. With this data, we calculated descriptive statistics for each escape room, studied the relationship between hints and performance, and to take into account the temporal aspect of students’ actions, we employed process mining to investigate the transitions between actions and the role of hints in helping students solve the puzzles. Results show that the use of hints was generally low and that participants relied more on their own problem-solving skills. However, there were instances in which hints were requested and correlated with longer gameplay duration and a higher number of failed attempts. In conclusion, the present study addresses a gap in the existing literature which highlights, after our analysis, the need for careful consideration of hint design and delivery strategies.

  educational escape roomshintsgame-based learninglearning analyticsprocess mining

  
  
• López-Fernández D., Gordillo A., López-Pernas S., Tovar E. (2024). Are remote educational escape rooms designed during the pandemic useful in a post-pandemic face-to-face setting?. IEEE Internet Computing (in-press). doi: 10.1109/MIC.2023.3336057. (JCR Q2).

  
  Numerous initiatives were conducted online during the COVID-19 pandemic and today it is necessary to analyze whether it is better to continue conducting these initiatives online or should they be done face-to-face and even readapted to this format. This paper compares an educational escape room for learning software engineering conducted online during the confinement caused by the pandemic and later face-to-face. The research involves 241 students, and employs instruments to explore the knowledge acquisition attained by the students and their perceptions towards the activity. The results provide insights to consider in the future use of this technique. The digital elements used in a remote escape room are suitable for a face-to-face escape room since the educational efficacy of the activity was similar when conducted online and face-to-face. However, some students’ perceptions related to enjoyment were worse in the face-to-face escape room, which could be improved by incorporating physical elements.

  
  
• Barra E., Quemada J., López-Pernas S., Gordillo A., Alonso Á., Carril A. (2024). An Autonomous Low-Cost Studio to Record Production-Ready Instructional Videos. Multimedia Tools and Applications (in-press). doi: 10.1007/s11042-024-18250-8. (JCR Q2).

  
  Producing high-quality educational videos usually requires a large budget as it involves the use of expensive recording studios, the presence of a technician during the entire recording session and often post-production tasks. The high costs associated with video production represent a major hindrance for many educational institutions and, thus, many teachers regard high-quality video recording as inaccessible. As a remedy to this situation, this article presents SAGA (Autonomous Advanced Recording Studio in its Spanish acronym), a low-cost autonomous recording set that allows teachers to produce educational content in video format in an agile way and without the need for post-production. The article provides an overview of SAGA, including a description of its hardware and software so that anyone with basic technical knowledge can replicate and operate the system. SAGA has been used to record more than 1,500 videos including the contents of six MOOCs hosted on the MiriadaX platform, as well as four courses at UPM. SAGA has been evaluated in two ways: (1) from the video producers’ perspective, it was evaluated with a questionnaire based on the Technology Acceptance Model, and (2) from the video consumers’ perspective, a questionnaire was conducted among MOOC participants to assess the perceived technical quality of the videos recorded with SAGA. The results show a very positive general opinion of the SAGA system, the recorded videos and the technical features thereof. Thus, SAGA represents a good opportunity for all those educational institutions and teachers interested in producing high-quality educational videos at a low cost.

  recording studioinstructional videosvideo equipmentmultimedia materialstechnology acceptance model.

  
  
• Vartiainen H., Kahila J., Tedre M., López-Pernas S., Pope N. (2024). Enhancing Children’s Understanding of Algorithmic Biases in and With Text-to-Image Generative AI. New Media and Society (in-press). (JCR Q1).

2023

• López-Pernas S. (2023). Educational Escape Rooms Are Effective Learning Activities Across Educational Levels and Contexts: A Meta-analysis. IEEE Transactions on Learning Technologies, vol. 17, pp. 711-724. doi: 10.1109/TLT.2023.3328913. (JCR Q1).

  
  Educational escape rooms are taxing in terms of the time needed to design, create, conduct, and evaluate. Therefore, a high “return on investment” is expected regarding their potential to improve teaching and learning. Whereas many studies have been conducted to assess the impact of educational escape rooms on learning, results have been so far inconclusive. Several studies have reported positive learning gains, whereas others have not demonstrated such learning gains. To provide a synthesis of the existing empirical body of knowledge, we performed a metaanalysis by pooling the effect sizes across 33 published studies (5,322 observations). Our results suggest that educational escape rooms are highly effective learning activities (Cohen’s d = 1.4). The impact on learning was consistent across diverse fields and educational levels, regardless of team size and technology involved. Yet, educational escape rooms conducted remotely yielded smaller learning gains than those conducted face-to-face. Studies comparing educational escape rooms to other educational activities —while scarce— did not suggest a significant superiority of these activities to traditional learning activities, e.g., lectures.

  educational escape roomsgame-based learninggamificationlearning analyticsmeta-analysis

  
  
• López-Pernas S., Barra E., Gordillo A., Alonso Á., Quemada J. (2023). Scaling student feedback in software engineering MOOCs. IEEE Software, vol. 40(5), pp. 50-57. doi: 10.1109/MS.2023.3275035. (JCR Q2).

  
  Training software professionals is essential to meet the high demands of today’s job market. MOOCs constitute a promising way to scale up such training, as they enable the participation of a massive number of learners. However, a major challenge that software engineering MOOCs face is the provision of formative assessment through continuous feedback to learners. This article presents a bot aimed at helping MOOC participants complete programming assignments by providing them with timely formative feedback. The bot has been used and evaluated in a software engineering specialization comprised of four MOOCs. Students’ perceptions towards the bot were very positive, and the usage statistics indicate that many students rely on it to complete their assignments. Overall, the results suggest that the bot constitutes a suitable solution for providing participants of software engineering MOOCs with formative feedback related to programming assignments, contributing to overcoming a well-known limitation of this type of training.

  automated assessmentsoftware engineeringmoocslearning analyticsgrowth mixture models

  
  
• Saqr M., López-Pernas S., Helske S., Hrastinski S. (2023). The longitudinal association between engagement and achievement varies by time, students’ profiles, and achievement state: A full program study. Computers and Education, vol. 199, art. no. 104787. doi: 10.1016/j.compedu.2023.104787. (JCR Q1).

  
  There is a paucity of longitudinal studies in online learning across courses or throughout programs. Our study intends to add to this emerging body of research by analyzing the longitudinal trajectories of interaction between student engagement and achievement over a full four-year program. We use learning analytics and life-course methods to study how achievement and engagement are intertwined and how such relationship evolves over a full program for 106 students. Our findings have indicated that the association between engagement and achievement varies between students and progresses differently between such groups over time. Our results showed that online engagement at any single time-point is not a consistent indicator for high achievement. It takes more than a single point of time to reliably forecast high achievement throughout the program. Longitudinal high grades, or longitudinal high levels of engagement (either separately or combined) were indicators of a stable academic trajectory in which students remained engaged —at least on average— and had a higher level of achievement. On the other hand, disengagement at any time point was consistently associated with lower achievement among low-engaged students. Improving to a higher level of engagement was associated with —at least— acceptable achievement levels and rare dropouts. Lack of improvement or “catching up” may be a more ominous sign that should be proactively addressed.

  learning analyticsmulti-channel sequence analysislongitudinal engagementacademic achievementmixture hidden markov modelsstudent profilesperson-centered methods

  
  
• Saqr M., López-Pernas S., Jovanovic J., Gasevic. G (2023). Intense, turbulent, or wallowing in the mire: A longitudinal study of cross-course online tactics, strategies, and trajectories. The Internet and Higher Education, vol. 57, art. no. 100902. doi: 10.1016/j.iheduc.2022.100902. (JCR Q1).

  
  Research has repeatedly demonstrated that students with effective learning strategies are more likely to have better academic achievement. Existing research has mostly focused on a single course or two, while longitudinal studies remain scarce. The present study examines the longitudinal sequence of students’ strategies, their succession, consistency, temporal unfolding, and whether students tend to retain or adapt strategies between courses. We use a large dataset of online traces from 135 students who completed 10 successive courses (i.e., 1350 course enrollments) in a higher education program. The methods used in this study have shown the feasibility of using trace data recorded by learning management systems to unobtrusively trace and model the longitudinal learning strategies across a program. We identified three program-level strategy trajectories: a stable and intense trajectory related to deep learning where students used diverse strategies and scored the highest grades; a fluctuating interactive trajectory, where students focused on course requirements, scored average grades, and were relatively fluctuating; and a light trajectory related to surface learning where students invested the least effort, scored the lowest grades, and had a relatively stable pathway. Students who were intensely active were more likely to transfer the intense strategies and therefore, they were expected to require less support or guidance. Students focusing on course requirements were not as effective self-regulators as they seemed and possibly required early guidance and support from teachers. Students with consistent light strategies or low effort needed proactive guidance and support.

  learning analyticslearning strategiessequence analysislongitudinal studies

  
  
• Saqr M., López-Pernas S. (2023). The temporal dynamics of online problem-based learning: Why and when sequence matters. International Journal of Computer-Supported Collaborative Learning, vol. 18, pp. 11-37. doi: 10.1007/s11412-023-09385-1. (JCR Q1).

  
  Early research about online PBL explored students’ satisfaction, effectiveness, and design. The temporal aspect of online PBL have rarely been addressed. Thus, a gap exists in how online PBL unfolds: when, and for how long a group engages in collaborative discussions. Similarly, little is known about if and what order or sequence of interactions could predict higher achievement. This study aims to bridge such a gap, by implementing the latest advances in temporal learning analytics to analyze the sequential and temporal phases of online PBL across a large sample (n=204 students) of qualitatively coded interactions (8,009 interactions). We analyzed the group level —to understand the group dynamics across the whole problem discussions— and at the students’ level —to understand the students’ contribution dynamics across different episodes. We followed such analysis by examining the association of interaction types and the sequences thereof with students’ performance using multilevel linear regression models. The analysis of the interactions reflected that the scripted PBL process is followed in sequence, yet often lacked enough depth. When cognitive interactions (e.g., arguments, questions, and evaluations) happened, they kindled high cognitive interactions, when low cognitive and social interactions dominated, they kindled low cognitive interaction. Order and sequence of interactions were more predictive of performance with higher explanatory power than frequencies. Starting or initiating interactions (even with low cognitive content) showed the highest association with performance, which points to the importance of timing and sequence.

  learning analyticssequence miningprocess miningcsclproblem-based learning

  
  
• Saqr M., López-Pernas S., Vogelsmeier L.V.D.E. (2023). When, how and for whom changes in engagement happen: A transition analysis of instructional variables. Computers and Education, vol. 207, art. no. 104934. doi: 10.1016/j.compedu.2023.104934. (JCR Q1).

  
  The pace of our knowledge on online engagement has not been at par with our need to understand the temporal dynamics of online engagement, the transitions between engagement states, and the factors that influence a student being persistently engaged, transitioning to disengagement, or catching up and transitioning to an engaged state. Our study addresses such a gap and investigates how engagement evolves or changes over time, using a person-centered approach to identify for whom the changes happen and when. We take advantage of a novel and innovative multistate Markov model to identify what variables influence such transitions and with what magnitude, i.e., to answer the why. We use a large data set of 1428 enrollments in six courses (238 students). The findings show that online engagement changes differently —across students— and at different magnitudes —according to different instructional variables and previous engagement states. Cognitively engaging instructions helped cognitively engaged students stay engaged while negatively affecting disengaged students. Lectures —a resource that requires less mental energy— helped improve disengaged students. Such differential effects point to the different ways interventions can be applied to different groups, and how different groups may be supported. A balanced, carefully tailored approach is needed to design, intervene, or support students’ engagement that takes into account the diversity of engagement states as well as the varied response magnitudes that intervention may incur across diverse students’ profiles.

  learning analyticstransition analysisonline engagementlongitudinal engagementlatent markov modeling

  
  
• Kleimola R., López-Pernas S., Väisänen S., Saqr M., Sointu E., Hirsto L. (2023). Learning Analytics to Explore the Motivational Profiles of Non-Traditional Practical Nurse Students: A Mixed-Methods Approach. Empirical Research in Vocational Education and Training, vol. 15, art. no. 11. doi: 10.1186/s40461-023-00150-0.

  
  Learning analytics provides a novel means to examine various aspects of students’ learning and to support them in their individual endeavors. The purpose of this study was to explore the potential of learning analytics to provide insights into non-traditional, vocational practical nurse students’ (N = 132) motivational profiles for choosing their studies, using a mixed-methods approach. Non-traditional students were somewhat older learners than those following a more straightforward educational pathway and had diverse educational or professional backgrounds. Institutional admission data and analytics were used to identify their specific study motives and distinct motivational profiles, and to illustrate the connections between the motives emerging in the motivational profiles. Furthermore, the association between the motivational profiles and study performance was examined. The results of qualitative content analysis indicated that non-traditional practical nurse students pursued such specialized training for various reasons, and that pragmatic, professional rationales were emphasized over prosocial, altruistic factors. Through the adoption of person-centered latent class analysis, three motivational profiles were identified: self-aware goal-achievers, qualification attainers, and widely oriented humanitarians. Additionally, the analyses of epistemic networks for the profiles showed the complex interplay between the motives, confirming that some motive connections appear to be more prominent than others. Moreover, the findings indicated that study motives reported at admission did not seem to dictate students’ later study performance, as no statistically significant associations were found between the motivational profile and the students’ final grade point average or study dropout. This investigation paves the way for more-targeted motivational support and the use of learning analytics in the context of vocational education and training.

  learning analyticsstudy motivesmotivational profilespractical nurse studentsnon-traditional studentsvocational education and trainingmixed-methods approachlatent class analysisepistemic network analysis

  
  
• Kahila J., Valtonen T., López-Pernas S., Saqr M., Vartiainen H., Kahila S., Tedre M. (2023). A typology of metagamers: Identifying player types based on beyond the game activities. Games & Culture (in-press). doi: 10.1177/15554120231187758. (JCR Q1).

  
  Previous research on player types is based on players’ in-game behaviors and their motivations to play games. However, there are many other activities related to digital games beyond playing the games proper. This study investigates the prevalence and interconnections between these different metagame activities, and classifies gamers based on their use thereof. The results show that digital game-related information-seeking activities are key metagame activities with connections to other metagame activities. Three distinct groups of players were identified based on their metagame activities: versatile metagamers, strategizers, and casual metagamers. The results contribute to the existing literature on metagaming and provide insights into game studies, game design and marketing, and into digital games and learning.

  digital gamesmetagamingmetagameplayer typesmodel-based clusteringepistemic network analysis

  
  
• Saqr M., Matcha W., Ahmad Uzir N-A., Jovanovic J., Gasevic. G, López-Pernas S. (2023). Transferring effective learning strategies across learning contexts matters: A study in problem-based learning. Australasian Journal of Educational Technology, vol. 39(3), pp. 35-57. doi: 10.14742/ajet.8303. (JCR Q1).

  
  Learning strategies are important catalysts of students’ learning. Research has shown that students with effective learning strategies are more likely to have better academic achievement and complete the program. Given the importance of learning strategies in problem-based learning —and in education in general— for student’s learning and achievement, this study aimed to investigate students’ adoption of learning strategies in different course implementations as well as the transfer of learning strategies between courses and relations to performance. We took advantage of recent advances in learning analytics methods, namely sequence and process mining, as well as statistical methods and visualizations to study how students regulate their online learning through using learning strategies. The study included 81,739 log traces of students’ learning related activities from two different problem-based learning medical courses. The results revealed a relation between the adopted learning strategies, course implementation and scaffolding. Students who applied deep learning strategies were more likely to score high grades, and students who applied surface learning strategies were more likely to score lower grades in either course. More importantly, students who were able to transfer deep learning strategies or continue to use effective strategies between courses proved to be the ones with the highest scores, and the least likely to adopt surface strategies in the subsequent course. These results highlight the need for supporting the development of effective learning strategies in PBL curricula so that students adopt and transfer effective strategies as they advance through the program.

  learning strategiesproblem-based learninglearning analyticssequence miningprocess mining

  
  

2022

• López-Pernas S., Saqr M., Gordillo A., Barra E. (2022). A learning analytics perspective on educational escape rooms. Interactive Learning Environments. doi: 10.1080/10494820.2022.2041045. (JCR Q1).

  
  Learning analytics methods have proven useful in providing insights from the increasingly available digital data about students in a variety of learning environments, including serious games. However, such methods have not been applied to the specific context of educational escape rooms and therefore little is known about students’ behavior while playing. The present work aims to fill the gap in the existing literature by showcasing the power of learning analytics methods to reveal and represent students’ behavior when participating in a computer-supported educational escape room. Specifically, we make use of sequence mining methods to analyze the temporal and sequential aspects of the activities carried out by students during these novel educational games. We further use clustering to identify different player profiles according to the sequential unfolding of students’ actions and analyze how these profiles relate to knowledge acquisition. Our results show that students’ behavior differed significantly in their use of hints in the escape room and resulted in differences in their knowledge acquisition levels. © 2022 The Author(s). Published by Informa UK Limited, trading as Taylor & Francis Group.

  computer science educationeducational escape roomsgame-based learninglearning analyticssequence miningserious games

  
  
• López-Pernas S., Munoz-Arcentales A., Aparicio C., Barra E., Gordillo A., Salvachua J., Quemada J. (2022). Educational Data Virtual Lab: Connecting the Dots Between Data Visualization and Analysis. IEEE Computer Graphics and Applications, vol. 42(5), pp. 76-83. doi: 10.1109/MCG.2022.3189557. (JCR Q3).

  
  Educational Data Virtual Lab (EDVL) is an open-source platform for data exploration and analysis that combines the power of a coding environment, the convenience of an interactive visualization engine, and the infrastructure needed to handle the complete data lifecycle. Based on the building blocks of the FIWARE European platform and Apache Zeppelin, this tool allows domain experts to become acquainted with data science methods using the data available within their own organization, ensuring that the skills they acquire are relevant to their field and driven by their own professional goals. We used EDVL in a pilot study in which we carried out a focus group within a multinational company to gain insight into potential users’ perceptions of EDVL, both from the educational and operational points of view. The results of our evaluation suggest that EDVL holds a great potential to train the workforce in data science skills and to enable collaboration among professionals with different levels of expertise. © 1981-2012 IEEE.

  
  
• Saqr M., López-Pernas S. (2022). How CSCL roles emerge, persist, transition, and evolve over time: A four-year longitudinal study. Computers and Education, vol. 189, art. no. 104581. doi: 10.1016/j.compedu.2022.104581. (JCR Q1).

  
  A prevailing trend in CSCL literature has been the study of students’ participatory roles. The majority of existing studies examine a single collaborative task or, at most, a complete course. This study aims to investigate the presence —or the lack thereof— of a more enduring disposition that drives student participation patterns across courses. Based on data from a 4-year program where 329 students used CSCL to collaborate in 10 successive courses (amounting up to 84,597 interactions), we identify the emerging roles using centrality measures and latent profile analysis (LPA) and trace the unfolding of roles over the entire duration of the program. Thereafter, we use Mixture Hidden Markov Models (MHMM) —methods that are particularly useful in detecting “latent traits” in longitudinal data— to identify how students’ roles, transition, persist or evolve over time. Relevant covariates were also examined to explain students’ membership of different trajectories. We identified three different roles (leader, mediator, isolate) at the course level. At the program level, we found three distinct trajectories: an intense trajectory with mostly leaders, a fluctuating trajectory with mostly mediators, and a wallowing-in-the-mire trajectory with mostly isolates. Our results show that roles re-emerge consistently regardless of the task or the course over extended periods of time and in a predictable manner. For instance, isolates “assumed” such a role in almost all of their courses over four years. © 2022 The Authors

  computer-supported collaborative learninglearning analyticslongitudinal analysisperson-centeredroles

  
  
• Valtonen T., López-Pernas S., Saqr M., Vartiainen H., Sointu E.T., Tedre M. (2022). The nature and building blocks of educational technology research. Computers in Human Behavior, vol. 128, art. no. 107123. doi: 10.1016/j.chb.2021.107123. (JCR Q1).

  
  Supporting teaching and learning with different technologies has a long and broad history. The theories of learning have changed in recent decades, and new technologies have been invented that provide possibilities for supporting learning processes based on different learning theories. Recently, this field has been studied using bibliometric methods with large datasets to gain an overview of this research area. This paper continues this approach by targeting and analyzing the most relevant journals in the field, covering 30,632 articles. The aim was to deepen the results gained from previous bibliometric studies by focusing on the journals within the educational technology field, the keywords used, the most-cited papers, and especially by outlining the theoretical backgrounds of the analyzed articles. Results show new journals with increasing numbers of articles published. By analyzing the publications used as the articles’ backgrounds, we can identify the large entities within the field. Articles targeted how technology can support learning processes based on different learning theories. Along with building an understanding of technology-related learning processes, the second large area of research targets the integration and factors affecting the use of technology for teaching and learning practices. The third large perspective focuses on learners’ characteristics, especially their learning skills, motivation, and self-efficacy. The results show that new technologies by themselves do not cause fast changes in research goals and topics; rather, significant changes in the research field evolve slowly. © 2021

  
  
• Apiola M., López-Pernas S., Saqr M., Pears A., Daniels M., Malmi L., Tedre M. (2022). From a National Meeting to an International Conference: A Scientometric Case Study of a Finnish Computing Education Conference. IEEE Access, vol. 10, pp. 66576-66588. doi: 10.1109/ACCESS.2022.3184718. (JCR Q2).

  
  Computerisation and digitalisation are shaping the world in fundamental and unpredictable ways, which highlights the importance of computing education research (CER). As part of understanding the roots of CER, it is crucial to investigate the evolution of CER as a research discipline. In this paper we present a case study of a Finnish CER conference called Koli Calling, which was launched in 2001, and which has become a central publication venue of CER. We use data from 2001 to 2020, and investigate the evolution of Koli Calling’s scholarly communities and zoom in on it’s publication habits and internalisation process. We explore the narrative of the development and scholarly agenda behind changes in the conference submission categories from the perspective of some of the conference chairs over the years. We then take a qualitative perspective, analysing the conference publications based on a comprehensive bibliometric analysis. The outcomes include classification of important research clusters of authors in the community of conference contributors. Interestingly, we find traces of important events in the historical development of CER. In particular, we find clusters emerging from specific research capacity building initiatives and we can trace how these connect research spanning the world CER community from Finland to Sweden and then further to the USA, Australia and New Zealand. This paper makes a strategic contribution to the evolution of CER as a research discipline, from the perspective of one central event and publication venue, providing a broad perspective on the role of the conference in connecting research clusters and establishing an international research community. This work contributes insights to researchers in one specific CER community and how they shape the future of computing education © 2013 IEEE.

  computer science educationcomputing educationcomputing education researchreviewscience mappingscientometrics

  
  
• Saqr M., López-Pernas S. (2022). The Curious Case of Centrality Measures: A Large-Scale Empirical Investigation. Journal of Learning Analytics, vol. 9(1), pp. 13-31. doi: 10.18608/jla.2022.7415.

  
  There has been extensive research using centrality measures in educational settings. One of the most common lines of such research has tested network centrality measures as indicators of success. The increasing interest in centrality measures has been kindled by the proliferation of learning analytics. Previous works have been dominated by single-course case studies that have yielded inconclusive results regarding the consistency and suitability of centrality measures as indicators of academic achievement. Therefore, large-scale studies are needed to overcome the multiple limitations of existing research (limited datasets, selective and reporting bias, as well as limited statistical power). This study aims to empirically test and verify the role of centrality measures as indicators of success in collaborative learning. For this purpose, we attempted to reproduce the most commonly used centrality measures in the literature in all the courses of an institution over five years of education. The study included a large dataset (n=3,277) consisting of 69 course offerings, with similar pedagogical underpinnings, using meta-analysis as a method to pool the results of different courses. Our results show that degree and eigenvector centrality measures can be a consistent indicator of performance in collaborative settings. Betweenness and closeness centralities yielded uncertain predictive intervals and were less likely to replicate. Our results have shown moderate levels of heterogeneity, indicating some diversity of the results comparable to single laboratory replication studies. © 2022, Society for Learning Analytics Research. All rights reserved.

  centrality measureslearning analyticsmeta-analysisreplicabilityreproducibilitysocial network analysis

  
  
• Tyni J., Tarkiainen A., López-Pernas S., Saqr M., Kahila J., Bednarik R., Tedre M. (2022). Games and Rewards: A Scientometric Study of Rewards in Educational and Serious Games. IEEE Access, vol. 10, pp. 31578-31585. doi: 10.1109/ACCESS.2022.3160230. (JCR Q2).

  
  In this study we provide a new viewpoint on the body of literature regarding rewards in serious and educational games. The study includes a quantitative bibliometric analysis of literature in this context from 1969 to 2020. The dataset from the Scopus abstract and citation database was analyzed with the Bibliometrix R library. The data set was manually cleaned to contain only the relevant articles and conference papers. The data was then categorized to match the common themes. From the remaining documents, the amount of annual numbers of publications is presented and the most contributing countries are shown. The most frequent terms from the abstracts and keywords set by the authors are presented, and a co-occurrence network is drawn from the same data. The results of this study reveal that the most occurring topics in this dataset are gamification, physical activity, health, game design, and game-based learning. New directions for research are provided as the most commonly used media appear to be video games and mobile devices in addition to the literature being mostly focused on theory and not practical application. © 2022 IEEE.

  bibliometricseducational gamesrewardsscientometric analysisserious games

  
  
• Saqr M., Poquet O., López-Pernas S. (2022). Networks in Education: A Travelogue Through Five Decades. IEEE Access, vol. 10, pp. 32361-32380. doi: 10.1109/ACCESS.2022.3159674. (JCR Q2).

  
  For over five decades, researchers have used network analysis to understand educational contexts, spanning diverse disciplines and thematic areas. The wealth of traditions and insights accumulated through these interdisciplinary efforts is a challenge to synthesize with a traditional systematic review. To overcome this difficulty in reviewing 1791 articles researching the intersection of networks and education, this study combined a scientometric approach with a more qualitative analysis of metadata, such as keywords and authors. Our analysis shows rapidly growing research that employs network analysis in educational contexts. This research output is produced by researchers in a small number of developed countries. The field has grown more recently, through the surge in the popularity of data-driven methods, the adoption of social media, and themes as teacher professional development and the now-declining MOOC research. Our analysis suggests that research combining networks and educational phenomena continues to lack an academic home, as well as remains dominated by descriptive network methods that depict phenomena such as interpersonal friendship or patterns of discourse-based collaboration. We discuss the gaps in existing research, the methodological shortcomings, the possible future directions and most importantly how network research could help advance our knowledge of learning, learners, and contribute to our knowledge and to learning theories. © 2013 IEEE.

  bibliometricseducationlearning analyticsnetwork sciencesocial network analysis

  
  
• Apiola M., Saqr M., López-Pernas S., Tedre M. (2022). Computing Education Research Compiled: Keyword Trends, Building Blocks, Creators, and Dissemination. IEEE Access, vol. 10, pp. 27041-27068. doi: 10.1109/ACCESS.2022.3157609. (JCR Q2).

  
  The need for organized computing education efforts dates back to the 1950s. Since then, computing education research (CER) has evolved and matured from its early initiatives and separation from mathematics education into a respectable research specialization of its own. In recent years, a number of meta-research papers, reviews, and scientometric studies have built overviews of CER from various perspectives. This paper continues that approach by offering new perspectives on the past and present state of CER: Analyses of influential papers throughout the years, of the theoretical backgrounds of CER, of the institutions and authors who create CER, and finally of the top publication venues and their citation practices. The results reveal influential contributions from early curriculum guidelines to rigorous empirical research of today, the prominence of computer programming as a topic of research, evolving patterns of learning-Theory usage, the dominance of high-income countries and a cluster of 52 elite institutions, and issues regarding citation practices within the central venues of dissemination. © 2013 IEEE.

  computer science educationcomputing educationcomputing education researchreviewscience mappingscientometrics

  
  
• Saqr M., Elmoazen R., Tedre M., López-Pernas S., Hirsto L. (2022). How well centrality measures capture student achievement in computer-supported collaborative learning? – A systematic review and meta-analysis. Educational Research Review, vol. 35, art. no. 100437. doi: 10.1016/j.edurev.2022.100437. (JCR Q1).

  
  Research has shown the value of social collaboration and the benefits it brings to learners. In this study, we investigate the worth of Social Network Analysis (SNA) in translating students’ interactions in computer-supported collaborative learning (CSCL) into proxy indicators of achievement. Previous research has tested the correlation between SNA centrality measures and achievement. Some results indicate a positive association, while others do not. To synthesize research efforts, investigate which measures are of value, and how strong of an association exists, this article presents a systematic review and meta-analysis of 19 studies that included 33 cohorts and 16 centrality measures. Achievement was operationalized in most of the reviewed studies as final course or task grade. All studies reported that one or more centrality measures had a positive and significant correlation with, or a potential for predicting, achievement. Every centrality measure in the reviewed sample has shown a positive correlation with achievement in at least one study. In all the reviewed studies, degree centralities correlated with achievement in terms of final course grades or other achievement measure with the highest magnitude. Eigenvector-based centralities (Eigenvector, PageRank, hub, and authority values) were also positively correlated and mostly statistically significant in all the reviewed studies. These findings emphasize the robustness and reliability of degree- and eigenvector-based centrality measures in understanding students’ interactions in relation to achievement. In contrast, betweenness and closeness centralities have shown mixed or weak correlations with achievement. Taken together, our findings support the use of centrality measures as valid proxy indicators of academic achievement and their utility for monitoring interactions in collaborative learning settings. © 2022 The Authors

  achievementcentrality measurescscleducational data mininglearning analyticspredicting performancesocial network analysis

  
  
• Conde J., Munoz-Arcentales A., Alonso A., López-Pernas S., Salvachua J. (2022). Modeling Digital Twin Data and Architecture: A Building Guide With FIWARE as Enabling Technology. IEEE Internet Computing, vol. 26(3), pp. 7-14. doi: 10.1109/MIC.2021.3056923. (JCR Q2).

  
  The use of digitial twins (DTs) in industry has become a growing trend in recent years, allowing improvement of the life cycle of any process by taking advantage of the relationship between the physical and virtual worlds. Existing literature posits several challenges for building DTs, as well as some proposals for overcoming them. However, in the vast majority of the cases, the architectures and technologies presented are strongly bounded to the domain where the DTs are applied. This article proposes the FIWARE Ecosystem, combining its catalog of components and smart data models as a solution for the development of any DT. We also provide a use case to show how to use FIWARE for building DTs through a complete example of a parking DT. We conclude that the FIWARE Ecosystem constitutes a real reference option for developing DTs in any domain. © 1997-2012 IEEE.

  
  

2021

• López-Pernas S., Gordillo A., Barra E. (2021). Technology-Enhanced Educational Escape Rooms: A Road Map. IT Professional, vol. 23(2), art. no. 9391739, pp. 26-32. doi: 10.1109/MITP.2021.3062749. (JCR Q2).

  
  In recent years, escape rooms have become one of the leading leisure activities worldwide. Derived from these activities, educational escape rooms have emerged as a new type of teaching practice with the promise of enhancing students’ learning through highly engaging experiences. The rapid rise of educational escape rooms has led to a misalignment between educators’ needs for being able to implement this novel teaching practice and the availability of tools to ease the process. Moreover, this lack of support is preventing teachers and students from taking full advantage of the potential of educational escape rooms. This article provides a road map of the most urgent issues to be addressed to bridge the aforementioned gap: easing the creation of digital puzzles, aiding in the logistical aspects of conducting an educational escape room, harnessing learning analytics, fostering remote collaboration, and integrating artificial intelligence to adapt the experience to each team. © 1999-2012 IEEE.

  
  
• López-Pernas S., Saqr M. (2021). Bringing Synchrony and Clarity to Complex Multi-Channel Data: A Learning Analytics Study in Programming Education. IEEE Access, vol. 9, pp. 166531-166541. doi: 10.1109/ACCESS.2021.3134844. (JCR Q2).

  
  Supporting teaching and learning programming with learning analytics is an active area of inquiry. Most data used for learning analytics research comes from learning management systems. However, such systems were not developed to support learning programming. Therefore, educators have to resort to other systems that support the programming process, which can pose a challenge when it comes to understanding students’ learning since it takes place in different contexts. Methods that support the combination of different data sources are needed. Such methods would ideally account for the time-ordered sequence of students’ learning actions. In this article, we use a novel method (multi-channel sequence mining with Hidden Markov Models, HMMs) that allows the combination of multiple data sources, accounts for the temporal nature of students’ learning actions, and maps the transitions between different learning tactics. Our study included 291 students enrolled in a higher education programming course. Students’ trace-log data were collected from the learning management system and from a programming automated assessment tool. Data were analyzed using multi-channel sequence mining and HMM. The results reveal different patterns of students’ approaches to learning programming. High achievers start earlier to work on the programming assignments, use more independent strategies and consume learning resources more frequently, while the low achievers procrastinate early in the course and rely on help forums. Our findings demonstrate the potentials of multi-channel sequence mining and how this method can be analyzed using HMM. Furthermore, the results obtained can be of use for educators to understand students’ strategies when learning programming. © 2013 IEEE.

  automated assessmentcomputer science educationhidden markov modelslearning analyticsprogrammingsequence mining

  
  
• López-Pernas S., Gordillo A., Barra E., Quemada J. (2021). Comparing Face-to-Face and Remote Educational Escape Rooms for Learning Programming. IEEE Access, vol. 9, art. no. 9405675, pp. 59270-59285. doi: 10.1109/ACCESS.2021.3073601. (JCR Q2).

  
  Existing literature has provided strong evidence that educational escape rooms are engaging and effective learning activities when they are properly conducted in the classroom. However, no prior research has determined whether the positive effects of these novel educational activities on students’ perceptions and learning persist when conducted remotely. This article performs, for the first time, a comparative study of the effectiveness of face-to-face and remote educational escape rooms. For this purpose, two versions of the same educational escape room were conducted: one in-class and one remotely. Both experiences were evaluated by means of three different instruments: (1) a pre-test and a post-test for measuring learning gains, (2) a questionnaire for assessing students’ perceptions, and (3) a web platform for recording student interaction data during the activities. The results obtained suggest that, although remote educational escape rooms for learning programming can be as engaging as their face-to-face counterparts, their learning effectiveness is somewhat lower. © 2013 IEEE.

  computer science educationdistance learningeducational escape roomseducational technologyelectronic learningengineering educationtechnology enhanced learning

  
  
• López-Pernas S., Gordillo A., Barra E., Quemada J. (2021). Escapp: A web platform for conducting educational escape rooms. IEEE Access, vol. 9, pp. 38062-38077. doi: 10.1109/ACCESS.2021.3063711. (JCR Q2).

  
  Educational escape rooms are emerging as a new type of learning activity with the potential to enhance students’ learning through highly engaging experiences. However, conducting educational escape rooms effectively is very complex and there are no software tools available for this purpose. This lack of support is hindering the widespread use and adoption of these activities. This article presents Escapp, a web platform that allows teachers to conduct effective and highly engaging educational escape rooms. The platform has been used for conducting three different educational escape rooms (one face-to-face and two remotely) in three higher education settings. Three case studies were conducted to empirically evaluate the usefulness of Escapp for conducting these activities, which involved more than 400 students. On the one hand, a questionnaire was administered to students to gather their opinions on the Escapp platform, obtaining very positive results in terms of overall usefulness, usability and engagement. On the other hand, data automatically recorded by Escapp during the three educational escape rooms are presented as evidence of the high number of student interactions that take place during activities of this kind and the need of using a software system for conducting them in an effective way. The results of this article show that Escapp is a well-suited solution for conducting effective face-to-face and remote educational escape rooms. © 2020 Institute of Electrical and Electronics Engineers Inc.. All rights reserved.

  breakout gamescollaborative learningdistance learningeducational activitieseducational technologyescape roomsgame-based learninggamificationsoftware tools

  
  
• López-Pernas S., Saqr M., Viberg O. (2021). Putting it all together: Combining learning analytics methods and data sources to understand students’ approaches to learning programming. Sustainability, vol. 13(9), art. no. 4825. doi: 10.3390/su13094825. (JCR Q3).

  
  Learning programming is a complex and challenging task for many students. It involves both understanding theoretical concepts and acquiring practical skills. Hence, analyzing learners’ data from online learning environments alone fails to capture the full breadth of students’ actions if part of their learning process takes place elsewhere. Moreover, existing studies on learning analytics applied to programming education have mainly relied on frequency analysis to classify students according to their approach to programming or to predict academic achievement. However, frequency analysis provides limited insights into the individual time‐related characteristics of the learning process. The current study examines students’ strategies when learning programming, combining data from the learning management system and from an automated assessment tool used to support students while solving the programming assignments. The study included the data of 292 engineering students (228 men and 64 women, aged 20–26) from the two aforementioned sources. To gain an in‐depth understanding of students’ learning process as well as of the types of learners, we used learning analytics methods that account for the temporal order of learning actions. Our results show that students have special preferences for specific learning resources when learning programming, namely, slides that support search, and copy and paste. We also found that videos are relatively less consumed by students, especially while working on programming assignments. Lastly, students resort to course forums to seek help only when they struggle. © 2021 by the authors. Licensee MDPI, Basel, Switzerland.

  automated assessmentcomputer sciencelearning analyticsprocess miningprogrammingsequence mining

  
  
• Saqr M., López-Pernas S. (2021). Modelling diffusion in computer-supported collaborative learning: a large scale learning analytics study. International Journal of Computer-Supported Collaborative Learning, vol. 16(4), pp. 441-483. doi: 10.1007/s11412-021-09356-4. (JCR Q1).

  
  This study empirically investigates diffusion-based centralities as depictions of student role-based behavior in information exchange, uptake and argumentation, and as consistent indicators of student success in computer-supported collaborative learning. The analysis is based on a large dataset of 69 courses (n = 3,277 students) with 97,173 total interactions (of which 8,818 were manually coded). We examined the relationship between students’ diffusion-based centralities and a coded representation of their interactions in order to investigate the extent to which diffusion-based centralities are able to adequately capture information exchange and uptake processes. We performed a meta-analysis to pool the correlation coefficients between centralities and measures of academic achievement across all courses while considering the sample size of each course. Lastly, from a cluster analysis using students’ diffusion-based centralities aimed at discovering student role-taking within interactions, we investigated the validity of the discovered roles using the coded data. There was a statistically significant positive correlation that ranged from moderate to strong between diffusion-based centralities and the frequency of information sharing and argumentation utterances, confirming that diffusion-based centralities capture important aspects of information exchange and uptake. The results of the meta-analysis showed that diffusion-based centralities had the highest and most consistent combined correlation coefficients with academic achievement as well as the highest predictive intervals, thus demonstrating their advantage over traditional centrality measures. Characterizations of student roles based on diffusion centralities were validated using qualitative methods and were found to meaningfully relate to academic performance. Diffusion-based centralities are feasible to calculate, implement and interpret, while offering a viable solution that can be deployed at any scale to monitor students’ productive discussions and academic success. © 2021, The Author(s).

  centrality measurescomputer-supported collaborative learningdiffusionlearning analyticssocial network analysisstudents’ rolesstudy success

  
  
• Saqr M., López-Pernas S. (2021). The longitudinal trajectories of online engagement over a full program. Computers and Education, vol. 175, art. no. 104325. doi: 10.1016/j.compedu.2021.104325. (JCR Q1).

  
  Student engagement has a trajectory (a timeline) that unfolds over time and can be shaped by different factors including learners’ motivation, school conditions, and the nature of learning tasks. Such factors may result in either a stable, declining or fluctuating engagement trajectory. While research on online engagement is abundant, most authors have examined student engagement in a single course or two. Little research has been devoted to studying online longitudinal engagement, i.e., the evolution of student engagement over a full educational program. This learning analytics study examines the engagement states (sequences, successions, stability, and transitions) of 106 students in 1396 course enrollments over a full program. All data of students enrolled in the academic year 2014–2015, and their subsequent data in 2015–2016, 2016–2017, and 2017–2018 (15 courses) were collected. The engagement states were clustered using Hidden Markov Models (HMM) to uncover the hidden engagement trajectories which resulted in a mostly-engaged (33% of students), an intermediate (39.6%), and a troubled (27.4%) trajectory. The mostly-engaged trajectory was stable with infrequent changes, scored the highest, and was less likely to drop out. The troubled trajectory showed early disengagement, frequent dropouts and scored the lowest grades. The results of our study show how to identify early program disengagement (activities within the third decile) and when students may drop out (first year and early second year). © 2021 The Author(s)

  learning analyticslongitudinal engagementsequence miningsurvival analysistrajectories of engagement

  
  
• Munoz-Arcentales A., López-Pernas S., Conde J., Alonso Á., Salvachúa J., Hierro J.J. (2021). Enabling context-aware data analytics in smart environments: An open source reference implementation. Sensors, vol. 21(21), art. no. 7095. doi: 10.3390/s21217095. (JCR Q1).

  
  In recent years, many proposals of context-aware systems applied to IoT-based smart environments have been presented in the literature. Most previous works provide a generic high-level structure of how a context-aware system can be operationalized, but do not offer clues on how to implement it. On the other hand, there are many implementations of context-aware systems applied to specific IoT-based smart environments that are context-specific: it is not clear how they can be ex-tended to other use cases. In this article, we aim to provide an open-source reference implementation for providing context-aware data analytics capabilities to IoT-based smart environments. We rely on the building blocks of the FIWARE ecosystem and the NGSI data standard, providing an agnostic end-to-end solution that considers the complete data lifecycle, covering from data acquisition and modeling, to data reasoning and dissemination. In other words, our reference implementation can be readily operationalized in any IoT-based smart environment regardless of its field of application, providing a context-aware solution that is not context-specific. Furthermore, we provide two example use cases that showcase how our reference implementation can be used in a variety of fields. © 2021 by the authors. Licensee MDPI, Basel, Switzerland.

  big datacontext-aware systemsdata analytics, smart environmentsfiwareiotmachine learning

  
  
• Conde J., López-Pernas S., Pozo A., Munoz-Arcentales A., Huecas G., Alonso Á. (2021). Bridging the gap between academia and industry through students’ contributions to the fiware european open-source initiative: A pilot study. Electronics, vol. 10(13), art. no. 1523. doi: 10.3390/electronics10131523. (JCR Q2).

  
  Although many courses in computer science and software engineering require students to work on practical assignments, these are usually toy projects that do not come close to real professional developments. As such, recent graduates often fail to meet industry expectations when they first enter the workforce. In view of the gap between graduates’ skills and industry expectations, several institutions have resorted to integrating open-source software development as part of their programs. In this pilot study, we report on the results of the contributions of eleven students to the FIWARE open-source project as part of their final year project. Our findings suggest that both teachers and students have a positive perception towards contributing to the FIWARE open-source initiative and that students increased their knowledge of technologies valued by the industry. We also found that this kind of project requires an additional initial effort for the students as well as for the instructor to monitor their progress. Consequently, it is important that the instructors have previous experience in FIWARE, as many of the students need help during the process. © 2021 by the authors. Licensee MDPI, Basel, Switzerland.

  active and experiential learninghigher educationstem education experiencesuniversity/industry experiencesuniversity/industry/government partnership

  
  
• Gordillo A., Barra E., López-Pernas S., Quemada J. (2021). Development of teacher digital competence in the area of e-safety through educational video games. Sustainability, vol. 13(15), art. no. 8485. doi: 10.3390/su13158485. (JCR Q2).

  
  There is a clear need to promote motivating and effective training actions for the development of teachers’ digital competence, especially in the area of e-safety. Although educational video game-based learning has proven effective to improve motivation and learning outcomes, the existing evidence about its effectiveness for the development of teachers’ digital competence is very lim-ited. This study examines the use of educational video games in an online course in MOOC format with the aim of developing teachers’ digital competence in the e-safety area. A total of 179 teachers from nonuniversity schools in the region of Castilla y León (Spain) participated in this study. A pre-test and a post-test were used to measure the knowledge acquired by the participants, and a questionnaire was used to measure their perceptions. The obtained results suggest that game-based learning using educational video games is an effective and viable approach to train teachers in the e-safety area of digital competence. © 2021 by the authors. Licensee MDPI, Basel, Switzerland.

  digital competencee-safetygame-based learningserious gamesteacher professional competenceteacher training

  
  
• Gordillo A., Barra E., Garaizar P., López-Pernas S. (2021). Use of a Simulated Social Network as an Educational Tool to Enhance Teacher Digital Competence. Revista Iberoamericana de Tecnologias del Aprendizaje, vol. 16(1), art. no. 9328278, pp. 107-114. doi: 10.1109/RITA.2021.3052686.

  
  There is a worrying gap between the digital competence that teachers must have to effectively develop their students’ digital competence and the one they actually have, especially in the area related to the safe and responsible use of technology. Further investigation is needed on the use of training activities, methods and tools aimed at enhancing this competence. This article examines, in the context of an online course in MOOC format, the usefulness of Social Lab, a simulated social network, as an educational tool to improve the digital competence of teachers in the area of safe and responsible use of technology. © 2013 IEEE.

  digital competencee-safetyeducational technologysocial engineeringsocial networkteacher education

  
  

2020

• Barra E., López-Pernas S., Alonso A., Sánchez-Rada J.F., Gordillo A., Quemada J. (2020). Automated Assessment in Programming Courses: A Case Study during the COVID-19 Era. Sustainability, vol. 12(18), art. no. 2428. doi: 10.3390/SU12187451. (JCR Q2).

  
  The COVID-19 pandemic imposed in many countries, in the short term, the interruption of face-to-face teaching activities and, in the medium term, the existence of a ‘new normal’, in which teaching methods should be able to switch from face-to-face to remote overnight. However, this flexibility can pose a great difficulty, especially in the assessment of practical courses with a high student-teacher ratio, in which the assessment tools or methods used in face-to-face learning are not ready to be adopted within a fully online environment. This article presents a case study describing the transformation of the assessment method of a programming course in higher education to a fully online format during the COVID-19 pandemic, by means of an automated student-centered assessment tool. To evaluate the new assessment method, we studied students’ interactions with the tool, as well as students’ perceptions, which were measured with two different surveys: one for the programming assignments and one for the final exam. The results show that the students’ perceptions of the assessment tool were highly positive: if using the tool had been optional, the majority of them would have chosen to use it without a doubt, and they would like other courses to involve a tool like the one presented in this article. A discussion about the use of this tool in subsequent years in the same and related courses is also presented, analyzing the sustainability of this new assessment method. © 2020 by the authors.

  assessmentassessment processassessment techniquesassessment toolsautomated assessmentcomputer science educatione-learningonline education

  
  
• Munoz-Arcentales A., López-Pernas S., Pozo A., Alonso A., Salvachúa J., Huecas G. (2020). Data usage and access control in industrial data spaces: Implementation using FIWARE. Sustainability, vol. 12(9), art. no. 3885. doi: 10.3390/su12093885. (JCR Q2).

  
  In recent years, a new business paradigm has emerged which revolves around effectively extracting value from data. In this scope, providing a secure ecosystem for data sharing that ensures data governance and traceability is of paramount importance as it holds the potential to create new applications and services. Protecting data goes beyond restricting who can access what resource (covered by identity and Access Control): it becomes necessary to control how data are treated once accessed, which is known as data Usage Control. Data Usage Control provides a common and trustful security framework to guarantee the compliance with data governance rules and responsible use of organizations’ data by third-party entities, easing and ensuring secure data sharing in ecosystems such as Smart Cities and Industry 4.0. In this article, we present an implementation of a previously published architecture for enabling access and Usage Control in data-sharing ecosystems among multiple organizations using the FIWARE European open source platform. Additionally, we validate this implementation through a real use case in the food industry. We conclude that the proposed model, implemented using FIWARE components, provides a flexible and powerful architecture to manage Usage Control in data-sharing ecosystems. © 2020 by the authors.

  data access controldata economydata governancedata usage controlfiwareindustry 4.0international data spacesiotshared datauconusage policiesxacml

  
  
• Gordillo A., Lopez-Fernandez D., López-Pernas S., Quemada J. (2020). Evaluating an Educational Escape Room Conducted Remotely for Teaching Software Engineering. IEEE Access, vol. 8, art. no. 9292916, pp. 225032-225051. doi: 10.1109/ACCESS.2020.3044380. (JCR Q2).

  
  With the rise of distance learning, new challenges have emerged for educators. Among these challenges, developing effective and motivating group activities for students in the remote classroom is one of the top priorities to be addressed. According to existing literature, educational escape rooms have proven to be engaging and effective learning activities when conducted face-to-face. However, no prior research has analyzed the instructional effectiveness of these activities when they are conducted remotely. Furthermore, none of the educational escape rooms reported in the literature has been designed for teaching software modeling. This article analyzes an educational escape room conducted remotely in a software engineering fundamentals course for teaching software modeling. A total of three evaluation instruments were used: a pre-test and a post-test to measure students’ learning gains, a questionnaire to collect students’ perceptions, and a web platform for automatically gathering data on students’ interactions. The contribution of this article is two-fold. On the one hand, it provides, for the first time, evidence that remote educational escape rooms can be effective learning activities. On the other hand, it provides, also for the first time, proof that educational escape rooms are effective and engaging activities for teaching software modeling. © 2013 IEEE.

  computer science educationdistance learningeducational escape roomseducational technologysoftware engineering education

  
  
• Velásquez W., Alvarez-Alvarado M.S., Munoz-Arcentales A., López-Pernas S., Salvachúa J. (2020). Body mass index in human gait for building risk assessment using graph theory. Sensors, vol. 20(10), art. no. 2899. doi: 10.3390/s20102899. (JCR Q1).

  
  This article presents a comprehensive study of human physiology to determine the impact of body mass index (BMI) on human gait. The approach followed in this study consists of a mathematical model based on the centre of mass of the human body, the inertia of a person in motion and the human gait speed. Moreover, the study includes the representation of a building using graph theory and emulates the presence of a person inside the building when an emergency takes place. The optimal evacuation route is obtained using the breadth-first search (BFS) algorithm, and the evacuation time prediction is calculated using a Gaussian process model. Then, the risk of the building is quantified by using a non-sequential Monte Carlo simulation. The results open up a new horizon for developing a more realistic model for the assessment of civil safety. © 2020 by the authors. Licensee MDPI, Basel, Switzerland.

  body mass indexbreadth-first searchevacuation routeshuman gaitmonte carlo simulation

  
  
• Alonso A., Pozo A., Gordillo A., López-Pernas S., Munoz-Arcentales A., Marco L., Barra E. (2020). Enhancing university services by extending the eIDAS European specification with academic attributes. Sustainability, vol. 12(3), art. no. 770. doi: 10.3390/su12030770. (JCR Q2).

  
  The European electronic IDentification, Authentication and trust Services (eIDAS) regulation makes available a solution to ensure the cross-border mutual recognition of electronic IDentification (eID) mechanisms among Member States. However, the basic set of attributes currently provided by each country only contains citizens’ personal and legal attributes, preventing e-services to take full advantage of citizens’ domain-specific information, such as academic or medical data. In this article, we propose an extension of the eIDAS specification to support academic attributes as part of citizens’ profiles. In addition, we present an architecture to enable the connection of eIDAS nodes to national attribute providers to enrich citizens’ profiles with additional academic attributes. We have deployed the eIDAS extension in the specific case of the Spanish eIDAS infrastructure, and we have connected it to an attribute provider of the Technical University of Madrid (UPM). We have also improved a set of institutional services of that university by enabling the connection to eIDAS and enhancing the features offered to students based on their academic profiles retrieved from the eIDAS extended infrastructure. Finally, we have evaluated the resulting services thanks to real students from two different countries, concluding that the widespread adoption of the proposed solution in the academic services of European universities will greatly improve their quality and usability. © 2020 by the authors.

  academic attributese-serviceseidaselectronic identificationerasmusidentitystudent mobility

  
  

2019

• López-Pernas S., Gordillo A., Barra E., Quemada J. (2019). Analyzing Learning Effectiveness and Students’ Perceptions of an Educational Escape Room in a Programming Course in Higher Education. IEEE Access, vol. 7, art. no. 8936344, pp. 184221-184234. doi: 10.1109/ACCESS.2019.2960312. (JCR Q1).

  
  In recent years, educational escape rooms have started to gain momentum in the academic community. Prior research has reported on the use of educational escape rooms in several fields. However, earlier works have failed to assess the impact of this sort of activities for teaching programming in terms of learning effectiveness. This work fills the existing gap in the literature by examining an educational escape room for teaching programming in a higher education setting by means of three different instruments: (1) a pre-test and a post-test for measuring learning gains, (2) a survey for assessing students’ perceptions, and (3) a web platform for recording student interaction data during the activity. The results of this work provide, for the first time, empirical evidence that educational escape rooms are an effective and engaging way of teaching programming. © 2013 IEEE.

  computer science educationeducational escape roomseducational technologyengineering education

  
  
• López-Pernas S., Gordillo A., Barra E., Quemada J. (2019). Examining the Use of an Educational Escape Room for Teaching Programming in a Higher Education Setting. IEEE Access, vol. 7, art. no. 8658086, pp. 31723-31737. doi: 10.1109/ACCESS.2019.2902976. (JCR Q1).

  
  In addition to being a well-liked form of recreation, escape rooms have drawn the attention of educators due to their ability to foster teamwork, leadership, creative thinking, and communication in a way that is engaging for students. As a consequence, educational escape rooms are emerging as a new type of learning activity under the promise of enhancing students’ learning through highly engaging experiences. These activities consist of escape rooms that incorporate course materials within their puzzles in such a way that students are required to master these materials in order to succeed. Although several studies have reported on the use of escape rooms in a wide range of disciplines, prior research falls short of addressing the use of educational escape rooms for teaching programming, one of the most valuable skills of the twenty-first century that students often have difficulties grasping. This paper reports on the use of an educational escape room in a programming course at a higher education institution and provide, for the first time, insights on the instructional effectiveness of using educational escape rooms for teaching programming. The results of this paper show that appropriate use of educational escape rooms can have significant positive impacts on student engagement and learning in programming courses. These results also suggest that students prefer these activities over traditional computer laboratory sessions. Finally, another novel contribution of this paper is a set of recommendations and proposals for educators in order to help them create effective educational escape rooms for teaching programming. © 2013 IEEE.

  computer science educationeducational escape roomseducational technologyengineering education

  
  
• Gordillo A., López-Pernas S., Barra E. (2019). Effectiveness of MOOCs for teachers in safe ICT use training. Comunicar, vol. 27(61), pp. 98-107. doi: 10.3916/C61-2019-09. (JCR Q1).

  
  Despite the efforts made, there is still an alarming difference between the digital competence that teachers have and the one they should have in order to develop their students’ digital competence. The lack of teacher training in safe and responsible use of ICT is a special cause for concern. Online courses in MOOC format meet all the required conditions to offer a possible solution to the unavoidable and urgent need for initial and in-service teacher training in this area of digital competence. However, there is currently no evidence in the literature on the effectiveness of these courses for this purpose. This study examines the instructional effectiveness of courses in MOOC format for teacher training in the safe and responsible use of ICT by analysing three different official courses. The courses were analysed using three different methods: a questionnaire to measure participants’ perceptions, pre-tests and post-tests to measure the knowledge acquired, and LORI (Learning Object Review Instrument) to measure the quality of digital educational resources created by the participants. The results suggest that online courses in MOOC format are an effective way to train teachers in the safe and responsible use of ICT, and that these courses can enable the development of digital competence in the area of content creation. © ISSN: 1134-3478.

  digital competencedigital contentsdigital literacymooconline coursesonline learningonline protectionteacher education

  
  

Editorials

2023

• Hirsto L., López-Pernas S., Saqr M., Valtonen T., Sointu E., Väisänen S. (2023). Preface. Bridging Education Learning Analytics and AI: Challenges of the Present and Thoughts for the Future. Proceedings of the 1st Finnish Learning Analytics and Artificial Intelligence in Education Conference (FLAIEC 2022) (CEUR Workshop Proceedings), vol. 3383, pp. 1-6. https://ceur-ws.org/Vol-3383/FLAIEC22_preface.pdf.
• Elmoazen R., López-Pernas S., Misiejuk K., Khalil M., Wasson B. and Saqr M. (2023). Reflections on Technology-enhanced Learning in Laboratories: Barriers and Opportunities. Proceedings of the Technology-Enhanced Learning in Laboratories Workshop (TELL 2023) (CEUR Workshop Proceedings), vol. 3393, pp. 1-4. https://ceur-ws.org/Vol-3393/TELL23_preface.pdf.

  
  This preface discusses the potential of virtual labs (VLs) as a flexible and immersive alternative to traditional physical labs on the light of Technology-Enhanced Learning in Laboratories workshop (TELL 2023), which features seven papers showing various methodologies and perspectives on using VLs. The papers cover topics such as VLs in biomedicine, students’ perception of VLs, and collaborative learning in VLs, and examine challenges and barriers to VL accessibility. We discuss some of the main findings of the papers, such as the potential of digital applications and online materials to enhance digital teaching and the importance of developing strategies to enhance team-based learning through encouraging students to reflect on their own work. Overall, the preface demonstrates the potential of VLs to enhance students’ practical skills, and learning outcomes, with insights into the challenges and barriers to VLs accessibility.

  virtual labsdigital labslearning analyticsonline learningcollaborative learning

  
  
• Conde M.Á., López-Pernas S., Saqr M., Raspopovic-Milic M. (2023). Addressing the complexity of online education: A learning analytics and big data perspective. Proceedings of the 13th International Conference on eLearning (eLearning-2022), vol. 3454(in-press), pp. 1-2. https://ceur-ws.org/Vol-3454/preface.pdf.

2021

• Saqr M., López-Pernas S. (2021). Toward self big data. International Journal of Health Sciences, vol. 15(5). https://pubmed.ncbi.nlm.nih.gov/34548857.

Conference Papers

2024

• Conde J., López-Pernas S., Barra E., Saqr M. (2024). The Temporal Dynamics of Procrastination and its Impact on Academic Performance: The Case of a Task-oriented Programming Course. Proceedings of the 39th ACM/SIGAPP Symposium On Applied Computing (in-press).
• Ito H., López-Pernas S., Saqr M. (2024). A Review of Idiographic Research in Education: Too Little, But Not Too Late. Proceedings - IEEE 24st International Conference on Advanced Learning Technologies, ICALT 2024 (in-press).

  
  It stands to reason that if we want to offer “personalized” education, our methods should be designed to capture the person and the intraindividual processes. However, an idiographic approach that investigates within-person processes and provides insights on the person has been so far lagging. We conducted a scoping review to explore how the idiographic approach has been applied in educational research (i.e., what methods, topics, data, and statistical approaches). We found that person-specific analysis has mostly been used to investigate education psychology constructs. In addition, many idiographic studies have employed basic statistical techniques, whereas advanced statistical methods have been applied only recently. Therefore, considering the recent development of educational data science, the potential of idiographic methodology needs to be explored further.

  idiographicperson-specificwithin-personlearning analyticsscoping review

  
  
• Saqr M., López-Pernas S. (2024). Momentary emotions emerge and evolve differently across students and time, yet are surprisingly stable. Proceedings - IEEE 24st International Conference on Advanced Learning Technologies, ICALT 2024 (in-press).

  
  Research on academic emotions has explored different granularities that range from a full program to a single task. Yet, most of the existing research stems from cross-sectional studies. While immensely useful, lacking a temporal depth obfuscates the process of emotions into a flat process. To fill this gap, this study takes a process-oriented approach to study the momentary changes in academic emotions as they unfold in time into phases, changes, and successions of sequences during two lectures. We use intensive longitudinal data in the form of ecological momentary surveys. We rely on mixture models to cluster the data into states, use sequence analysis to map the longitudinal unfolding and mixture hidden Markov models to answer why certain longitudinal patterns emerge. Our findings point to differences among students in their reactions to contextual variables, yet, such reactions are relatively stable within the short time window that we studied.

  learning analyticsacademic emotionsaffectgaussian mixture modelssequence analysismixture hidden markov modelspanavavasstra

  
  
• Kaliisa R., Misiejuk K., López-Pernas S., Khalil M., Saqr M. (2024). Have Learning Analytics Dashboards Lived Up to the Hype? A Systematic Review of Impact on Students’ Achievement, Motivation, Participation and Attitude (Best paper award). Proceedings of the Fourteenth International Conference on Learning Analytics & Knowledge (LAK’24) (in-press). https://arxiv.org/abs/2312.15042.

  
  While learning analytics dashboards (LADs) are the most common form of LA intervention, there is limited evidence regarding their impact on students learning outcomes. This systematic review synthesizes the findings of 38 research studies to investigate the impact of LADs on students’ learning outcomes, encompassing achievement, participation, motivation, and attitudes. As we currently stand, there is no evidence to support the conclusion that LADs have lived up to the promise of improving academic achievement. Most studies reported negligible or small effects, with limited evidence from well-powered controlled experiments. Many studies merely compared users and non-users of LADs, confounding the dashboard effect with student engagement levels. Similarly, the impact of LADs on motivation and attitudes appeared modest, with only a few exceptions demonstrating significant effects. Small sample sizes in these studies highlight the need for larger-scale investigations to validate these findings. Notably, LADs showed a relatively substantial impact on student participation. Several studies reported medium to large effect sizes, suggesting that LADs can promote engagement and interaction in online learning environments. However, methodological shortcomings, such as reliance on traditional evaluation methods, self-selection bias, the assumption that access equates to usage, and a lack of standardized assessment tools, emerged as recurring issues. To advance the research line for LADs, researchers should use rigorous assessment methods and establish clear standards for evaluating learning constructs. Such efforts will advance our understanding of the potential of LADs to enhance learning outcomes and provide valuable insights for educators and researchers alike.

  
  

2023

• López-Pernas S., Gordillo A., Barra E., Saqr M. (2023). Game learning analytics: The case of online educational escape rooms. Proceedings of the 1st Finnish Learning Analytics and Artificial Intelligence in Education Conference (FLAIEC 2022) (CEUR Workshop Proceedings), vol. 3383, pp. 121-122. https://ceur-ws.org/Vol-3383/FLAIEC22_paper_7043.pdf.

  
  Educational escape rooms are team-based learning activities in which students solve puzzles related to a certain subject to accomplish a final goal (usually escaping from a room) [1]. These activities have proven capable of increasing students’ knowledge in a variety of subjects and contexts while improving motivation and engagement. A key characteristic of educational escape rooms is that they are time-constrained. Therefore, if students do not complete all the puzzles in time they will not gain exposure to part of the learning materials in the activity. As such, it is crucial to provide timely support to students to prevent them from getting stuck and frustrated, and ensuring they progress through the activity. However, providing such support can be challenging for instructors since they often have to monitor several students at the same time, which becomes even harder in online teaching environments [2]. The Escapp platform [3] provides a solution for this challenge. Escapp is a web platform that allows to conduct online educational escape rooms. Besides providing all the features needed for instructors to set up their escape rooms both online or face-to-face, Escapp provides a learning analytics dashboard that allows to closely monitor students while they play, enabling the detection of lagging players and the provision of hints to help them advance through the escape room. The Escapp platform has been used to conduct several escape rooms at Universidad Politécnica de Madrid [4]–[8] where the learning analytics dashboard has been used to detect lagging students and to optimize the game design. In this presentation, we will show an example of one of these educational escape rooms and how the learning analytics dashboard has played a crucial role in the correct development of the activity. We will discuss current and potential uses of the dashboard and of the data collected from the students. Our goal is to offer an innovative perspective on learning analytics and how they can be adapted to the specific learning scenario of educational escape rooms.

  learning analyticsgame learning analyticseducational escape roomsdashboardgame-based learning

  
  
• López-Pernas S., Kleimola R., Väisänen S., Hirsto L. (2023). Early detection of dropout factors in vocational education: A large-scale case study from Finland. Proceedings of the 1st Finnish Learning Analytics and Artificial Intelligence in Education Conference (FLAIEC 2022) (CEUR Workshop Proceedings), vol. 3383, pp. 44-50. https://ceur-ws.org/Vol-3383/FLAIEC22_paper_8117.pdf.

  
  The aim of this study is to analyze which factors from students’ admission data can predict dropout in initial vocational education and training (VET) in Finland. The sample included 15,523 students in different fields of VET that started an initial VET between 2014 and 2021 in a large-size vocational school in Finland. The results of fitting a logistic regression model to the admission data showed that students who started a VET program after basic education were more likely to drop out, as well as students who combined their studies with a job or job-seeking. Our findings pave the pathway for further research to implement support measures for decreasing dropout that are tailored to each specific “risk group”.

  ocational education and training (vet)dropoutlearning analytics, prediction

  
  
• López-Pernas S., Saqr M. (2023). From Variables to States to Trajectories (VaSSTra): A Method for Modelling the Longitudinal Dynamics of Learning and Behaviour. Proceedings TEEM 2022: Tenth International Conference on Technological Ecosystems for Enhancing Multiculturality. TEEM 2022. Lecture Notes in Computer Science., pp. 1169-1178. doi: 10.1007/978-981-99-0942-1_123.

  
  Research in learning analytics needs longitudinal studies that explore the learner’s behaviour, disposition, and learning practices across time, a gap this article aims to bridge. We present VaSSTra: an innovative method for the longitudinal analysis of educational data that can be applied at different time scales (e.g., days, weeks, or courses), and allows the study of different aspects of learning as well as the factors that explain how such aspects evolve over time. Our method combines life-events methods with sequence analysis and consists of three steps: (1) converting variables to states (where variables are grouped into homogenous states); (2) from states to sequences (where the states are used to construct sequences across time), and (3) from sequences to trajectories (where similar sequences are grouped in trajectories). VaSSTra enables us to map the longitudinal unfolding of events while taking advantage of the wealth of life-events methods to visualize, model and describe the temporal dynamics of longitudinal activities. We demonstrate the method with a practical case study example.

  longitudinal methodslearning analyticslife-events methodssequence analysisclustering

  
  
• López-Pernas S., Gordillo A., Barra E., Saqr M. (2023). The dynamics of students’ playing profiles in a programming educational escape room (Best paper award). Proceedings TEEM 2023: Eleventh International Conference on Technological Ecosystems for Enhancing Multiculturality. TEEM 2023. Lecture Notes in Computer Science. (in-press).

  
  Educational escape rooms have gained recognition as immersive and engaging learning activities. While existing research has primarily focused on students’ perceptions and learning outcomes, little attention has been given to their performance, behavior, and interactions during these activities. This study aims to fill this gap by employing person-centered methods to analyze students’ gameplay data from a computer-based educational escape room. Using Gaussian mixture models, we identified four distinct profiles of players using their gameplay data: efficient players who complete the game smoothly with little help, supported players who make good progress with the help of hints, relentless players who devote their time to seeking help rather than working on the escape room puzzles, and laggers who make little progress and fail to obtain the help they need. We further investigate the relationships between puzzle completion times and hint-requesting behavior using Bayesian Gaussian graphical models. The findings point to hints as the key to support profiles of players that lack the necessary skills to complete the activity on their own. Lastly, we analyze the relationship between the identified profiles and knowledge acquisition during the escape room. We found that students’ initial and final knowledge differed by profile but learning gains were comparable except for the laggers who make little progress in the activity.

  educational escape roomslearning analyticsgaussian mixture modelsgame-based learning

  
  
• Saqr M., López-Pernas S. (2023). The idiographic paradigm shift needed: Bringing the person back into research and practice. Proceedings of the 1st Finnish Learning Analytics and Artificial Intelligence in Education Conference (FLAIEC 2022) (CEUR Workshop Proceedings), vol. 3383, pp. 116. https://ceur-ws.org/Vol-3383/FLAIEC22_paper_6676.pdf.

  
  Self-efficacy, self-directedness, self-regulation, autonomy, and self-control –inter alia– have all been around for decades. While such concepts may differ, they share a quintessential element which is the “self” component. As we currently stand, we are not short of theories around the importance of the “self” or the student who has become the center of all initiatives for improving education over the last decades. Furthermore, we have a vast count of empirical studies, systematic reviews, and meta-analyses of intervention that promise meaningful intervention. Yet, the quest to bring real changes on the ground has so far fell short of promise. The reasons pertain to the discord between how research is conducted, assessed, or applied. While we –theoretically– embrace and emphasize the value of the “self” or “student” as a central point of departure from existing methods or theories, research is conducted by using data from a “group” of many others. That is, data are collected from a sample of students to explore their inter-individual differences and their average behavior to derive generalizable laws or norms. Such norms are expected to apply to everyone, and the lessons learnt from studying others are expected to be generalizable. Nonetheless, such group data, are barely –if at all– represent any single person, ergo a paradigm shift is needed to bring the very person into our approach to research and practice. We show how data can be collected to model the within-person behavior and learning process. Such analysis is more representative of the “self”, offer more valid inferences about the personal processes and a better potential for personalizing and adapting education.

  learning analyticsidiographicwithin-person

  
  
• Vartiainen H., López-Pernas S., Saqr M., Kahila J., Parkki T., Tedre M., Valtonen T. (2023). Mapping students’ temporal pathways in a computational thinking escape room. Proceedings of the 1st Finnish Learning Analytics and Artificial Intelligence in Education Conference (FLAIEC 2022) (CEUR Workshop Proceedings), vol. 3383, pp. 77-88. https://ceur-ws.org/Vol-3383/FLAIEC22_paper_9625.pdf.

  
  This case study explored the applicability of sequence mining and process mining methods on qualitative video data of a group-based problem-solving situation. For the case study, audio and video data were collected from a pilot experience of an educational escape room, which was designed to practice the application of computational thinking (CT) skills. The escape room combined digital and physical affordances into CT puzzles and challenges. To examine processes and patterns of collaborative learning and problem-solving in the context of the CT escape room, video data from pre-service teachers’ game activities were collected. A unique contribution of this case study is that it demonstrates how sequence and process mining methods can be applied to a type of qualitative content analysis often found in research on collaborative learning.

  computer science educationeducational escape roomsteacher educationcollaborative learning

  
  
• Heikkinen S., López-Pernas S., Malmberg J., Tedre M., Saqr M. (2023). How do business students self-regulate their project management learning? A sequence mining study. Proceedings of the 1st Finnish Learning Analytics and Artificial Intelligence in Education Conference (FLAIEC 2022) (CEUR Workshop Proceedings), vol. 3383, pp. 51-59. https://ceur-ws.org/Vol-3383/FLAIEC22_paper_2583.pdf.

  
  The relation between learning strategies and academic achievement has been proven to be strong in multiple studies. Still, the connection between micro-level SRL processes and the academic achievement of business students in learning project management remains unstudied. The current study aims to find how sequence mining can identify students using different learning tactics and strategies in terms of micro-level SRL processes. Our findings show that there are differences in the use of tactics and strategies between low and high performing students. Understanding the differences in how low and high performing students apply different micro-level SRL processes can help practitioners identify students in need of support for SRL.

  sequence miningmicro-level srl processeslearning tacticsacademic achievementlearning analyticsproject management

  
  
• Conde M.Á., López-Pernas S., Peltekova E., Pancheva K., Raspopovic Milic M., Saqr M. (2023). Multi-stakeholder Perspective on the Gap Between Existing Realities and New Requirements for Online and Blended Learning: An Exploratory Study. Proceedings TEEM 2022: Tenth International Conference on Technological Ecosystems for Enhancing Multiculturality. TEEM 2022. Lecture Notes in Computer Science., pp. 1109-1116. doi: 10.1007/978-981-99-0942-1_117.

  
  Online and blended learning are teaching modalities that have become very popular and widespread all over the world. Applying these modalities requires specific knowledge as well as an appropriate technological infrastructure. The COVID-19 pandemic caused an important online migration in most educational institutions. In this regard, existing literature covers issues such as the impact, the challenges, the tools, the problems, etc. What could also be interesting is to understand students’, teachers’ and administrative staff’s perspectives about how blended and online learning were developed and how it is going to be applied in the future. With this in mind, the ILEDA project team has carried out an exploratory study, which takes into account these three collectives in four different European universities. From the study, it is possible to see that the institutions and their lecturers and staff were probably not prepared for the online migration and the possibilities they had were quite different from students’ expectations.

  higher educatione-learningblended learningcovid-19 periodlearning analytics

  
  
• Mairinoja L., López-Pernas S., Elmoazen R., Niskanen E.A., Kuningas T., Wärri A., Saqr M., Strauss L. (2023). International Online Team-Based Learning in Higher Education of Biomedicine - Evaluation by Learning Analytics. Proceedings of the Technology-Enhanced Learning in Laboratories Workshop (TELL 2023) (CEUR Workshop Proceedings), vol. 3393, pp. 49-60. https://ceur-ws.org/Vol-3393/TELL23_paper_1668_5.pdf.

  
  Teamwork skills are important to practice during higher educational studies to prepare students for the future working life. Since online learning has established itself as a relevant part of higher education, we present here an approach to online team-based learning and show the performance of students during the teamwork, proven by learning analysis data. In addition, results from a feedback survey of students´ opinions on teamwork are presented. Online teamwork was implemented for master level biomedicine students from four different Universities in Nordic countries, and student interaction was evaluated. Learning analytics data were collected from Discord, which was the communication platform for students and teachers during the teamwork. The Community of Inquiry (CoI) framework was used as guidance, and indicators of CoI’s social, cognitive, and teaching presences were used as a scheme for coding the interaction. To recognize the process of collaboration, the data were first analyzed by using process mining. Further, to understand the multidimensional property of collaboration, we developed a network analysis and visualized the results using Gephi and the Fruchterman-Reingold layout algorithm. The quantitative results of the feedback survey were analyzed by using descriptive statistics and visualized using the R package likert. The learning analytics data included 316 posts divided to 686 annotations, which were categorized to codes. Our results indicate that the most frequent codes were the ones related to the social dimension of CoI, determined with attributes such as ‘interactive’ (173), ‘cohesion’ (119) and ‘affective’ (116). The remaining most frequent codes alternated between ‘facilitation’ and ‘cognition’. Thus, social presence, in the context of CoI was considerable in our online team-based learning approach. However, to enhance students’ cognitive presence, and thereby their ability to construct and confirm meaning of what they are learning, students’ work should be facilitated by increasing teaching presence through teacher’s contribution online. In line with the learning analytics data, the results of the survey pointed out the need of more in-depth instructions on how to carry out the team exercises, which belongs to the teaching presence category in the frame of CoI. Based on the results of this study and the existing literature, we aim to improve our teambased learning approach and outcomes in the future by increasing students’ contribution through regular feedback assignments during the work and encouraging learners to reflect on their own work, contribution and thinking.

  online teamworklearning analyticsdiscordvirtual collaborative learningonline teambased learningcommunity of inquiry

  
  
• Huhta K., López-Pernas S., Saqr M. (2023). Mapping the topics, trends, and themes of education technology in legal education with topic modeling and network analysis. Proceedings TEEM 2023: Eleventh International Conference on Technological Ecosystems for Enhancing Multiculturality. TEEM 2023. Lecture Notes in Computer Science. (in-press).

  
  The digital transformation varies markedly across disciplines in the way technologies are used, how much, and for what purposes to foster educational innovation. In legal education, they are used for various purposes to respond to the capacities and competencies that are required from contemporary lawyers and legal professionals. This article addresses a gap in existing research by approximating digital technologies and digital transformation in legal education research. It uses topic modeling and network analysis to explore the digital transformation in legal education research and to demonstrate how digital technologies used for pedagogical purposes are reflected in legal education research. It finds that while digitalization is a clear recent trend in legal education research, the role of digital technologies in legal education research is not as strong as in other fields of higher education and that practical skills and the practice of law continue to have a central role in legal education irrespective of the education technologies used.

  legal educationeducation technologybibliometricstopic modeling

  
  
• Saqr M., López-Pernas S. (2023). What learning analytics can tell about students’ approach in a learning analytics course. Proceedings TEEM 2023: Eleventh International Conference on Technological Ecosystems for Enhancing Multiculturality. TEEM 2023. Lecture Notes in Computer Science. (in-press).

  
  Learning analytics emerged more than a decade ago to harness the power of data to understand and optimize learning, learners’ behavior, and learning environments. Ever since, the field has grown to encompass a diverse range of methods, research strands and traditions. Recent literature reviews tell us that most common applications of learning analytics include predictive analytics, social network analysis, sequence and process analysis, visualizations, and dashboards to mention a few. In the same vein, the research field has attracted several interdisciplinary researchers and practitioners from computer science, education, data science, engineering, administration, and from the education technology industry. Whereas such diverse backgrounds and perspectives bring a wealth of different perspectives to the field, it makes teaching and learning analytics hard to narrow down in a single course. This study reports on the analysis of students’ approach to learning learning analytics, reflects on the insights that learning analytics offers, and makes recommendations for future researchers who are teaching or investigating similar courses.

  learning analyticssequence analysisdata miningpsychological networkscomputer science education

  
  
• Akçapınar G., López-Pernas S., Er E., Saqr M. (2023). How a learning analytics dashboard intervention influences the dynamics of students’ learning behavior (Best paper award)More data is not always better data: A learning analytics case study in early prediction. Proceedings TEEM 2023: Eleventh International Conference on Technological Ecosystems for Enhancing Multiculturality. TEEM 2023. Lecture Notes in Computer Science. (in-press).

  
  Interventions play a crucial role in completing the learning analytics cycle. However, there is limited research available on how students utilize these interventions or whether there is any change in their learning behaviors following the intervention. Existing studies primarily rely on students’ self-report perceptions, while neglecting the temporal aspect of the data in data-driven studies. This study examines the impact of a learning analytics intervention in the form of a learning analytics dashboard provided to students in a remote programming course on their learning behaviors. To achieve this goal, learning sessions before and after the introduction of the dashboard were identified using students’ learning traces in the learning management system. Subsequently, these learning sessions were analyzed using sequence analysis, process mining, and Bayesian Gaussian graphical models. Assignment submissions, formative quizzes, forum interactions, interactions with video materials, and participation in live classes were considered to determine students’ learning behaviors. The findings of the study indicate that there were changes in students’ learning behaviors after the introduction of the dashboard. Specifically, before the dashboard, learning sessions were mainly focused on graded activities such as assignments and quizzes, whereas after the dashboard, there was an increase in interactions with non-graded activities such as video materials. The results are also supported by the process mining analysis.

  learning analyticsdashboard interventionsequence analysisprocess miningbayesian gaussian graphical models

  
  
• Rai P., López-Pernas S., Saqr M. (2023). More data is not always better data: A learning analytics case study in early prediction. Proceedings TEEM 2023: Eleventh International Conference on Technological Ecosystems for Enhancing Multiculturality. TEEM 2023. Lecture Notes in Computer Science. (in-press).

  
  This study aimed to explore the early prediction and forecasting of students’ performance using learning analytics methods. We do so by examining a large number of students at four different time points and employing four machine learning algorithms. We seek to fill a gap in the literature regarding the effect of data volume across time. The results revealed several important findings. To begin with, the final course data did not consistently outperform earlier time points across all performance indicators. Surprisingly, forecasts based on data from the first week had reasonable accuracy, implying that preemptive interventions can be implemented early on. Furthermore, predictions made on second-week data performed the best, probably due to students’ initial motivation and early differentiation among those who were actively involved. Furthermore, decision trees (DT) emerged as the most effective early prediction method, consistently displaying acceptable performance across all time points. This study has far-reaching ramifications. It indicates that employing learning analytics for early prediction is not only viable, but also dependable, with decent accuracy. Although further research is needed to corroborate these findings in diverse circumstances, the second week of the course looks to be a vital stage for generating correct predictions. Furthermore, when compared to other algorithms, DT stands out as a superior early prediction algorithm.

  learning analyticsearly predictionspredicting performancedata mining

  
  
• Hirsto L., Saqr M., López-Pernas S., Valtonen T. (2023). A systematic narrative review of learning analytics research in K-12 and schools. Proceedings of the 1st Finnish Learning Analytics and Artificial Intelligence in Education Conference (FLAIEC 2022) (CEUR Workshop Proceedings), vol. 3383, pp. 60-67. https://ceur-ws.org/Vol-3383/FLAIEC22_paper_9536.pdf.

  
  The field of learning analytics emerged in the last decade to take advantage of the increasing availability of data about learners that digital systems generate. Existing research in learning analytics has focused on higher education, as this context often relies heavily on digital platforms such as online learning management systems, making data collection easier. In this paper, we focus on LA research in the context of elementary level teaching. We provide a systematic narrative review in which we analyze the articles that had the most impact in the field. Our results show the existence of some recurring themes such as gamification and multimodal methods. We make a distinction between papers in which learning analytics is the target of the study (e.g., dashboards) and papers in which learning analytics methods were used as a means to study a given behavior/skill/phenomenon (e.g., problem-solving skills). Lastly, we found that most studies lack a strong theoretical foundation on education science and, thus, there is a need to develop more elaborated theoretical perspectives in future research on school-level learning analytics, as well as papers that deliver a real impact on learning and teaching.

  learning analyticsk-12elementary schoolliterature revieweducational data mining

  
  
• Conde M.Á., Georgiev A., López-Pernas S., Jovic J., Crespo-Martínez I., Raspopovic Milic M., Saqr M., Pancheva K. (2023). Definition of a Learning Analytics Ecosystem for the ILEDA Project Piloting. Learning and Collaboration Technologies. HCII 2023. Lecture Notes in Computer Science., vol. 14040, pp. 444-453. doi: 10.1007/978-3-031-34411-4_30.

  
  Understanding how students progress in their learning is an important step towards achieving the success of the educational process. One way of understanding student progress is by using learning analytics methods on different student data. The ILEDA project aims to improve online and blended learning by using educational data analytics. For this purpose, the project involves four universities from four different countries and develops several activities. One of these activities. That aims to facilitate the analysis of student progress, is the definition of a Learning Analytics Ecosystem. The aim of defining the ecosystem is to generate solutions that will benefit all institutions and that will allow to look for possible patterns and common issues needing addressing. This paper describes the development of such an ecosystem and its future implementations.

  learning analyticsinteroperabilityecosystemsevidencesdashboards

  
  

2022

• Saqr M., López-Pernas S., García Á.H., Conde M.Á., Poquet O. (2022). Networks and Learning Analytics: Addressing Educational Challenges. Proceedings of the NetSciLA22 workshop (CEUR Workshop Proceedings), vol. 3258, pp. 1-3. https://ceur-ws.org/Vol-3258/xpreface.pdf.

  
  Network Analysis is an established method in learning analytics research. Network Analysis has been used to analyze learners’ interactions, to inform learning design, and to model students’ performance. The workshop entitled “Using Network Science in Learning Analytics: Building Bridges towards a Common Agenda”, carried out within the LAK2021 conference, resulted in valuable insights and outcomes: guidelines for better reporting, methodological improvements, and discussions of several novel research threads. Traditionally, the focus of the conversation has been on methodological issues of network analysis. This year, we would like to extend the conversation by slightly shifting the focus to what network analysis can do to improve learning and educational opportunities. As such, this new edition of the workshop aims to build on the fruitful achievements of the previous iteration to address new themes, which we refer to as “challenges and opportunities” in relation to practice. This edition of the workshop sought contributions around examples of applications and impact, including those that can help address societal challenges embedded within educational practices and those that foster an open conversation about privacy and ethical implications of network data. © 2022 Copyright for this paper by its authors.

  learning analyticsnetwork analysisnetwork sciencesocial network analysis

  
  
• Saqr M., López-Pernas S. (2022). The Why, the What and the How to Model a Dynamic Relational Learning Process with Temporal Networks. Proceedings of the NetSciLA22 workshop (CEUR Workshop Proceedings), vol. 3258, pp. 33-40. https://ceur-ws.org/Vol-3258/article_4.pdf.

  
  Research on online learning has benefited from intensive data collection to understand students’ online behavior and performance. Several learning analytics techniques have been operationalized to examine the temporal nature of learning that includes changes, phases, and sequences of students’ online actions. Moreover, to account for the relational nature of learning, researchers have harnessed the power of network analysis to model the relational dimensions of data, mapping connections between learners and resources, and discovering interacting communities. However, prior research has rarely combined the two aspects (temporal and relational), but rather most researchers rely on aggregate networks where the time dimension has been ignored. To combine both these aspects, temporal networks provide a rich framework of statistical and visualization techniques that allow to fully understand, for instance, the evolution and building up of learning communities, the sequence of co-construction of knowledge, the flow of information, and the building of social capital, to name a few examples. Since temporal networks have been rarely used in educational research, with this study, we aim to provide an introduction to this method, with an emphasis on the differences with conventional static networks. We explain the basics of temporal networks, the different subtypes thereof, and the measures that can be taken, as well as examples from the few existing prior works. © 2022 Copyright for this paper by its authors.

  learning analyticssocial network analysistemporal network analysis

  
  
• Saqr M., López-Pernas S., Hernández-García A., Conde M.A., Poquet O. (2022). Concluding remarks of the NetSciLA22 Workshop. Proceedings of the NetSciLA22 workshop (CEUR Workshop Proceedings), vol. 3258, pp. 41-46. https://ceur-ws.org/Vol-3258/article_5.pdf.

  
  The NetScila22 workshop builds on the previous iterations of network analysis workshops. The current year themes addressed educational challenges as well as opportunities for future research and for strengthening the community. The workshop included valuable discussions and interactions with both experts and emerging researchers. Such discussions were augmented by a survey that gathered insights form workshop attendees. The discussants recommended improving methodological rigor, leveraging methods that positively impact learning, address data issues, e.g., collection, privacy and reporting as well as better alignment with theory. Other recommendations proposed human-centred artificial intelligence approaches grounded on cognitive science, better communication with stakeholders, sharing ideas within the community and organizing hands-on seminar. The workshop also included presentations that address methodological advances and future opportunities, e.g., temporal networks, semantic networks and attention network. © 2022 Copyright for this paper by its authors.

  learning analyticsnetwork analysisnetwork sciencesemantic networkstemporal network analysis

  
  
• Saqr M., López-Pernas S. (2022). Instant or Distant: A Temporal Network Tale of Two Interaction Platforms and Their Influence on Collaboration. Educating for a New Future: Making Sense of Technology-Enhanced Learning Adoption. EC-TEL 2022. Lecture Notes in Computer Science, vol. 13450, pp. 594-600. doi: 10.1007/978-3-031-16290-9_55.

  
  This study compared two iterations of the same course where students had the same assignments. In the first iteration, the students had to use the typical discussion forums offered by the popular Moodle learning management system. In the second iteration, students had to use Discord, the popular gaming chat application. Students’ interactions were retrieved from both platforms and cleaned. Two social networks were constructed using the same methods to evaluate the differences in patterns of interaction between the two platforms, the group interactivity, the reciprocity, and the quality of interactions. The aim is to study how far an instant messenger facilitates or otherwise constrains collaboration. We use temporal network methods to further understand the pace, rhythm, and temporality of interactions. © 2022, The Author(s), under exclusive license to Springer Nature Switzerland AG.

  cscllearning analyticssocial network analysistemporal network analysis

  
  

2021

• López-Pernas S., Saqr M. (2021). Idiographic learning analytics: A within-person ethical perspective. Companion Proceedings 11th International Conference on Learning Analytics & Knowledge (LAK21), pp. 369-374. https://www.solaresearch.org/core/lak21-companion-proceedings/.

  
  One of the main obstacles impeding the widespread use and adoption of learning analytics is the threat that it poses to students’ data privacy. In this article, we present a proposal for generating person-centered insights for learners by combining data from multiple sources while preserving students’ privacy. The basis of our approach is idiographic learning analytics, in which data are collected and insights are generated for each student individually. On the one hand, all the data collection and processing are performed locally on the student’s device, thus preserving student privacy. On the other hand, being based on person-based methods, the idiographic approach helps deliver personalized insights.

  ethicslearning analyticsidiographicprivacy

  
  
• Saqr M., López-Pernas S. (2021). Idiographic learning analytics: A definition and a case study. Proceedings - IEEE 21st International Conference on Advanced Learning Technologies, ICALT 2021, pp. 163-165. doi: 10.1109/ICALT52272.2021.00056.

  
  Idiographic methods have emerged as a way to examine individual behavior by using several data points from each subject to create person-specific insights. In the field of learning analytics, such methods could overcome the limitations of cross-sectional group-level data that may fail to capture the dynamic processes that unfold within each individual learner and less likely to offer relevant personalized learning or support. In this study, we provide a definition of idiographic learning analytics and we explore the possible potentials of this method to zoom in on the fine-grained dynamics of a single student. Specifically, we make use of Gaussian Graphical Models - an emerging trend in network science - to analyze a single student’s dispositions and devise insights specific to him/her. Our findings offer a proof of concept of the potential of this novel method in revealing personalized valuable insights about students’ self-regulation. While our specific findings apply to a single student, our method applies to every student regardless of context. © 2021 IEEE.

  idiographic learning analyticslearning analyticsnetwork sciencepsychological networksraphical gaussian models

  
  
• Saqr M., López-Pernas S. (2021). The Dire Cost of Early Disengagement: A Four-Year Learning Analytics Study over a Full Program. Technology-Enhanced Learning for a Free, Safe, and Sustainable World. EC-TEL 2021. Lecture Notes in Computer Science, vol. 12884, pp. 122-136. doi: 10.1007/978-3-030-86436-1_10.

  
  Research on online engagement is abundant. However, most of the available studies have focused on a single course. Therefore, little is known about how students’ online engagement evolves over time. Previous research in face-to-face settings has shown that early disengagement has negative consequences on students’ academic achievement and graduation rates. This study examines the longitudinal trajectory of students’ online engagement throughout a complete college degree. The study followed 99 students over 4 years of college education including all their course data (15 courses and 1383 course enrollments). Students’ engagement states for each course enrollment were identified through Latent Class Analysis (LCA). Students who were not engaged at least one course in the first term was labeled as “Early Disengagement”, whereas the remaining students were labeled as “Early Engagement”. The two groups of students were analyzed using sequence pattern mining methods. The stability (persistence of the engagement state), transition (ascending to a higher engagement state or descending to a lower state), and typology of each group trajectory of engagement are described in this study. Our results show that early disengagement is linked to higher rates of dropout, lower scores, and lower graduation rates whereas early engagement is relatively stable. Our findings indicate that it is critical to proactively address early disengagement during a program, watch the alarming signs such as presence of disengagement during the first courses, declining engagement along the program, or history of frequent disengagement states. © 2021, Springer Nature Switzerland AG.

  early disengagementlearning analyticstrajectories of engagement

  
  
• Saqr M., López-Pernas S. (2021). Idiographic learning analytics: A single student (N=1) approach using psychological networks. Proceedings of the NetSciLA21 workshop (CEUR Workshop Proceedings), vol. 2868, pp. 16-22. https://ceur-ws.org/Vol-2868/article_4.pdf.

  
  Recent findings in the field of learning analytics have brought to our attention that conclusions drawn from cross-sectional group-level data may not capture the dynamic processes that unfold within each individual learner. In this light, idiographic methods have started to gain grounds in many fields as a possible solution to examine students’ behavior at the individual level by using several data points from each learner to create person-specific insights. In this study, we introduce such novel methods to the learning analytics field by exploring the possible potentials that one can gain from zooming in on the fine-grained dynamics of a single student. Specifically, we make use of Gaussian Graphical Models -an emerging trend in network science- to analyze a single student’s dispositions and devise insights specific to him/her. The results of our study revealed that the student under examination may be in need to learn better self-regulation techniques regarding reflection and planning. © 2021 Copyright for this paper by its authors.

  graphical gaussian modelsidiographic learning analyticsnetwork sciencepsychological networks

  
  
• Saqr M., López-Pernas S. (2021). Idiographic learning analytics: A single student (N=1) approach using psychological networks. Proceedings of the NetSciLA21 workshop (CEUR Workshop Proceedings), vol. 2868, pp. 16-22. https://ceur-ws.org/Vol-2868/article_4.pdf.

  
  Recent findings in the field of learning analytics have brought to our attention that conclusions drawn from cross-sectional group-level data may not capture the dynamic processes that unfold within each individual learner. In this light, idiographic methods have started to gain grounds in many fields as a possible solution to examine students’ behavior at the individual level by using several data points from each learner to create person-specific insights. In this study, we introduce such novel methods to the learning analytics field by exploring the possible potentials that one can gain from zooming in on the fine-grained dynamics of a single student. Specifically, we make use of Gaussian Graphical Models -an emerging trend in network science- to analyze a single student’s dispositions and devise insights specific to him/her. The results of our study revealed that the student under examination may be in need to learn better self-regulation techniques regarding reflection and planning. © 2021 Copyright for this paper by its authors.

  graphical gaussian modelsidiographic learning analyticsnetwork sciencepsychological networks

  
  
• Apiola M., Tedre M., López-Pernas S., Saqr M., Daniels M., Pears A. (2021). A Scientometric Journey Through the FIE Bookshelf: 1982-2020. Proceedings - Frontiers in Education Conference, FIE. doi: 10.1109/FIE49875.2021.9637209.

  
  IEEE/ASEE Frontiers in Education turned 50 at the 2020 virtual conference in Uppsala, Sweden. This paper presents an historical retrospective on the first 50 years of the conference from a scientometric perspective. That is to say, we explore the evolution of the conference in terms of prolific authors, communities of co-authorship, clusters of topics, and internationalization, as the conference transcended its largely provincial US roots to become a truly international forum through which to explore the frontiers of educational research and practice. The paper demonstrates the significance of FIE for a core of 30% repeat authors, many of whom have been members of the community and regular contributors for more than 20 years. It also demonstrates that internal citation rates are low, and that the co-authoring networks remain strongly dominated by clusters around highly prolific authors from a few well known US institutions. We conclude that FIE has truly come of age as an international venue for publishing high quality research and practice papers, while at the same time urging members of the community to be aware of prior work published at FIE, and to consider using it more actively as a foundation for future advances in the field. © 2021 IEEE.

  
  

2020

• López-Pernas S., Jimenez A., Gordillo A., Barra E., Marco L., Quemada J. (2020). Ediphy: A modular and extensible open-source web authoring tool for the creation of interactive learning resources. Proceedings of the 2020 16th International Conference on Intelligent Environments, IE 2020, art. no. 9154949, pp. 115-121. doi: 10.1109/IE49459.2020.9154949.

  
  Nowadays, the use of authoring tools has become the most popular way in which teachers create learning resources. These tools commonly facilitate the creation of educational content by combining text, self-assessed activities, and multimedia resources, often allowing authors to export the result in a standard format. Although there are many tools available, it is difficult to find a tool that meets the needs of all users. In order to fulfill the requirements of specific learning scenarios, one possible solution is to adapt an existing e-Learning authoring tool. However, to do so in an easy way, authoring tools should be built on the principles of modular architecture to facilitate and encourage developers to contribute, which is often not the case. This article presents Ediphy, a modular and extensible open-source web authoring tool for the creation of interactive educational content. Its modular nature, based on plugins, makes it easy to extend the tool with new functionalities to meet the requirements of specific educational settings. In order to evaluate Ediphy, two surveys have been conducted: one among end users who have used Ediphy to create learning resources and another one among developers who have contributed to its improvement. Results show that end users have a great opinion of Ediphy and developers find it easy to contribute. © 2020 IEEE.

  authoring toolscomputer-aided instructione-learningeducational technologylearning objects

  
  

2019

• López-Pernas S., Gordillo A., Barra R., Quemada J. (2019). Identification, analysis of requirements for a web platform for managing educational escape rooms. Proceedings of the 12th annual International Conference of Education, Research and Innovation (ICERI 2019), pp. 4874-4883. doi: 10.21125/iceri.2019.1191.

  
  Educational escape rooms demand a great effort on the part of instructors, not only when designing and building the activities, but also when the time comes to conduct them. Although previous works have presented tools to ease the creation of educational puzzle-like games, no prior studies have reported on the use, development or design of any tool for managing educational escape rooms. A tool of this kind would help teachers overcome one of the main barriers they face when trying to incorporate these activities into their teaching. This work identifies and analyses for the first time the requirements that a platform for managing educational escape rooms should satisfy. A platform meeting all the requirements gathered would be able to help teachers in all the steps of the process of conducting an educational escape room, including configuration, integration of the activity into a virtual learning environment, student registration, team creation, control of activity execution (management of resources, narrative events and gamification elements during the activity), progress monitoring, hint delivery, assessment, and gathering of student feedback and data on learning effectiveness.

  educational escape roomsgamificationeducational software

  
  
• Mareca P., López-Pernas S. (2019). Learning 3.0: Animations and creativity on Wikipedia [Aprendizaje 3.0: Animaciones y creatividad en Wikipedia]. Iberian Conference on Information Systems and Technologies, CISTI, art. no. 8760974. doi: 10.23919/CISTI.2019.8760974.

  
  Digital supports in education with their different orientations are, nowadays, a reality. It is essential to refine and improve the procedures and educational tools in order to contribute to the improvement of the quality of students’ learning as well as to a wider dissemination of knowledge. Hence, a new educational model arises, e-Learning 3.0, supported by Web 3.0. With this work we aim to contribute to e-Learning 3.0 by highlighting the importance of the image in achieving student learning, especially that of the first courses of the University and Technical Institutes. In this article we detail and highlight the importance of figures and graphics in an educational project that is carried out at Universidad Politécnica de Madrid, ‘Wikifísica’, in which imagery and creative media become prime vehicles for the process of learning, highlighting the advantages they bring to it.. © 2019 AISTI.

  animationscollaborative learningimage editionlearning 3.0wikipedia

  
  
• Munoz-Arcentales A., López-Pernas S., Pozo A., Alonso A., Salvachua J., Huecas G. (2019). An architecture for providing data usage and access control in data sharing ecosystems. Procedia Computer Science, vol. 160, pp. 590-597. doi: 10.1016/j.procs.2019.11.042.

  
  We are experiencing a new digital revolution in which data are becoming a key pillar for business and industry. Promoting data sharing, without compromising data sovereignty and traceability, is fundamental since it provides a heterogeneous ecosystem with the potential to enrich the variety of applications and services that take part in this digital revolution. In this scope, the use of secure and trusted platforms for sharing and processing personal and industrial data is crucial for the creation of a data market and a data economy. Protecting data goes beyond restricting who can access what resource (covered by identity and access control respectively): It becomes necessary to control how data are treated, which is known as data usage control. Data usage control provides a common and trustful security framework to guarantee the sovereignty and the responsible use of organizations’ data by third-party entities, easing and ensuring data sharing in ecosystems such as industry or smart cities. In this article, we present an architecture proposal for achieving access and usage control in shared data ecosystems among multiple organizations. The proposed architecture is based on the UCON (Usage Control) model and an extended XACML (extensible Access Control Markup Language) Reference Architecture, relying on key aspects of the IDS (International Data Spaces) Reference Architecture Model. Its modular design and technology-agnostic nature provide an integral solution while maintaining flexibility of implementation. © 2019 The Authors. Published by Elsevier B.V.

  data economydata usage controlinternational data spacesuconusage policiesxacml

  
  
• Gordillo A., López-Pernas S., Barra E. (2019). Students’ perceptions toward the use of teacher-created educational games in a secondary education setting. Proceedings of the 12th annual International Conference of Education, Research and Innovation (ICERI 2019), pp. 1986-1996. doi: 10.21125/iceri.2019.0557.

  
  Substantial research has been devoted to educational video games, which has provided broad empirical evidence that playing educational video games can lead to positive impacts in terms of motivation and learning outcomes. However, there is still a lack of studies examining the acceptance and learning effectiveness of educational video games created by teachers using authoring tools. This paper contributes to filling this gap in the literature by examining secondary school students’ perceptions toward the use of educational video games created by teachers using an authoring tool. A student survey was used as data collection instrument. A total of 62 students (47 seventh grade students and 15 eighth grade students) assessed 5 different teacher-created educational games. The results show that students had a very good overall opinion of the games, and that they found them engaging, easy to use and beneficial for their learning. The results also show that students agreed that the games made learning fun and that they prefer the game-based learning approach over traditional teaching materials. In conclusion, this paper provides evidence that teachers can easily create educational video games toward which students have positive attitudes if they are provided with suitable authoring tools.

  game-based learning;educational games;serious games;authoring tools

  
  
• Alonso Á., Gordillo A., Pozo A., López-Pernas S., Marcos L., Barra E. (2019). Extending the EIDAS European Specification for Supporting Academic Attributes. Proceedings of the 12th annual International Conference of Education, Research and Innovation (ICERI 2019), pp. 2008-2014. doi: 10.21125/iceri.2019.0560.

  
  Secure Electronic Identification (eID) is one of the key enablers of data protection, privacy and prevention of online fraud. However, to date, the lack of a common legal basis prevented European Member States from recognising and accepting eIDs issued by the other Member States. The electronic IDentification, Authentication and trust Service (eIDAS) regulation solves these issues by allowing citizens of any European country to use their national eIDs to securely access public and private e-services provided in other European countries. However, the minimum dataset typically provided by the Member States only contains citizens’ personal attributes. Therefore, academic services that aim to facilitate the mobility of students within the European Union cannot exploit the advantages of integrating students’ eIDs to the same extent as if they included attributes related to their academic profile as well. In this article, we propose an extension of the eIDAS specification in order to support academic attributes. Thanks to this extension, services can request students’ information from the eIDAS nodes: not only their personal profiles but also additional attributes related to their academic profile. In this work, we also propose an architecture that allows the connection of the national eIDAS nodes to academic attribute providers in order to enrich the student minimum dataset with their academic attributes. We conclude that thanks to the extension of the eID profile of students with academic attributes, e-services in higher education sectors will be able to fully benefit from the integration of the eIDAS initiative, b​r​e​a​

  academic attributeselectronic identificationeidaserasmusidentity

  
  
• Quemada J., Barra E., Gordillo A., Pavón S., Salvachúa, J., Vazquez I., López-Pernas S. (2019). AMMIL: A methodology for developing video-based learning courses. Proceedings of the 12th annual International Conference of Education, Research and Innovation (ICERI 2019), pp. 4893-4901. doi: 10.21125/iceri.2019.1195.

  
  Videos are extensively used nowadays to support learners in a variety of educational settings such as traditional online courses, MOOCs (Massive Open Online Courses), flipped classrooms, and blended courses. Therefore, it is of prime importance to develop methodologies capable of producing effective video-based learning courses which can lead to the highest success and learner satisfaction. AMMIL (Active Meaningful Micro-Inductive Learning) is a methodology for creating video-based learning courses, which aims to maximise the instructional effectiveness in terms of motivation and academic performance in self-learning environments. This methodology has been successfully used to create several MOOCs, as well as to support a blended programming course at a higher education institution. This paper presents the AMMIL methodology and an evaluation of two different MOOCs and a higher education programming course in which this methodology was applied. This evaluation was conducted by using student surveys as data collection instruments. The results are very promising since they show that students were very satisfied with the courses created applying the AMMIL methodology.

  moocflipped classroomself-learningproject-based learningactive learningvideo in educationmeaningful learninginductive learningmicrolearning

  
  

2018

• López-Pernas S., Benito A., Marco L., Gordillo A. (2018). Improval of an educational platform through the integration of an extensible e-learning authoring tool. Proceedings of the 11th annual International Conference of Education, Research and Innovation (ICERI 2018), pp. 10117-10124. doi: 10.21125/iceri.2018.0898.

  
  E-learning authoring tools play a crucial role in the development of learning content facilitating the creation of digital learning resources. However, these tools are frequently independent pieces of software which do not encourage users to share their resources and create a community of educators and learners. By including authoring tools as part of collaborative Virtual Learning Environments, content creators can easily publish their resources for other users to enjoy and contribute. This paper presents a new extensible e-Learning authoring tool called Ediphy that has been included as a part of ViSH, a social and collaborative e-Learning platform focused on the creation and sharing of open educational resources. The details of the integration of Ediphy into ViSH are explained in this paper, along with the complete set of benefits that this authoring tool provides to the e-Learning platform. A survey was conducted in order to collect ViSH users’ opinions on Ediphy. The results of this survey show that users had a very good perception of Ediphy and believed it signifies a substantial improvement of the ViSH platform.

  e-learningauthoring toolsvirtual learning environmentscomputer aided instruction

  
  
• Gordillo A., López-Pernas S., Barra E. (2018). RESCORM: A boilerplate for creating SCORM-compliant React applications. Proceedings of the 11th annual International Conference of Education, Research and Innovation (ICERI 2018), pp. 8843-8853. doi: 10.21125/iceri.2018.0632.

  
  Interactive web-based learning resources created according to e-Learning standards like SCORM can benefit students, teachers and the whole education system in a wide variety of ways. Although the use of authoring tools is the preferable way to create these resources since it does not require authors to have strong computer skills, the created resources are very limited in terms of customization and functionality. Therefore, on many occasions web-based learning resources still need to be created by developers using web technologies. Despite this fact, little work has been done to provide developers with resources to facilitate the creation of web-based learning resources compliant with e-Learning standards, especially for novel web technologies like React. This paper presents RESCORM (https://github.com/agordillo/rescorm): a boilerplate for creating SCORM-compliant React applications that aims to facilitate developers the creation of SCORM-compliant learning resources. This paper also presents the results of a survey conducted among nine developers who used RESCORM to create learning resources and the results of an evaluation of two resources developed using RESCORM that were used in an online course. These results show that RESCORM achieved high user acceptance and was perceived as very useful, and that the resources created with RESCORM were rated high in terms of usefulness.

  web-based educationscormreacteducational applicationslearning objects

  
  
• Marco L., López-Pernas S., Alonso A. (2018). Accessibility review for web-based learning tools and materials. Proceedings of the 11th annual International Conference of Education, Research and Innovation (ICERI 2018), pp. 2393-2402. doi: 10.21125/iceri.2018.1525.

  
  Society has changed significantly in the last few years. Technology has evolved rapidly, and new devices and tools are transforming the way we communicate. One of the most important milestones of this transformation was the democratization of the Internet and specifically the Web, a key factor for determining people’s connectivity. However, not every people have the same conditions to access web resources and most of the available services and applications in the Internet are not ready for being used in an inclusive manner. This paper presents an accessibility review for web-based learning tools and materials. We establish the difference between reactive and proactive methods for designing web applications, considering the former as a response once the application is ready and the latter as the inclusion of accessibility requirements from the design phase. After the analysis, we conclude that nowadays the most common accessibility approaches for designing web-based learning tools are those whose methodological basis are grounded in reactive principles. It is necessary to move forward and propose solutions that contemplate inclusive design from the early stages of development.

  e-learninglearning toolsweb accessibilityuniversal designapproachable development

  
  

Book Chapters

2024

• López-Pernas S., Misiejuk K., Kaliisa R., Conde M. Á., Saqr M. (2024). Capturing the Wealth and Diversity of Learning Processes with Learning Analytics Methods. Learning analytics methods and tutorials: A practical guide using R (in-press). https://lamethods.github.io/chapters/ch01-intro/intro.html.

  
  The unique position of learning analytics at the intersection of education and computer science while reaching out to several other disciplines such as statistics, psychometrics, econometrics, mathematics, and linguistics has accelerated the growth and expansion of the field. Therefore, it is a crucial endeavor for learning analytics researchers to stay abreast of the latest methodological and computational advances to drive their research forward. The diversity and complexity of the existing methods can make this task overwhelming both for newcomers to the learning analytics field and for experienced researchers. With the motivation to accompany researchers in this challenging journey, the book “Learning Analytics Methods and Tutorials - A Practical Guide Using R” aims to provide a methodological guide for researchers to study, consult, and embark upon the first steps toward innovation in the learning analytics field. Thanks to the unique wealth of authors’ backgrounds and expertise, which include authors of R packages and experts in methods and applications, the book offers a comprehensive array of methods that are described thoroughly with a primer on their usage in prior research in education. These methods include sequence analysis, Markov models, factor analysis, process mining, network analysis, predictive modeling, and cluster analysis among others. A step-by-step tutorial using the R programming language with real-life datasets and case studies is presented for each method. In addition, the initial chapters are devoted to getting novice researchers up to speed with the R programming learners and the basics of data analysis. The present chapter serves as an introduction to the book describing its main aim and intended audience. It describes the structure of the book and the methods covered by each chapter. It also points the readers to the companion code and data repositories to facilitate following the tutorials present in the book chapter.

  learning analyticseducational data miningtutorialmethodslearning analyticscurriculum

  
  
• López-Pernas S., Misiejuk K., Tikka S., Saqr M., Kopra J., Heinäniemi M. (2024). Visualizing and Reporting Educational Data with R. Learning analytics methods and tutorials: A practical guide using R (in-press). https://lamethods.github.io/chapters/ch06-data-visualization/ch6-viz.html.

  
  Visualizing data is central in learning analytics research, underpins learning dashboards, and is a prime method for reporting results and insights to stakeholders. In this chapter, the reader will be guided through the process of generating meaningful and aesthetically pleasing visualizations of different types of student data using well-known R packages. The main visualization types will be demonstrated with an explanation of their usage and use cases. Furthermore, learning-related examples will be discussed in detail. For instance, readers will learn how to visualize learners’ logs extracted from learning management systems to show how trace data can be used to track students’ learning activities. In addition to creating compelling plots, readers will also be able to generate professional-looking tables with summary statistics.

  learning analyticsdata visualizationggplot2visual analytics

  
  
• López-Pernas S., Saqr M. (2024). Modelling the Dynamics of Longitudinal Processes in Education: The VaSSTra Method. Learning analytics methods and tutorials: A practical guide using R (in-press). https://lamethods.github.io/chapters/ch11-vasstra/ch11-vasstra.html.

  
  Modeling a longitudinal process in educational research brings a lot of variability over time. The modeling procedure becomes even harder when using multivariate continuous variables, e.g., clicks on learning resources, time spent online, and interactions with peers. In fact, most human behavioral constructs are an amalgam of interrelated features with complex fluctuations over time. Modeling such processes requires a method that takes into account the multidimensional nature of the examined construct as well as the temporal evolution. In this chapter we describe the VaSSTra method, which combines person-based methods, sequence analysis and life-events methods. Throughout the chapter, we discuss how to derive states from different variables related to students, how to construct sequences from students’ longitudinal progression of states, and how to identify and study distinct trajectories of sequences that undergo a similar evolution. We also cover some advanced properties of sequences that can help us analyze and compare trajectories. We illustrate the method through a tutorial using the R programming language.

  learning analyticssequence analysislife-events methodsperson-based methodslongitudinal methods

  
  
• López-Pernas S., Saqr M. (2024). The Why, the How, and the When of Educational Process Mining in R. Learning analytics methods and tutorials: A practical guide using R (in-press). https://lamethods.github.io/chapters/ch14-process-mining/ch14-process.html.

  
  This chapter presents the topic of process mining applied to learning analytics data. The chapter begins by introducing the fundamental concepts of the method, with a focus on event log construction and visual representation using directly-follows graphs. A review of the existing literature on educational process mining is also presented to introduce the reader to the state of the art of the field. The chapter follows with a guided tutorial in R on how to apply process mining to trace log data extracted from an online learning management system. The tutorial uses the bupaverse framework for data handling and visualization. We finish the chapter with a reflection on the method and its reliability and applicability.

  process miningbupaverselearning analyticseducational data mining

  
  
• López-Pernas S., Saqr M., Conde J., Del-Río-Carazo L. (2024). A Broad Collection of Datasets for Educational Research Training and Application. Learning analytics methods and tutorials: A practical guide using R (in-press). https://lamethods.github.io/chapters/ch02-data/ch2-data.html.

  
  In this chapter, we present the main types of data that are commonly used in learning analytics research. Learning analytics has grown to encompass the digital trails left by online learning technologies —clicks, events, and interactions—, sensor data and self-reports among others. We present a collection of curated real-life open datasets that represent the most common types of educational data. The datasets have been collected from diverse sources such as learning management systems, online forums, and surveys. These datasets are used throughout the book to illustrate methods of analysis such as sequence analysis, social network analysis, Markov models, predictive analytics and structure equation modeling, to mention a few. Each data set in the chapter is presented with its context, main properties, links to the original source, as well as a brief exploratory data analysis.

  learning analyticsdatasetsopen dataeducational data mining

  
  
• López-Pernas S., Saqr M., Helske S., Murphy K. (2024). Multichannel Sequence Analysis in Educational Research Using R. Learning analytics methods and tutorials: A practical guide using R (in-press). https://lamethods.github.io/chapters/ch13-multichannel/ch13-multi.html.

  
  This chapter introduces multi-channel sequence analysis, a novel method that examines two or more synchronised sequences. While this approach is relatively new in social sciences, its relevance to educational research is growing as researchers gain access to diverse multimodal temporal data. Throughout this chapter, we describe multi-channel sequence analysis in detail, with an emphasis on how to detect patterns within the sequences, i.e., clusters —or trajectories— of multi-channel sequences that share similar temporal evolutions (or similar trajectories). To illustrate this method we present a step-by-step tutorial in R that analyses students’ sequences of online engagement and academic achievement, exploring their longitudinal association. We cover two approaches for clustering multi-channel sequences: one based on using distance-based algorithms, and the other employing mixture hidden Markov models inspired by recent research.

  learning analyticsmultichannel sequence analysismixture hidden markov modelsmultimodal data

  
  
• Jongerling J., López-Pernas S., Saqr M., Vogelsmeier L. (2024). Structural Equation Modeling with R for Education Scientists. Learning analytics methods and tutorials: A practical guide using R (in-press). https://lamethods.github.io/chapters/ch21-sem/ch21-sem.html.

  
  Structural Equation Modeling (SEM) is a method for modeling the multitude of relationships between latent variables and the observable indicators, as well as the relationship between the latent variables themselves to test theories. In its most common form, SEM combines confirmatory factor analysis (CFA with another method named path analysis. Just like CFA, SEM relates observed variables to latent variables that are measured by those observed variables and, as path analysis does, SEM allows for a wide range of regression-type relations between sets of variables (both latent and observed). This chapter presents an introduction to SEM, an integrated strategy for conducting SEM analysis that is well-suited for educational sciences, and a tutorial on how to carry out an SEM analysis in R.

  structural equation modelinglearning analyticssem

  
  
• Jovanovic J., López-Pernas S., Saqr M. (2024). Predictive Modelling in Learning Analytics using R. Learning analytics methods and tutorials: A practical guide using R (in-press). https://lamethods.github.io/chapters/ch07-prediction/ch7-pred.html.

  
  Prediction of learners’ course performance has been a central theme in learning analytics (LA) since the inception of the field. The main motivation for such predictions has been to identify learners who are at risk of low achievement so that they could be offered timely support based on intervention strategies derived from analysis of learners’ data. To predict student success, numerous indicators, from varying data sources, have been examined and reported in the literature. Likewise, a variety of predictive algorithms have been used. The objective of this chapter is to introduce the reader to predictive modelling in LA, through a review of the main objectives, indicators, and algorithms that have been operationalized in previous works as well as a step-by-step tutorial of how to perform predictive modelling in LA using R. The tutorial demonstrates how to predict student success using learning traces originating from a learning management system, guiding the reader through all the required steps from the data preparation all to the evaluation of the built models.

  learning analyticspredictive modellingr programmingtutorialmachine learning

  
  
• Murphy K., López-Pernas S., Saqr M. (2024). Dissimilarity-based Cluster Analysis of Educational Data: A Comparative Tutorial using R. Learning analytics methods and tutorials: A practical guide using R (in-press). https://lamethods.github.io/chapters/ch08-clustering/ch8-clus.html.

  
  Clustering is a collective term which refers to a broad range of techniques aimed at uncovering patterns and subgroups within data. Interest lies in partitioning heterogeneous data into homogeneous groups, whereby cases within a group are more similar to each other than cases assigned to other groups, without foreknowledge of the group labels. Clustering is also an important component of several exploratory methods, analytical techniques, and modelling approaches and therefore has been practiced for decades in education research. In this context, finding patterns or differences among students enables teachers and researchers to improve their understanding of the diversity of students —and their learning processes— and tailor their supports to different needs. This chapter introduces the theory underpinning dissimilarity-based clustering methods. Then, we focus on some of the most widely-used heuristic dissimilarity-based clustering algorithms; namely, K-Means, K-Medoids, and agglomerative hierarchical clustering. The K-Means clustering algorithm is described including the outline of the arguments of the relevant R functions and the main limitations and practical concerns to be aware of in order to obtain the best performance. We also discuss the related K-Medoids algorithm and its own associated concerns and function arguments. We later introduce agglomerative hierarchical clustering and the related R functions while outlining various choices available to practitioners and their implications. Methods for choosing the optimal number of clusters are provided, especially criteria that can guide the choice of clustering solution among multiple competing methodologies —with a particular focus on evaluating solutions obtained using different dissimilarity measures— and not only the choice of the number of clusters for a given method. All of these issues are demonstrated in detail with a tutorial in R using a real-life educational data set.

  agglomerative hierarchical clusteringaverage silhouette widthdissimilarity-based clusteringk-meansk-medoidsleaning analytics

  
  
• Saqr M., López-Pernas S., Conde M. Á., Hernández-García Á. (2024). Social Network Analysis: A Primer, a Guide and a Tutorial in R. Learning analytics methods and tutorials: A practical guide using R (in-press). https://lamethods.github.io/chapters/ch15-sna/ch15-sna.html.

  
  This chapter introduces the concept and methods of social network analysis (SNA) with a detailed guide to analysis with real world data using the R programming language. The chapter first introduces the basic concepts and types of networks. Then the chapter goes through a detailed step by step analysis of networks, computation of graph level measures as well as centralities with a concise interpretation in a collaborative environment. The chapter concludes with a discussion of network analysis, next steps as well as a list of further readings.

  learning analyticssocial network analysiscentrality measurestutorial

  
  
• Saqr M., López-Pernas S., Helske S., Durand M., Murphy K., Studer M., Ritschard G. (2024). Sequence Analysis in Education: Principles, Technique, and Tutorial with R. Learning analytics methods and tutorials: A practical guide using R (in-press). https://lamethods.github.io/chapters/ch10-sequence-analysis/ch10-seq.html.

  
  Sequence analysis is a data mining technique that is increasingly gaining ground in learning analytics. Sequence analysis enables researchers to extract meaningful insights from sequential data, i.e., to summarize the sequential patterns of learning data and classify those patterns into homogeneous groups. In this chapter, readers will become familiar with sequence analysis techniques and tools through real-life step-by-step examples of sequential trace log data of students’ online activities. Readers will be guided on how to visualize the common sequence plots and interpret such visualizations. An essential part of sequence analysis is the discovery of patterns within sequences through clustering techniques. Therefore, this chapter will demonstrate the various sequence clustering methods, calculator of cluster indices, and evaluation of clustering results.

  sequence analysissequence mininglearning analytics

  
  
• Saqr M., Beck E., López-Pernas S. (2024). Psychological Networks: A Modern Approach to Analysis of Learning and Complex Learning Processes. Learning analytics methods and tutorials: A practical guide using R (in-press). https://lamethods.github.io/chapters/ch19-psychological-networks/ch19-psych.html.

  
  In the examination of psychological phenomena within educational environments, a multitude of variables come into play, and these variables have the potential to interact with, trigger, and exert influence on one another. To grasp the intricate dependencies among these variables, investigating the linear associations between each variable pair is not enough. Instead, this complexity demands the application of more advanced techniques that capture the full spectrum of interactions between these variables. One of such techniques is psychological networks. In contrast to social networks, where nodes typically represent individuals and edges signify their interactions or relationships, psychological networks differ in that the nodes represent observed psychological variables, and the edges denote the statistical relationships between them. This chapter serves as an introduction to psychological networks within educational research, offering a tutorial on their estimation, visualization, and interpretation using the R programming language.

  psychological networkspartial correlation networkscomplex systemslearning analytics

  
  
• Saqr M., Schreuder M. J., López-Pernas S. (2024). Why educational research needs a complex system revolution that embraces individual differences, heterogeneity, and uncertainty. Learning analytics methods and tutorials: A practical guide using R (in-press). https://lamethods.github.io/chapters/ch22-conclusion/ch22-conclusion.html.

  
  Whereas the field of learning analytics has matured, several methodological and theoretical issues remain unresolved. In this chapter, we discuss the potentials of complex systems as an overarching paradigm for understanding the learning process, learners and the learning environments and how they influence learning. We show how using complex system methodologies open doors for new possibilities that may contribute new knowledge and solve some of the unresolved problems in learning analytics. Furthermore, we unpack the importance of individual differences in advancing the field bringing a much-needed theoretical perspective that could help offer answers to some of our pressing issues.

  learning analyticscomplex systemsindividual differences

  
  
• Scrucca L., Saqr M., López-Pernas S., Murphy K. (2024). An Introduction and R Tutorial to Model-based Clustering in Education via Latent Profile Analysis. Learning analytics methods and tutorials: A practical guide using R (in-press). https://lamethods.github.io/chapters/ch09-model-based-clustering/ch9-model.html.

  
  Heterogeneity has been a hot topic in recent educational literature. Several calls have been voiced to adopt methods that capture different patterns or subgroups within students’ behavior or functioning. Assuming that there is “an average” pattern that represents the entirety of student populations requires the measured construct to have the same causal mechanism, same development pattern, and affect students in exactly the same way. Using a person-centered method (finite Gaussian mixture model or latent profile analysis), the present tutorial shows how to uncover the heterogeneity within engagement data by identifying three latent or unobserved clusters. This chapter offers an introduction to the model-based clustering that includes the principles of the methods, a guide to choice of number of clusters, evaluation of clustering results and a detailed guide with code and a real-life dataset. The discussion elaborates on the interpretation of the results, the advantages of model-based clustering as well as how it compares with other methods.

  gaussian mixture modellatent profile analysismodel-based clusteringlearning analytics

  
  
• Vogelsmeier L. V. D. E., Saqr M., López-Pernas S., Jongerling J. (2024). Factor Analysis in Education Research using R. Learning analytics methods and tutorials: A practical guide using R (in-press). https://lamethods.github.io/chapters/ch20-factor-analysis/ch20-factor.html.

  
  Factor analysis is a method commonly employed to reduce a large number of variables into fewer numbers of factors. The method is often used to identify which observable indicators are representative of latent, not directly-observed constructs. This is a key step in developing valid instruments to assess latent constructs in educational research (e.g., student engagement or motivaion). The chapter describes the two main approaches for conducting factor analysis in detail and provides a tutorial on how to implement both techniques with the R programming language. The first is confirmatory factor analysis (CFA), a more theory-driven approach, in which a researcher actively specifies the number of underlying constructs as well as the pattern of relations between these dimensions and observed variables. The second is exploratory factor analysis (EFA), a more data-driven approach, in which the number of underlying constructs is inferred from the data, and all underlying constructs are assumed to influence all observed variables (at least to some degree).

  factor analysisexploratory factor analysisconfirmatory factor analysislearning analytics

  
  
• Helske J., Helske S., Saqr M., López-Pernas S., Murphy K. (2024). A Modern Approach to Transition Analysis and Process Mining with Markov Models in Education. Learning analytics methods and tutorials: A practical guide using R (in-press). https://lamethods.github.io/chapters/ch12-markov/ch12-markov.html.

  
  This chapter presents an introduction to Markovian modelling for the analysis of sequence data. Contrary to the deterministic approach seen in the previous sequence analysis chapters, Markovian models are probabilistic models, focusing on the transitions between states instead of studying sequences as a whole. The chapter provides an introduction to this method and differentiates between its most common variations: first-order Markov models, hidden Markov models, mixture Markov models, and mixture hidden Markov models. In addition to a thorough explanation and contextualisation within the existing literature, the chapter provides a step-by-step tutorial on how to implement each type of Markovian model using the R package seqHMM. The chapter also provides a complete guide to performing stochastic process mining with Markovian models as well as plotting, comparing and clustering different process models.

  markov modelslearning analyticssequence analysistransition analysis

  
  
• Hernández-García Á., Cuenca-Enrique C., Traxler A., López-Pernas S., Conde M. Á., Saqr M. (2024). Community Detection in Learning Networks Using R. Learning analytics methods and tutorials: A practical guide using R (in-press). https://lamethods.github.io/chapters/ch16-community/ch16-comm.html.

  
  In the field of social network analysis, the quest for understanding interactions and group structures takes a center stage. This chapter focuses on finding such groups, constellations or ensembles of actors who can be grouped together, a process often referred to as community detection, particularly in the context of educational research. Community detection aims to uncover tightly knit subgroups of nodes who share strong connectivity within a network or have connectivity patterns that demarcates them from the others. This chapter explores various algorithms and techniques that unveil these groups or cohesive clusters. Using well-known R packages, the chapter primarily delves into the core approach of identifying and visualizing densely connected subgroups, offering practical insights into its application within educational contexts. Ultimately, the chapter aims to serve as a guide to unraveling learning communities, providing educators and researchers with valuable tools to discern and harness the power of interconnectedness in learning networks.

  community detectionsocial network analysislearning analyticseducational data mining

  
  
• Kopra J., Tikka S., Heinäniemi M., López-Pernas S., Saqr M. (2024). An R Approach to Data Cleaning and Wrangling for Education Research. Learning analytics methods and tutorials: A practical guide using R (in-press). https://lamethods.github.io/chapters/ch04-data-cleaning/ch4-datacleaning.html.

  
  Data wrangling, also known as data cleaning and preprocessing, is a critical step in the data analysis process, particularly in the context of learning analytics. This chapter provides an introduction to data wrangling using R and covers topics such as data importing, cleaning, manipulation, and reshaping with a focus on tidy data. Specifically, readers will learn how to read data from different file formats (e.g. CSV, Excel), how to manipulate data using the dplyr package, and how to reshape data using the tidyr package. Additionally, the chapter covers techniques for combining multiple data sources. By the end of the chapter, readers should have a solid understanding of how to perform data wrangling tasks in R.

  data wranglingdata cleaningtidyverser programminglearning analytics

  
  
• Tikka S., Kopra J., Heinäniemi M., López-Pernas S., Saqr M. (2024). Getting started with R for Education Research. Learning analytics methods and tutorials: A practical guide using R (in-press). https://lamethods.github.io/chapters/ch03-intro-r/ch3-intor.html.

  
  The R programming language has become a popular tool for conducting data analysis in the field of learning analytics. This chapter provides an introduction to the basics of R programming, with a focus on the Rstudio integrated development environment and the tidyverse programming paradigm. The chapter covers topics such as data types and structures, control structures, pipes, functions, loops, and input/output operations. By the end of the chapter, readers should have a solid understanding of the basics of R programming and have the tools necessary to learn more in-depth topics such as data wrangling and basic statistics using R.

  r programmingr studiolearning analytics

  
  
• Tikka S., Kopra J., Heinäniemi M., López-Pernas S., Saqr M. (2024). Introductory Statistics with R for Educational Researchers. Learning analytics methods and tutorials: A practical guide using R (in-press). https://lamethods.github.io/chapters/ch05-basic-stats/ch5-stats.html.

  
  Statistics play a fundamental role in learning analytics, providing a means to analyze and make sense of the vast amounts of data generated by learning environments. This chapter provides an introduction to basic statistical concepts using R and covers topics such as measures of central tendency, variability, correlation, and regression analysis. Specifically, readers will learn how to compute descriptive statistics, conduct hypothesis tests, and perform simple linear regression analysis. The chapter also includes practical examples using realistic data sets from the field of learning analytics. By the end of the chapter, readers should have a solid understanding of the basic statistical concepts and methods commonly used in learning analytics, as well as a practical understanding of how to use R to conduct statistical analysis of learning data.

  statisticslearning analyticshypothesis testing

  
  
• Muñoz-Arcentales J.A., Conde J., Alonso A., Salvachua J., Velasquez W., López-Pernas S. (2024). Data fusion and homogenization: Two key aspects for building digital twins of smart spaces. Smart Spaces. doi: 10.1016/B978-0-443-13462-3.00002-9.

  
  Digital twins are virtual replicas of physical objects, systems, or processes. With the advancement of artificial intelligence and internet of things, digital twins have become an increasingly valuable tool in the context of smart spaces to improve efficiency, reduce costs, and enhance user experience. However, one of the main challenges for the successful implementation of digital twins is combining data from multiple sources in a homogeneous format so they can be operationalized together to generate more accurate, comprehensive, and valuable information. This chapter thoroughly describes the implementation of a digital twin of a smart space using different technologies from the FIWARE ecosystem that help combat data fusion and homogenization problems. This process involves several steps, including data selection, preprocessing, integration, and interpretation. In addition, a case study on low-emission zones is presented, which can be easily replicated in different scenarios within a smart city. Lastly, we discuss the advantages of using FIWARE as the underlying technology in our implementation to overcome the data fusion and homogenization challenge.

  digital twinsdata fusionfiwarengsi-ldsmart spaces

  
  

2023

• López-Pernas S., Saqr M., Apiola M. (2023). Scientometrics: A Concise Introduction and a Detailed Methodology for Mapping the Scientific Field of Computing Education Research. Past, Present and Future of Computing Education Research: A Global Perspective. Springer, pp. 79-99. doi: 10.1007/978-3-031-25336-2_5.

  
  Scientometrics has emerged as a research field for the evaluation and mapping of scientific fields, exploring research themes, collaboration clusters and identifying gaps and future trends. While early implementations have focused on quantitative metrics, recent directions emphasize a more nuanced approach that combines qualitative methods with quantitative analysis that triangulates several aspects, e.g., temporal trends, network, and semantic analysis. This chapter reviews scientometrics as a research methodology and discusses the strengths and weaknesses and how such weaknesses can be amended. The chapter also discusses the main methodological approach, and its theoretical underpinnings, used in some of the book chapters that make use of scientometrics as a means to map the field of CER.

  scientometricscomputing education researchbibliometrics

  
  
• López-Pernas S., Apiola M., Saqr M., Pears A., Tedre M. (2023). A Scientometric Perspective on the Evolution of the SIGCSE Technical Symposium: 1970–2021. Past, Present and Future of Computing Education Research: A Global Perspective, Springer, pp. 193-212. doi: 10.1007/978-3-031-25336-2_10.

  
  ACM’s Special Interest Group on Computer Science Education’s (SIGCSE) Technical Symposium is considered by many to be ACM’s flagship conference for computing educators. SIGCSE Technical Symposium has been held annually since 1970 as an in-person conference, with the exception of 2020, when it was cancelled (with some papers presented in 2021). The conference attracts many computing educators, numbering in the thousands in recent times and is by far the top publication outlet in computer science education with regards to number of published articles. This chapter explores the evolution of the first 51 years of SIGCSE from its inception in 1970 to the present day, using primarily scientometric data. We explore the evolution of the SIGCSE conference with regards to shifts in research themes, influential authors, author networks and clusters of keywords. We also explore the potential for internationalization of the conference. Participation in the SIGCSE symposium has strong US roots, and we examine the impact on participation as ACM SIGCSE membership expanded to Europe and Australasia, and new conferences such as ACE, ICER and Koli Calling established themselves.

  computing education researchscientometrics

  
  
• Apiola M., López-Pernas S., Saqr M., Malmi L., Daniels M. (2023). Exploring the Past, Present and Future of Computing Education Research: An Introduction. Past, Present and Future of Computing Education Research: A Global Perspective, Springer, pp. 1-7. doi: 10.1007/978-3-031-25336-2_1.

  
  This chapter is an introduction to the book “Past, Present and Future of Computing Education Research: A Global Perspective.” This book uses a mixture of scientometrics, meta-research and case studies to offer a new view about the evolution and current state of computing education research (CER) as a field of science. In its 21 chapters, this book presents new insights of authors, author communities, publication venues, topics of research, and of regional initiatives and topical communities of CER. This chapter presents an overview of the contents of this book.

  
  
• Apiola M., López-Pernas S., Saqr M. (2023). The Venues that Shaped Computing Education Research: Dissemination Under the Lens. Past, Present and Future of Computing Education Research: A Global Perspective, Springer, pp. 121-150. doi: 10.1007/978-3-031-25336-2_7.

  
  A fundamental part of science is dissemination. Previous research has analysed dissemination practices in few of the central publication outlets. In this chapter, we add into that research by analysing dissemination practices of CER in a range of 1523 journals and conference series that are (a) exclusively dedicated to CER, and (b) in outlets that publish CER together with other topics, such as general education, engineering or computer science. Our results show that a small and highly dedicated core of venues publishes a remarkable share of CER, that CER has a conference-oriented publication tendency, significant variations in citation rates, and differences in the diversity of topics within and between the dedicated and non-dedicated publication outlets. In all, our macro-level analysis makes a significant contribution into the research of dissemination in CER.

  
  
• Saqr M., López-Pernas S., Apiola M. (2023). Capturing the Impact and the Chatter Around Computing Education Research Beyond Academia in Social Media, Patents, and Blogs. Past, Present and Future of Computing Education Research: A Global Perspective. Springer, pp. 171-191. doi: 10.1007/978-3-031-25336-2_9.

  
  Research impact goes beyond academia and exists in the multiplicity of digital platforms that we use to read, share, and discuss knowledge. Computing education research (CER) is no exception: it is created in academia and typical research institutions but is talked about widely on social media, blogs, and news websites. The aim of this study is to have a comprehensive analysis of how research in CER has been received, talked about in social media, discussed on blogs, and spread to the news and media. In addition to common analysis of trends of growth, we analyze trends of usage of social media and quantitative analysis of platforms, articles, and venues. The analysis also includes which articles and in which subfields had a wide impact, and for whom (i.e., which platforms had more impact). The results show that Altmetrics adoption is weak, yet increasingly growing fast. Gender and diversity issues made it to popular news sites, e.g., Scientific American, Los Angeles Times, and Christian Science Monitor, while articles about ethics, programming education, introductory courses as well as computational thinking and inclusion have captured the attention of social media users. There was weak—or no—correlation between article, author or topic impact and the traditional impact measures, e.g., citation count.

  theoretical computer sciencemedia studiescomputing educationcomputer science educationaltmetricbibliometricsscientometricssocial mediatwitter

  
  
• Agbo F.J., Ntinda M., López-Pernas S., Saqr M., Apiola M. (2023). Computing Education Research in the Global South. Past, Present and Future of Computing Education Research: A Global Perspective, Springer, pp. 311-333. doi: 10.1007/978-3-031-25336-2_15.

  
  Scientometric analyses of publication data from all major computing education research (CER) outlets show that many countries and whole continents are greatly underrepresented on the global map of contributions to CER. For example, only a minor portion of CER has originated from countries in the Global South (GS) or has addressed challenges of computing education in the GS. In this chapter, we shift the focus to scientometrically analyse CER papers that originate from countries in the GS. From the metadata of all CER publications in central publication outlets of CER, we have selected a subset of articles with authors affiliated to an institution in a GS Country, as defined by the United Nations (UN). The analysis shows publication trends, prolific authors, and country collaboration patterns. A number of crucial and interesting avenues for future research and collaboration are presented.

  
  
• Apiola M., Saqr M., López-Pernas S. (2023). The Hands that Made Computing Education Research: Top Authors, Networks, Collaboration and Newcomers. Past, Present and Future of Computing Education Research: A Global Perspective, Springer, pp. 101-119. doi: 10.1007/978-3-031-25336-2_6.

  
  Computing Education Research (CER), like any other discipline of science, is fundamentally driven by people and their networks of collaboration. Previous research has analysed authorship patterns and collaboration in distinct well known dissemination outlets of CER. In this chapter, we add to that approach by analysing a comprehensive set of metadata of CER publications. We analyse author productivity including newcomer patterns, clusters of co-authorship and international collaboration, and authors who build bridges between communities. Our results reveals top authors and their production before and after 2000, clusters of collaborators and their areas of topics as revealed by top keywords, a healthy evolution of newcomer-patterns, and a set of authors who build bridges between communities. In all, our macro-level analysis adds a significant contribution to our understanding of the role of authors in the evolution of CER.

  computing education researchauthorsscientometrics

  
  
• Apiola M., Saqr M., López-Pernas S. (2023). The Evolving Themes of Computing Education Research: Trends, Topic Models, and Emerging Research. Past, Present and Future of Computing Education Research: A Global Perspective, Springer, pp. 151-169. doi: 10.1007/978-3-031-25336-2_8.

  
  A combined body of reviews, meta-research and other analyses demonstrates the evolution of computing education research (CER) through the decades with experience reports evolving to empirical research, increased attention paid to educational research, methods and reporting rigor. Previous analyses of CER publications show the sustained focus of CER on programming education, which has, by far, been the all-time most popular topic in CER. In the recent decade, other top researched areas include K-12 computing education and computational thinking. In this chapter, we add new insights to the top research areas of CER. We followed the PRISMA-S (Preferred Reporting Items for Systematic reviews and Meta-Analyses) literature search extension to capture the relevant literature on CER. The process of data retrieval, screening, and pre-processing resulted in a total of 16,863 articles included in the dataset. We use a combination of keyword analysis and structural topic modeling, and introduce a model of 29 topics. We also introduce emerging topics in recent years through an analysis of emerging common words in abstracts and titles during recent years. The results paint a unique picture about the dominating and trending research areas of CER, and of how common research topics are connected with each other. The analysis also reveals under-researched areas of CER.

  computing education researchscientometricskeywords

  
  
• Dagienė V., Gülbahar Y., Grgurina N., López-Pernas S., Saqr M., Apiola M., Stupurienė G. (2023). Computing Education Research in Schools. Past, Present and Future of Computing Education Research: A Global Perspective, Springer, pp. 481-520. doi: 10.1007/978-3-031-25336-2_20.

  
  One of the most researched domains of computing education research (CER) that attracts attention is computing education in schools, starting from pre-primary level up to upper secondary level (K-12). A high number of initiatives and related research contributions have appeared over half a century of computing history in schools. This chapter presents an overview of CER in the K-12 domain, including globally influential movements such as that of Logo pedagogy, constructionism, inquiry based learning or computational thinking (CT). Development of CT in education, based on a number of previous reviews on CT and K-12, paints a diverse picture of the approaches, educational technologies, pedagogical innovations, and related challenges such as lack of teacher training or shortage of learning resources. This article presents also a scientometric overview of CER research in the K-12 domain. The analysis identifies the top topics of research, and foundational articles. While much of the research is centered around the US, key research from other parts of the globe is also highlighted. Emergence of new trends such as teaching artificial intelligence and machine learning in schools are also discussed.

  
  
• Sointu E., Väisänen S., Hirsto L., Paavilainen T., Saqr M., López-Pernas S., Valtonen T. (2023). Creatively Opening the Constraints of Learning Analytics in Inclusive, Elementary School-Level STEAM Education. The Power of Creativity in the Classroom (in-press).

  
  Learning analytics has been a topic of great interest to researchers and practitioners over the past decade. The challenge, however, is translating rich data into practice. Moreover, learning analytics in inclusive elementary school settings has been little studied. This article discusses using learning analytics in elementary school STEAM education, focusing on outer space content to address this gap. We adopted a collaborative approach in preservice teacher training with in-service teachers and researchers in the university practice school setting. Fifty-two 11–13- year-old students from the Finnish school context participated in this study. We used a process-oriented approach with sequence and process mining for data analysis. The results showed little to no difference between students with and without pedagogical support. Technologies such as learning analytics in various learning management systems can present both opportunities and challenges for students with support needs. However, this study’s results challenge the assumption that students with support needs in inclusive settings are less able to work independently and find coherent learning strategies in digital learning environments.

  learning analyticselementary schoolinclusive educationsteamsupport

  
  
• Malmi L., Hellas A., Ihantola P., Isomöttönen V., Jormanainen I., Kilamo T., Knutas A., Korhonen A., Laakso M.-J., López-Pernas S., Poranen T., Salakoski T., Suhonen J. (2023). Computing Education Research in Finland. Past, Present and Future of Computing Education Research: A Global Perspective, Springer, pp. 335-372. doi: 10.1007/978-3-031-25336-2_16.

  
  Despite being a small country, Finland has been highly visible in international Computing Education Research (CER). This is demonstrated by the presence of several important research groups, dozens of graduated PhD students in CER during the last 20 years, and the success of the Koli Calling International Conference of CER, which has been running for 20 years now. In this chapter, we present the development of the CER field in Finland, the profiles of various research groups, and the roles of several national level networking activities which have supported the field. We discuss factors behind the strong presence and success of CER in Finnish universities.

  
  

Publications %>%   tidyr::separate_rows(Keywords,sep="; ") %>% mutate(Keywords=stringr::str_to_lower(Keywords)) %>% group_by(Keywords) %>% count() %>% arrange(desc(n))

# A tibble: 292 × 2
# Groups:   Keywords [292]
   Keywords                       n
   <chr>                      <int>
 1 learning analytics            62
 2 <NA>                          18
 3 computer science education    12
 4 educational escape rooms      11
 5 sequence analysis              9
 6 social network analysis        9
 7 sequence mining                8
 8 bibliometrics                  7
 9 educational technology         7
10 game-based learning            7
# ℹ 282 more rows