Cohn, C.; Snyder, C.; Fonteles, J.H.; Ashwin, T.S.; Montenegro, J.; Biswas, G. “A multimodal approach to support teacher, researcher and AI collaboration in STEM+C learning environments.” British Journal of Educational Technology, 2024, DOI: 10.1111/bjet.13518.
Advances in generative AI and multimodal learning analytics (MMLA) are creating innovative ways to support K-12 students’ collaborative learning in STEM+C fields. However, AI systems alone often struggle to interpret students’ emotions, understand social interactions, or grasp domain-specific challenges, especially in open-ended learning environments. To address these limitations, this study explores a hybrid human-AI approach that combines AI efficiency with human insight.
A collaboration framework is introduced where teachers and researchers use an AI-generated multimodal timeline to guide feedback for students tackling STEM+C problems, such as building computational models. Through discussions with a high school teacher, key moments when students face challenges were identified: the “difficulty threshold” (when problems arise) and the “intervention point” (when teacher feedback should occur). The teacher emphasized the need for a delay between these moments to allow students to work through challenges independently—a feature often missing in AI-driven systems.
Findings show that the multimodal timeline helps teachers provide better feedback while researchers refine the timeline based on teacher input. This iterative process highlights how human-AI collaboration can improve learning outcomes. Future applications could include developing tools for teacher interventions, creating pedagogical agents, and aiding curriculum design.
FIGURE 1
C2STEM Truck Task (Snyder et al., 2024).