Using Machine Learning to Detect SMART Model Cognitive Operations in Mathematical Problem-Solving Process
Abstrakt
Self-regulated learning (SRL) is a critical component of mathematics problem-solving. Students skilled inSRL are more likely to effectively set goals, search for information, and direct their attention and cognitiveprocess so that they align their efforts with their objectives.
An influential framework for SRL, the SMARTmodel (Winne, 2017), proposes that five cognitive operations (i.e., searching, monitoring, assembling,rehearsing, and translating) play a key role in SRL. However, these categories encompass a wide range ofbehaviors, making measurement challenging – often involving observing individual students and recordingtheir think-aloud activities or asking students to complete labor-intensive tagging activities as they work.
Inthe current study, to achieve better scalability, we operationalized indicators of SMART operations anddeveloped automated detectors using machine learning. We analyzed students’ textual responses andinteraction data collected from a mathematical learning platform where students are asked to thoroughlyexplain their solutions and are scaffolded in communicating their problem-solving process.
Due to the rarityin data for one of the seven SRL indicators operationalized, we built six models to reflect students’ use offour SMART operations. These models are found to be reliable and generalizable, with AUC ROCs rangingfrom .76-.89.
When applied to the full test set, these detectors are relatively robust to algorithmic bias,performing well across different student populations and with no consistent bias against a specific group ofstudents.