Is Academ ic Tracking Related to Gains in Learning Competence? Combining Propensity Score Matching and Analysis of Differential Item Functioning
Abstract
Academic tracking has been present in many countries. The central argument for tracking is that homogeneous classes permit a focused curriculum and appropriately paced instruction that leads to the maximum learning by all students (Hanushek & Wößmann, 2006).
The question whether tracking leads to better learning is still open. Hattie's (2002) meta-analysis found minimal effect of grouping students into classes according to their ability on learning outcomes, unless the ability grouping is associated with changes in curriculum and teaching practices.
On the contrary, studies from Germany where students are allocated to different schools rather than classes, provide evidence that academically selective schools can generate higher achievement gains than non-academic schools in mathematics, foreign languages, reading skills and also in intelligence (Becker et al., 2012, Guill, Lüdtke, & Köller, 2017), even if intake differences between tracks are accounted for. While some studies warn that academic tracking is associated with larger disparities in educational achievement (Hanushek & Wößmann, 2006) and with over-representation of students from minorities and socioeconomically disadvantaged families in low-track schools or classes (Hattie, 2002), studies showing higher gains in selective tracks may support the claim that the benefits of tracking outweigh its negative impact on educational equity.
Nevertheless, the effect of selective schools may differ in different countries and for various constructs of interest. This study focuses on learning competence, one of the most frequently cited key competences.
Our understanding of learning competence is inspired by the efforts of EU network on learning to learn, which defined three dimensions of learning competence: affective, cognitive, and metacognitive. Bearing this complex structure in mind, our study focuses on the cognitive part, i.e. general cognitive skills that facilitate learning across school subjects.
Just like other competences, these skills can - and should - be developed by school instruction. To our best knowledge, no previous research has studied gains in learning competence with respect to academic tracking.
On one hand, selective academic tracks may help develop learning competence by providing more demanding learning tasks and more stimulating peer interactions (Guill et al., 2017). On the other hand, heavier load of frontal teaching, if present in academic tracks, may prevent the students from using hihger-order thinking skills during school instruction and result in minimal gains in learning competence.
We will analyze: 1. Gains in overall learning competence in two student cohorts based on ability tracking, 2.
Scores on subscales at the baseline and in gains, and 3. Binary data of answers to individual items.
Previous research (Martinková et al., 2017) suggests that even if there is no difference in the overall gain between two tracks, the difference may exist in subscales or in functioning of individual items. Such detailed analysis of student results may draw the educators' attention to specific domains of school instruction where different tracks succeed or fall short in improving student learning.
Method We use data from Czech Longitudinal Study in Education (CLoSE). We follow a sample of Grade 4 students who took the 2011 TIMSS and PIRLS test in their transition to lower secondary level in Grade 6 and Grade 9.
Representative sample of basic schools (BS) was complemented by representative sample of Gymnasia (AS). We use data of 4,326 students (2,953 from BS, 1,373 from AS) who completed tests of mathematics, Czech language, reading, as well as learning competence, and have no missing data on context variables.
We measure the learning competence by a Czech adaptation of Learning to Learn instrument (LtL), developed from a Finnish assessment tool (Hautamäki & Kupiainen, 2014) by Chvál and Straková (2014). The instrument we use includes 7 cognitive subscales measuring analytical reasoning, spatial reasoning, formal operational thinking and other higher-order thinking skills that are considered to be malleable by good teaching.
Meta-cognitive and motivational dimensions of learning competence are not included in the present study. Propensity score matching (PSM) is used to form parallelized samples of AS and non-academic BS and to eliminate the effect of the selective school intake.
We fit linear regression models on total scores and subscores to test for overall school context effect using the same variables as covariates as in PSM. Further, we analyze scores and gains on subcomponents, and moreover, we check for differential item functioning (DIF) in Grade 6, Grade 9 using logistic regression model and in change (DIF-C) using multinomial regression model.
For all analyses, free statistical software R, version 3.5.1 (R Core Team, 2018) is used. For the PSM procedure, we use R package MatchIt (Ho et al., 2011).
For the DIF detection, we use package difR (Magis et al., 2010) and plots from R package ShinyItemAnalysis (Martinková & Drabinová, 2018). DIF-C is analyzed using R package difNLR (Drabinová & Martinková, 2018).
Expected Outcomes While baseline LtL distributions were identical for the two groups due to used restriction in propensity score matching, we find significant baseline difference in Grade 6 on subscale Mental arithmetics and significant DIF in items 1A and 1E of this subscale. This may have to do with the preparation of AS students for the admission test, which includes this type of mathematical word problems.
There is no difference in change of the total LtL score from Grade 6 to Grade 9 between the two tracks. This contradicts not only to general expectations, but also to studies (Becker et al., 2012; Guill et al., 2017) from Germany, a country with similarly tracked school system, where significant effects of academic-track schools on intelligence gains were found.
This may relate to the fact that in Czech Republic, a significant proportion of high-ability students stay in BS and enter short academic track after Grade 7 or Grade 9. Another reason may be a limited coverage of learning competence by school instruction in both tracks.
Most importantly, while no significant difference between AS and BS is present for the overall learning competence gains, we find significantly higher gains in Mathematical concepts subscale and significant DIF-C in some items. This may suggest that AS succeeds in developing abstract reasoning (items 6A, 6B, 6D, and 6E) and logical thinking (item 7B) in their students more than BS.
This may be result of different approach to teaching mathematics, or possibly also of different background and training of the mathematics teachers in AS. We argue that item-level analysis is important for deeper understanding of the tracking implications and may provide basis for more precise evidence-based decisions regarding the tracking policy.
References Becker, M., Lüdtke, O., Trautwein, U., Köller, O., & Baumert, J. (2012). The differential effects of school tracking on psychometric intelligence: Do academic-track schools make students smarter? Journal of Educational Psychology, 104(3), 682-699. http://dx.doi.org/10.1037/a0027608 Chvál, M., & Straková, J. (2014).
Možnosti měření kompetencí k učení - aplikace finského nástroje v českém prostředí. [Possible ways to measure the learning competence - application of a Finnish instrument in Czech context]. Pedagogika, 64(3), 307-326.
Guill, K., Lüdtke, O., & Köller, O. (2017). Academic tracking is related to gains in students' intelligence over four years: Evidence from a propensity score matching study.
Learning and Instruction, 47, 43-52. https://doi.org/10.1016/j.learninstruc.2016.10.001 Drabinová, A., & Martinková, P. (2018). difNLR: Generalized logistic regression models for DIF and DDF detection. The R Journal.
Submitted. Hanushek, E.
A., & Wößmann, L. (2006). Does Educational Tracking Affect Performance and Inequality? Differences-in-Differences Evidence across Countries.
Economic Journal, 116(510), C63-C76. Hautamäki, J., Arinen, P., Eronen, S., Hautamäki, A., Kupianien, S., Lindblom, B., Niemivirta,M., Pakaslahti, L., Rantanen, P. and Scheinin, P. (2002).
Assessing learning-to-learn: A framework. Helsinki: Centre for Educational Assessment, Helsinki University/ National Board of Education. https://www.oph.fi/download/47716_learning.pdf Hattie, J.
A. C. (2002).
Classroom composition and peer effects. International Journal of Educational Research, 37(5), 449-481.
DOI: 10.1016/S0883-0355(03)00015-6 Ho, D. E., Imai, K., King, G., & Stuart, E.
A. (2011). MatchIt: Nonparametric Preprocessing for Parametric Causal Inference.
Journal of Statistical Software, 42(8), 1-28. URL http://www.jstatsoft.org/v42/i08/ Magis, D., Beland, S., Tuerlinckx, F., & De Boeck, P. (2010).
A general framework and an R package for the detection of dichotomous differential item functioning. Behavior Research Methods, 42, 847-862.
Martinková, P., Drabinová. A., Liaw, Y.-L., Sanders, E.
A., McFarland, J. L., & Price, R.
M. (2017). Checking equity: Why DIF analysis should be a routine part of developing conceptual assessments.
CBE Life Sciences Education, 16(2) rm2. Martinková, P., & Drabinová, A. (2018).
ShinyItemAnalysis for Teaching Psychometrics and to Enforce Routine Analysis of Educational Tests. The R Journal.
Accepted. R Core Team (2018).
R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria.
URL https://www.R-project.org/. Vainikainen, M.-P. (2014).
Finnish primary school pupils' performance in learning to learn assessments: A longitudinal perspective on educational equity. Helsinki: Unigrafia.
University of Helsinki, Department of Education Research Reports 360.