Analyzing heterogeneity in students' reporting behavior across cultures: results from PISA 2015, anchoring vignettes data on motivation for learning
Abstract
Student motivation is an important determinant of their school success and one of the key concerns of both teachers and parents world-wide. It is also of great interest to researchers and policy-makers.
Motivation of students is commonly measured in international large-scale assessment studies like Programme for International Student Assessment (PISA) or Trends in International Mathematics and Science Study (TIMSS). Such data can then be used to compare different countries in terms of how motivated the students are.
When using questionnaires to measure student motivation, self-report items with rating scales are often employed. However, there is a long-term concern in the literature about the comparability of such data.
For example, it might be the case that two students with the same actual level of motivation assess their motivation on a scale differently (one as strong while the other only as medium). These differences in the scale usage between respondents might distort the data and lead researchers to erroneous conclusions about the actual level of motivation of respondents.
He and van de Vijver (2016) call attention to what they call "motivation-achievement paradox". The authors use PISA 2012 aggregated country-level data and document negative relationship between student motivation to learn mathematics and their mathematics achievement.
Clearly, these results are contrary to the expected relationship between these variables and might be the result of differential scale usage by students in different countries. Similar paradoxical findings in international large-scale assessments have also been documented by other authors (e.g.
Kyllonen & Bertling, 2013; Vonkova, Zamarro, & Hitt, 2018). One of the approaches that has been proposed to adjust for the differences in scale usage between respondents is the anchoring vignette method (King et al., 2004).
In social sciences the method has been applied in various domains (e.g. health, life satisfaction, political efficacy) and there is a growing number of applications in the domain of education. Using the method, the respondents are asked to evaluate not only themselves but also an anchoring vignette.
It is a short story describing a hypothetical person in the domain of interest. Given that all respondents evaluate the same vignette, the differences in their evaluation can be interpreted as the differences in their use of scale.
The anchoring vignette method was used in PISA 2012 student questionnaire in two domains - teacher's classroom management and teacher's support behaviors. The secondary analysis of the data strongly supports the use of the anchoring vignette method for enhancing international comparability of student self-reports (Vonkova et al., 2018).
In PISA 2015, a set of anchoring vignettes for student motivation was included in the student questionnaire. We use PISA 2015 data from 57 countries (N = 419,045) on student evaluations of anchoring vignettes related to their motivation to learn.
We study the heterogeneity in student's reporting behavior when evaluating the vignettes on motivation across countries. We also analyze how the heterogeneity in student reporting behavior differs for vignettes describing different levels of motivation.
PISA 2015 student questionnaire contained three vignettes representing low, medium, and high level of motivation assessed on a four point scale: strongly disagree, disagree, agree, strongly agree. The vignette representing low level of motivation reads as follows: (Name 1) gives up easily when confronted with a problem and is often not prepared for his classes. (Name 1) is motivated.
If we rank the countries according to their mean assessment of the vignette, we can see that countries like Thailand, Dominican Republic, Uruguay, Colombia, or Malaysia have low standards for evaluation (i.e. they are more likely to evaluate the vignette using the category disagree than strongly disagree compared to other countries), while countries like Germany, Denmark, Japan, Austria, or Singapore have high standards for evaluation (i.e. they are more likely to evaluate the vignette using the category strongly disagree than disagree compared to other countries). The second (medium level) and the third (high level) vignettes read as follows: (Name 2) mostly remains interested in the tasks she starts and sometimes does more than what is expected from her. (Name 2) is motivated. and (Name 3) wants to get top grades at school and continues working on tasks until everything is perfect. (Name 3) is motivated.
For these two vignettes, which describe considerably higher levels of motivation (and their mean assessments for particular countries are very similar), the results are quite different. For both vignettes, some of the countries with lowest standards are Japan and Portugal (other countries are Austria or Switzerland for medium-level vignette and Ireland or Singapore for high-level vignette) while the countries with highest standards are Brazil, Slovak Republic, Greece, Montenegro, or Cyprus.
As we can see, for example in the case of Japan or Singapore, the use of scale differs between the low-level and the high-level vignettes. Students in these two countries have rather high standard for the low-level vignette, but they have rather low standard for the high-level vignette.
Indeed, the country-level correlation between the mean ratings of the two vignettes is r = -.53. Accurate international comparison of student non-cognitive outcomes requires the adjustment of the self-reported data for the differences in reporting behavior between students.
Our results: (1) confirm that there is a significant heterogeneity in the use of scale between students from different countries; (2) emphasize the need to further examine how different groups of students (cultural, gender) use the scale in different domains (e.g. teacher's classroom management, motivation); (3) raise some methodological considerations related to the appropriate use of innovative approaches like the anchoring vignette method.