Gender differences in reporting behavior across different cultures: An analysis of PISA 2015 anchoring vignette data on motivation for learning
Abstrakt
Student academic motivation is an important determinant of their school performance (e.g. Fortier et al., 1995) and one of the key concerns of researchers and policy-makers world-wide.
It is commonly investigated in international large-scale studies like PISA or TIMSS and such data can be then used to compare th e level of motivation of different groups of students (based on, e.g., country, gender, socioeconomic status, immigrant status). However, in the literature, there is a long-term concern about the comparability of data collected using self-report questionnaires containing items with rating scales due to the differences in scale usage between respondents (He & van de Vijver, 2016).
It might be the case that two students with the same actual level of motivation assess their motivation on a scale differently (one only as medium while the other as strong). Such differences might distort the data and lead researchers to incorrect conclusions.
Recent literature has brought attention to some paradoxical findings in international large-scale assessments that might be attributed to the heterogeneity in student reporting behavior (e.g. Kyllonen & Bertling, 2013; Vonkova, Zamarro, & Hitt, 2018).
He and van de Vijver (2016) investigated "motivation-achievement paradox" using PISA 2012 data and found a negative country-level relationship between student motivation to learn mathematics and their mathematics achievement. In PISA 2015, similar findings occur when the country-level correlation between student instrumental motivation to learn science and science performance is r = -0,58 (OECD, 2016).
Such findings are contrary to the expected relationship between the two variables and might be the result of differential scale usage by students in different countries. Currently, innovative methodological approaches that would help solve this problem are being sought.
One of such approaches is the anchoring vignette method (King et al., 2004). The basic principle of the method is that the respondents are asked to evaluate not only themselves but also so called anchoring vignettes, which are short stories describing a hypothetical person in the domain of interest.
Given that all respondents evaluate the same vignettes, the differences in their evaluation can be interpreted as the differences in their use of scale (Vonkova, Bendl, & Papajoanu, 2017). There has been an increasing number of applications of the method in educational research, and anchoring vignettes were also included in PISA 2012 (teacher's classroom management and teacher's support behaviors) and PISA 2015 (motivation to learn) student questionnaires.
The secondary analysis of PISA 2012 anchoring vignette data by Vonkova et al. (2018) strongly supports the use of the anchoring vignette method for the identification of the differences in scale usage among different countries and enhancing international comparability of student self-reports. Previous research using anchoring vignettes has also shown that student reporting behavior might differ by gender (in the case of life satisfaction see Vonkova, 2019, and school behavior see Vonkova, Bendl, & Papajoanu, 2017).
The main aim of our paper was to investigate (a) gender differences in reporting behavior across different countries participating in PISA 2015 study and (b) how the adjustment for these differences changes the scores on achievement motivation for the two genders. In this paper, we use PISA 2015 data from 57 countries (N = 411,367) on student evaluations of achievement motivation questions (I want top in most or all of my courses; I want to be able to select from among the best opportunities available when I graduate; I want to be the best, whatever I do; I see myself as an ambitious person; I want to be one of the best students in my class.) and three anchoring vignettes related to their motivation to learn (for example, the medium level motivation vignette reads as follows: mostly remains interested in the tasks she starts and sometimes does more than what is expected from her. is motivated.).
The vignettes and the self-report questions were rated on a four-point scale from 1 = strongly agree to 4 = strongly disagree. First, we found that on average, boys seem to be more strict (have higher evaluation standards) when assessing their motivation than girls.
Only in one country (Mexico) girls were significantly more strict when assessing their motivation than boys. Second, we compared the country-average levels of achievement motivation for boys and girls before the adjustment using the anchoring vignette method and after the adjustment for differences in scale usage.
Before the adjustment, boys reported significantly higher levels of motivation in 21 countries, significantly lower level of motivation in 24 countries and in 12 countries the difference was nonsignificant. These results are quite similar to the official PISA 2015 results for instrumental motivation to learn science, where the instrumental motivation index was higher for boys in 21 countries, and in 21 countries the index was higher for girls (OECD, 2016).
After the adjustment, the reported achievement motivation was higher for boys in 23 countries, but higher for girls only in 14 countries. For the remaining 20 countries the difference was nonsignificant.
The geographical analysis of the adjusted achievement motivation scores reveals patterns across different world regions. In Western Europe, we find a consistent pattern of higher achievement motivation of boys across all countries.
Similar results were obtained for South America with the exception of Brazil, where the difference between genders was nonsignificant. Southern Europe divided into two groups of countries.
In Spain, Portugal, and Italy boys had higher level of achievement motivation, while in Croatia, Montenegro, and Slovenia the difference between genders was nonsignificant. Nonsignificant differences were found also in Eastern European countries like Poland, Slovakia and Russia.
Boys reported lower level of motivation in Oceania and North America (with the exception of Massachusetts with nonsignificant difference between genders). These results concerning achievement motivation levels are mainly driven by answers to questions focused on school unrelated domains (I want t o be the best, whatever I do; I see myself as an ambitious person.) where boys have much higher level of achievement motivation both before and after the adjustment.