Tackling the Incomparability of Assessments of Pupils' ICT Knowledge and Skills by Using Anchoring Vignettes
Abstract
In educational research very frequent use is made of (self )evaluating questions intended to measure values, subjective norms and (non)cognitive skills, but there is a great deal of evidence that respondents use the categories of the evaluation scales in diff erent ways. Th e method of anchoring vignettes makes it possible to correct for the divergent use of the scale in such a way as to ensure comparability of self-evaluations.
In our article we use the parametric model of anchoring vignettes to identify the heterogeneity of the use of a scale in assessment of own ICT knowledge and skills by a sample of students at middle vocational schools focused on ICT (ICT students, N = 228) and students specialising in education and economics (non-ICT students, N = 147), and subsequently by girls and boys. We set the results of our research in the context of the ICILS 2013 survey.
Direct comparison of self-evaluations led to an unexpected conclusion - ICT students have slightly worse ICT knowledge and skills than non-ICT students. ICT students, however, used higher standards in self-evaluation, assessing their knowledge and skills across the whole range of the assessment scale as lower in comparison with the students without a focus on ICT.
After correction for the diff erent use of the scale we then reach the opposite conclusion - ICT students have signifi cantly higher ICT knowledge and skills than non-ICT students. In the framework of the ICILS 2013 Survey no relationship was found between the results in the test of computer and information literacy and the evaluation of own advanced ICT knowledge and skills.
Furthermore, girls achieved better results in the test than boys, but their evaluation of their own advanced ICT knowledge and skills is lower than among boys. One reason for these conclusions may be the diff erent use of the scale, which according to our research could be successfully corrected for by the use of the method of anchoring vignettes.