Abstract
(Self-)assessment questions are commonly used in the measurement of many educational concepts like (non-)cognitive skills, attitudes and values. Based on (self-)assessments we compare different countries, groups and individuals.
Self-assessments are also used outside of academic research. However, there is one major concern with (self-)assessments - they may depend not only on the objective situation but also on respondents' use and interpretation of the different scales in the provided questions.
For example, two students with the same actual motivation level may give different self-assessments, one may call his motivation low, the other medium. The anchoring vignette method has been introduced to adjust self-assessments so they take account of the differences in scale usage and allow them to be used to make comparisons between respondents.
We will explain the basic idea and models of the anchoring vignette method. The method has been used widely in many research domains like political efficacy, health, life satisfaction and work disability but its use in educational research is rare despite its high potential for enhancing the validity of research results.We will provide an overview of applications in the educational research.
We will mainly focus on comparison of teacher classroom management and teacher support behavior in different PISA 2012 countries (before and after correction for the differences in the scale usage using the anchoring vignette method). Next, we will show first results of comparison of motivation for learning levels in different PISA 2015 countries.
We will also address a peculiar but consistent finding in international large-scale assessment research: self-reported skills have been found to present negative correlation with achievement outcomes. For example, student's scores on mathematics self-concept scales, based on items such as "I learn mathematics quickly - "strongly agree", "agree", "disagree", "strongly disagree"", where found to present negative correlations with mathematics achievement between PISA countries.
Another example comes from the International Computer and Information Literacy Study 2013 where no consistent associations overall between advanced ICT self-efficacy, based on items such as "Use software to find and get rid of viruses - "I know how to do this", "I could work out how to do this" and "I do not think I could do this"", and CIL test scores were found. We will also show how this issue can be potentially solved using the anchoring vignette method.