Abstract
Respondents' self-reports are often employed in educational surveys (e.g. PISA, TIMSS) and are frequently used to compare different groups of respondents (based on country, socioeconomic status etc.).
However, serious concerns have been raised about the comparability of such data, which may be hindered by bias - the score differences on the indicator of a construct do not correspond to the differences in the underlying trait or ability. Such differences in reporting behavior are well-documented across cultures or different groups of respondents.
One of the potential sources of scale scores distortion is socially desirable responding (SDR), a tendency for some people to self-enhance when describing themselves. A promising approach to overcome SDR is the overclaiming technique (OCT).
The technique asks respondents to rate their familiarity with a set of items from a particular field of knowledge (e.g. astronomy, history, literature). Some of the items (usually about 20%), however, do not actually exist (foils).
By using signal detection analysis, the technique allows us to measure respondents' knowledge exaggeration (the overall tendency to report familiarity with both existent and nonexistent items) and accuracy (the ability to discriminate between existent and nonexistent items). Here we investigate the potential of the overclaiming technique to enhance the cross-country comparability of students' self-reported mathematical knowledge.
We also investigate the comparability of self-reported ICT knowledge between different groups of students within a single country. The cross-country analysis has been conducted using the questions on familiarity with mathematical concepts used in PISA 2012 student questionnaire.
The data include the observations of 275 904 students in 64 countries and economies. We show that there are significant differences in responding patterns between particular countries, however, we identify similar patterns of responding in geographically and culturally close country-regions.
We also validate the overclaiming scores using external variables like PISA math test scores, GDP and public expenditure in education. Furthermore, we investigated the potential of the overclaiming technique using the questions on familiarity with ICT concepts administered to two different groups of Czech university students (N=374) - one group studying ICT and the other studying educational sciences (non-ICT).
The technique has never been used in the area of ICT skills and knowledge before, even though the self-reports of ICT skills are widely used. Surprisingly, ICT students report being almost twice as much more familiar with non-existing ICT concepts than non-ICT students.
This could be interpreted that those who believe their knowledge in certain domain to be excellent may be more 22 prone to exaggerate (self-enhance) their knowledge. The differences in the self-reported familiarity with ICT concepts between ICT and non-ICT students are substantial both before and after the adjustment using the OCT, however, the adjusted results reflect the tendency of ICT students to exaggerate their knowledge and, to a certain degree, decrease the absolute differences between these groups.