Exploring How Czech Learners Report Their English as a Foreign Language Knowledge Using the Overclaiming Technique
Abstract
Contribution The importance of English in the modern world cannot be denied. English language knowledge and skills can provide students with more opportunities to enhance their education by studying abroad.
In addition to educational benefits, English skills can also create possibilities to travel and make cross-cultural connections and can provide more job opportunities. Correspondingly, there is a demand for the improvement of students' English as a foreign language (EFL) knowledge and skills across education systems in different countries.
An important source of information about learners' EFL proficiency is their self-assessment. Whether self-assessment occurs for consideration in regard to employment and study opportunities (e.g. job market applications, CV creation) or in the form of questionnaires for research purposes.
However, there might be differences in the way different groups of learners (based on gender and other characteristics) self-assess their knowledge and skills. For example, when asked to assess their proficiency in a certain domain on a scale (1) very good, (2) good, (3) poor, (4) very poor, two learners with the identical level of the proficiency might assess themselves differently - one as very good while the other as good (Vonkova & Hrabak, 2015).
Such differences in reporting behavior might have many practical consequences (e.g., for employers, academic institutions), as well as consequences for researchers, who might arrive at inaccurate conclusions about the learners' actual proficiency. Previous research has repeatedly pointed to problems with learners' self-report data comparability when making cross-country comparisons and paradoxical findings that might be attributed to the differences in learners' reporting behavior (e.g., Kyllonen & Bertling, 2013; He & van de Vijver, 2016).
Further, differences in the way learners report their knowledge, skills, and behaviors have also been identified when comparing different groups of learners within a single country. For example, in the domain of information and communication technology (ICT), Vonkova and Hrabak (2015) found that upper secondary learners studying at ICT schools reported lower levels of ICT skills (though not significantly) than students studying at business and pedagogy schools.
Such striking, unexpected findings warrant further attention to the way different groups of learners self-assess their knowledge and skills. Indeed, methodological approaches to identify and adjust for the differences in reporting behavior among different groups of learners are being investigated in the current literature.
These include, for example, the overclaiming technique (OCT; Paulhus et al., 2003), the anchoring vignette method, and response style identification. The overclaiming technique has been, for example, successfully used to enhance comparability of student self-reports in the domain of mathematics (e.g., Kyllonen & Bertling, 2013) and information and communication technology (e.g., Vonkova et al., 2021).
In this study, we elaborate on the previous literature using the OCT in the domain of language knowledge (e.g., DuBois, 2015; Ziegler et al., 2013). We explore the possible use of the overclaiming technique in the domain of English as a foreign language and focus on the identification of the differences in response patterns between different groups of lower secondary EFL learners in the Czech Republic.
In our analysis, we focus more closely on the differences between male and female learners. Specifically, our research question is: What are the differences in response patterns identified using the overclaiming technique between male and female learners when evaluating their EFL knowledge? Method The overclaiming technique (OCT) has been suggested as a promising approach for the identification of differences in response patterns between different groups of respondents and improving the comparability of their self-reported knowledge.
The basic principle of the technique is to present the respondents with a list of items and ask them to rate their familiarity with each of the items. Most of the items are existing concepts (reals) from a particular field of knowledge (e.g., geography, history).
However, some of the items (usually about 20%) do not actually exist (foils). Using signal detection analysis, the respondents' answers can be sorted into four categories (Paulhus et al., 2003): (a) hits - claims that reals are familiar; (b) false alarms - claims that foils are familiar; (c) misses - claims that reals are unfamiliar; and (d) correct rejections - claims that foils are unfamiliar.
For each respondent, we can then calculate four basic indices: proportion of hits - PH (proportion of reals claimed to be familiar), proportion of false alarms - PFA (proportion of foils claimed to be familiar), index of accuracy - IA (accuracy in distinguishing between reals and foils, calculated as IA=PH-PFA), and index of exaggeration - IE (overall tendency to claim familiarity with both reals and foils, calculated as IE=[PH+PFA]/2). To examine learners' response patterns to OCT items in the domain of EFL, we used an OCT questionnaire specifically suited for English as a foreign language learners (developed as a part of our GA ČR 20-05484S Czech Science Foundation project).
The construction of the questionnaire was based on the English Vocabulary Profile (EVP) online wordlist (Cambridge University Press, 2015), and it covers different language proficiency levels (A1 to C2) according to the The Common European Framework of Reference for Languages (CEFR). The OCT items were rated on a 4-point familiarity scale.
For each learner, we computed the four OCT indices PH, PFA, IA, and IE. In this study, we use the data from a large-scale sample of lower secondary level EFL learners from different regions and types of schools in the Czech Republic.
Expected Outcomes Our preliminary results suggest that there are notable differences in the reporting behavior between different groups of learners when responding to OCT EFL items. For example, a larger proportion of male students seem to have higher OCT exaggeration values in comparison to females.
The finding that a larger proportion of males reached maximum exaggeration value than females is especially notable. However, in terms of accuracy, a larger proportion of females has positive accuracy values (i.e., report higher familiarity with reals than foils) than males.
A previous small-scale study examining self-reports of EFL skills among Czech lower secondary students (Papajoanu et al., 2021) showed that male students reported higher levels (though not significantly) of general EFL proficiency as well as individual EFL skills (reading, writing, listening, speaking). Using the OCT in the ICT domain on a sample of Czech students (ISCED 3 level), Vonkova et al. (2021) found that males scored significantly higher on all the OCT indices.
The results of this study extend the knowledge we have about self-assessment of EFL skills among Czech lower secondary learners and the way different groups of Czech learners respond to OCT items in different knowledge domains. Also, we contribute to the literature on the use of the OCT in (foreign) language learning research.
Further examination of differences in reporting behavior among learners of foreign languages, that takes a wide range of student characteristics (e.g., their language background, type of school they study) into consideration is recommended. Also, future studies using the overclaiming technique could focus on the comparison of learners' reporting behavior in the EFL domain with their reporting behavior in other domains (e.g., history, geography) as well as other commonly taught foreign languages (e.g., German, Russian).