Keywords

1 Introduction

This paper presents pilot study findings of a research project about the application of anchoring vignettes in the analysis of Czech secondary school students’ self-assessment of ICT skills. As explained by King et al. [1], anchoring vignettes is a method to identify response scale differences in survey questions and to adjust the self-assessments caused by response scale differences. For example, when students are asked to self-assess their ICT skills in a survey question on a scale of 1 to 7 (1 being the lowest and 7 the highest), students might either overvalue or undervalue their ability in the self-assessment. Hence, anchoring vignettes, which are brief texts describing hypothetical people who illustrate a certain level of the trait of interest (e.g. ICT skills), enable researchers to identify response scale differences in self-assessments and thus, to adjust them.

Since the early 2000s, the anchoring vignette method has been implemented in a variety of areas of research, such as political efficacy [1], work disability [2], job satisfaction [3], health [4,5,6], health system performance [7], life satisfaction [8], and satisfaction with social contacts [9]. However, there are only a few examples of the application of this method for educational research. Regarding one of these, Buckley and Scheider [10] implemented anchoring vignettes when investigating charter schools in the United States of America (USA) and parents’ satisfaction with different types of schools. Vonkova and Hrabak’s study [11] focused on Czech upper secondary school students’ self-assessment of ICT knowledge and skills through the anchoring vignette method. Moreover, Vonkova et al. [12] investigated Czech lower-level secondary school students’ self-assessment of dishonest behaviour in school by using the anchoring vignette method. Von Davier et al. [13] examined the effects of vignette scoring on reliability and validity of student self-assessment, according to the Programme for International Student Assessment (PISA) 2012 dataset. He et al. [14] conducted a study on effects of this method on comparability and the predictive validity of student self-assessment in 64 countries based on data from PISA 2012. Vonkova et al. [15, p. 3] looked into cross-country heterogeneity in students’ self-assessment of their teacher’s classroom management also based on the PISA 2012 dataset and they found that the anchoring vignette method was potentially a useful tool to enhance the comparability of the self-reported measures in education. To contribute to research in the field of education, we have conducted a study aimed at designing an enhanced data collection method based on anchoring vignettes to explore different realms in educational research, such as digital literacy.

2 Literature Review: The Application of Anchoring Vignettes in the Self-assessment of ICT Skills

This section presents a review of existing studies that are pertinent to the application of anchoring vignettes in the self-assessment of ICT skills in the context of education. Before moving to reviewing the existing studies, it is worthwhile mentioning the European Computer Driving Licence (ECDL) foundation study [16], which highlighted how people’s self-assessment of their ICT skills can be different from their actual ability. The ECDL foundation’s study was conducted in five European countries, namely Austria, Denmark, Finland, Germany and Switzerland, to identify people’s ICT skills. Respondents (aged 15–64 years) were given a questionnaire in two parts, with the first focusing on self-assessment of ICT skills, and the second testing their real level. The findings of the study showed that in all the countries surveyed, respondents overestimated their ICT skills. Taking into account the study of the ECDL foundation [16], we suggest that anchoring vignettes could provide dependable findings from respondents’ answers with respect to self-assessment in surveys.

Regarding studies that have applied anchoring vignettes in self-assessments, Vonkova and Hrabak [11] compared the ICT knowledge and skills of two distinct groups of upper secondary school students by examining their self-assessed perspective on these, both before and after the anchoring vignette adjustment for the different usage of scale. The study findings revealed that the anchoring vignette method enabled the researchers to distinguish between the two groups of students’ differences in terms of scale usage and showed how adjusted self-assessments corresponded to the assumed level of students’ ICT knowledge and skills. Cerna [17] investigated the self-assessment of undergraduate university students’ ICT knowledge by applying the anchoring vignette method. Respondents were students from the Faculty of Education at Charles University and specialised in three different study programmes (information technology (IT), social science, and mathematics). The findings of Cerna’s study showed significant differences between respondents’ self-assessment and their assumed actual ICT knowledge. Moreover, the author found that those who study IT as their specialisation in education have a greater tendency to exaggerate the level of their ICT knowledge compared to other respondents studying different subject areas.

3 Pilot Study Methodology

3.1 Aims of the Pilot Study

Generally, a pilot study refers to “a small-scale version or trial run, done in preparation for the major study” [18, p. 467]. As described by Vogt [19], it can be considered as a ‘dress rehearsal’ to identify any possible problems before undertaking the major study. The main aim of our pilot study was to inform and design the main study methodology, specifically to test the feasibility of the data collection method, which consists of a questionnaire with a set of vignettes focused on self-assessment of the ICT knowledge and skills of young people.

In the pilot study, we aimed to test how comprehensible our questionnaire was for upper secondary school students, who are at Year 1 (age 15 years) and 4 (age 19 years). The pilot was conducted in December 2017 among (N = 166) students from four upper secondary schools in the Czech Republic. Experiences gained from the pilot study were then used in the main study, conducted in spring 2018 with a representative sample of 2,600 students from 56 secondary schools in the Czech Republic.

3.2 Procedures of Developing the Data Collection Method

“The aim of the anchoring vignette method is to clear/correct the self-assessment of respondents so that they can be comparable” [20, p. 14]. Respondents in the area (in our case ICT skills and knowledge) use a self-assessment question, as well as evaluating hypothetical people described in a short story (anchor). The presented verbal characteristics of hypothetical people in vignettes in the case of categorical assessment scales can be evaluated by different respondents in different ways. Respondents’ answers to self-assessment can be affected by the different use of scale categories. For example, respondents with a lot of ICT experience and a deep interest in ICT can use different scale categories for evaluating a given level of ICT knowledge than beginners and ordinary ICT users. Our main data collection method was a questionnaire, which consisted of a set of anchoring vignettes and fixed-choice questions to obtain background information about each respondent (age, sex, school, grade in ICT subjects, field of study, interest about ICT, participation in programming or informatics competitions, number of hours spent on a computer at school or at home, respondents’ use of ICT within different types of activities, parents’ education and parents’ use of ICT in their job or free time). Briefly, in our pilot study, we used one self-assessment question (S) and three vignettes (V1, V2, V3), with a scale of 1 to 7 (1 – the lowest level and 7 – the highest level).

In the vignettes designed for our pilot study (see Table 1), we focused on five domains of computer literacy (information and data literacy, communication and collaboration, digital content creation, safety, and problem-solving) in accordance with the concept of digital literacy, as defined in DigComp (see [21, pp. 8–9]).

Table 1. Overview of the self-assessment question and three vignettes
Full size table

In formulating the vignettes, we presented stories that would be comprehensible to students in the social sciences areas as well as the students of technical fields, including IT specialisation, in addition to being accessible to the two chosen age groups (Year 1 aged 15–16, and Year 4 aged 18–19 years). The vignettes should be understandable to all respondents also from a curricular point of view (note that in the Czech Republic, the current curriculum for ICT subjects does not cover all domains of DigComp).

The level of ICT skills and knowledge presented in vignettes is as follows: V1 describes a basic level (Filip’s knowledge and skills in ICT correspond to the knowledge of a basic school graduate in the Czech Republic). V2 describes a more advanced level (Kristin uses ICT for creative activities and some problem-solving; she behaves safely on social networks); the majority of respondents from all secondary schools should achieve this level. V3 describes highly advanced ICT skills and knowledge beyond curriculum requirements (programming additional functions for more difficult tasks, programming a website, using multi-layered security, etc., which are not included in the curriculum for general education). This vignette should mainly distinguish students with ICT specialisation. By providing a score on the self-assessment question and for all three vignettes in Table 1, students provided a personal rating and calibration points for inter-participant comparison.

3.3 Procedures of Data Collection

We gathered data in December 2017 through an on-line questionnaire from 166 students of Year 1 and 4, attending four different public upper secondary schools (see Table 2): School 1 focuses on general education (a gymnasium) and is located in Prague. School 2 focuses on humanities and is located in Beroun, whilst School 3 is a technical school specialising in IT and located in Prague, and lastly, School 4 is a technical school specialising in transport and mechanisation in Mladá Boleslav. These schools were not selected at random, so respondents do not represent a representative sample of the secondary school population. However, these schools represent different specialisations among Czech upper secondary schools. Respondents filled out the on-line questionnaire at their schools with the participation of the researcher.

Table 2. Characteristics of the pilot schools
Full size table

3.4 Procedures of Data Analysis

For data analysis, we used Microsoft Office Excel 2011 and the statistical software called GRETL [http://gretl.sourceforge.net]. Table 3 shows how all respondents (N = 166) assessed the self-assessment question and three vignettes from across scale 1 to 7 (‘1’ was the lowest and ‘7’ was the highest). The variability of the use of scale for both self-assessments and vignettes is high; respondents use (almost) the whole range of the scale categories. Concerning vignettes, it shows a high heterogeneity in reporting behaviour of respondents – the same level of ICT skills described in the vignettes is evaluated differently by different respondents. We expected the respondents would assess vignettes in the natural order V1 < V2 < V3 (the order given by a researcher). It means the expected value for V1 would be lower than the value for V2 and the value for V2 would be lower than that for V3. However, 5.4% of respondents assessed the vignettes in another way; they typically tied their assessment to two subsequent vignettes (see Table 3).

Table 3. Variability of the use of scale for self-assessment
Full size table

4 Findings from the Pilot Study

For further data analysis, we analysed each school separately and tried to distinguish the differences in scale usage among different schools. Using the correction based on the non-parametric approach, which consists of how the self-assessment S of a respondent relates to his/her vignette evaluations V1, V2 and V3 “assuming that the vignettes are naturally ordered (for example, from the lowest skills level of a hypothetical vignette person to the highest)” (see [11, p. 192]), we corrected the values for each respondent for each school (see Table 4).

Table 4. Average self-assessments of respondents in all schools before and after correction
Full size table

The findings in Table 4 show that the ranking of two schools has changed after the correction, with the most considerable change being in the ranking for School 4 – its position has decreased more, indicating its low standards for evaluating ICT skills. Table 5, and Figs. 1 and 2 show comparisons of self-assessments before and after correction for School 4 and the best performing School 3. We showed there were statistically significant differences between Schools 3 and 4, not only before the self-assessment correction, but also after the correction.

Table 5. Comparison of self-assessment of respondents from School 3 and School 4 before and after correction
Full size table
Fig. 1.
figure 1

Comparison of self-assessment of respondents from School 3 and School 4 before correction

Full size image
Fig. 2.
figure 2

Comparison of self-assessment of respondents from School 3 and School 4 after correction

Full size image

To understand better how respondents evaluated their skills, we compared schools using the background variables of respondents. Analyses of questions focused on respondents’ interests and ways of using ICT in their free time, which showed respondents do not differ too much. Generally speaking, ICT knowledge, skills and interests of respondents correspond to topics they have learned mainly at school. Regarding respondents’ grades, 69.9% of all the respondents from Schools 1–4 have obtained an average grade between 1 to 2 in ICT school subjects (1 is the best grade and 5 is the worst grade in Czech schools); only respondents from School 4 got worse grades (the average is 2.45). This corresponds to our ranking of schools based on adjusted self-assessments. Nonetheless, 14.5% of respondents reported that they had never studied such a subject before.

As already mentioned, School 3 is a technical upper secondary school specialising in ICT located in Prague and School 4 is a secondary school without any specialisation in ICT located in an industrial city. In both schools, practically the same number of respondents (about 65%) agreed with a statement “I am doing my best to have good results in ICT because I am expected to do so”. Respondents from School 3 differed in some characteristics from respondents in School 4. The ratio between university-educated mothers and fathers of respondents from School 3 was 2.2 and for School 4 was 0.7. Respondents of School 3 spent 1.8-times more time on computers at weekends compared to working days, while at School 4 only 1.3-times more. On weekends, respondents in School 3 spent an average 4.2 h/day on computers, while respondents in School 4 only 3.4 h/day. Seventy-four per cent of respondents in Schools 3 and 4 spent practically every day surfing on the Internet for fun. Comparing respondents in School 4 with those in School 3, more enjoyed creating digital music, were active on social networks and enjoyed playing computer games. Respondents in School 3 dedicated more time to web design activities, work with graphics software, publishing on YouTube or creating digital animations than respondents in School 4. Respondents in School 3 were more interested (53%) in the latest ICT news, new technologies, computer graphics, etc., than respondents in School 4 (24%). Respondents in School 3 liked learning new things in ICT (87%) much more than respondents in School 4 (48%). Respondents in School 3 (68%) were fond of creative activities using ICT much more than respondents in School 4 (36%). Ninety-two per cent of the respondents in School 3 agreed with the statement: “To do my best in the ICT lessons pays off because it can help me to get a job I want to do in the future”, while in School 4 this was only 60% of respondents. To summarise, the background characteristics of respondents are in line with our adjusted self-assessments using anchoring vignettes.

5 Conclusion

The main aim of our pilot study was to inform and design the main study methodology, specifically to test the feasibility of the data collection method, consisting of a questionnaire with a set of vignettes focused on self-assessment of ICT knowledge and skills of young people. From this pilot study, we identified some problems that shed light on the main research, conducted in spring 2018 in the Czech Republic.

The pilot study showed us that in terms of data interpretation, it was very important that researchers could visit all schools to instruct respondents what to do and how to fill out the on-line questionnaire. The researchers could understand better some contexts related to students’ ICT knowledge, skills, motivation, and approaches to ICT.

Some questions need to be adapted for the main research (questions about type of school, number of inhabitants living in a respondent’s town/village, grades from ICT subjects, arrangement and ordering vignettes on a page, questions about the family). The pilot also highlighted questions about how to organise data collection through an on-line questionnaire in school computer laboratories, how to support teacher co-operation to motivate students to answer the questionnaire responsibly and maintain classroom discipline, etc. It is necessary in the classes to ensure peace and discipline in order for the respondents to read attentively all questions, especially vignettes.

Limitations of this study are as follows. Findings obtained in the pilot cannot be generalised; sampling does not allow this. However, findings are definitely of interest since they indicate huge differences in scale usage between different types of schools – students from some schools have high/low standards when evaluating their ICT skills. However, all three vignettes V1, V2 and V3 were presented to respondents on one page of the questionnaire - this might have affected respondents’ assessment of vignettes.