Background

The mental process of metacognition is a growing subject of neuro-psychological research, with particular relevance for the processes of teaching and learning, and thus for the education system [1]. Metacognition is defined as awareness and cognition about one’s own cognitive processes [2]. Individuals’ perceptions of their internal mental states, as well as their self and non-self attributions, are determined by a set of affective and cognitive skills, broadly described as meta-cognitive abilities [3]. Metacognitive problems are associated with impairments to the affected person’s social functioning, which in turn decrease their quality of life as well as their ability to respond to treatment [3]. Metacognitive impairments are associated with affective disorders such as depression, stress, and anxiety [3,4,5]. However, all of these affective states are commonly reported to occur among university students across the world [6, 7]. Furthermore, substance use, such as alcohol consumption is generally associated with metacognitive dysfunctions, and is a prevalent activity among university students in many parts of the world [8]. It was recently found that the prevalence of alcohol consumption and chewing of khat, an indigenous psychoactive substance, was, respectively, 32.3 and 27.9% among Ethiopian university students [8]. These relationships among metacognitive dysfunctions, affective disorders, and substance use that are prevalent in student populations highlight the need for a tool to screen for dysfunctions in metacognition and its aspects among university students.

Meta-memory and meta-concentration are two very important dimensions of metacognition [9, 10]. Meta-memory and meta-concentration are associated with success in everyday functioning. Furthermore, there is an interaction effect between these two metacognitive aspects that is essential for success in daily routine activities [10]. Those with meta-cognitive and meta-memory deficits develop a protectionist approach to avoid challenging situations, thus affecting their capacity to deal with similar situations in the future, and thus having broadly detrimental effects for dealing with life problems and adjustment [9, 11]. There is a reciprocal relationship between meta-memory and other metacognitive characteristics such as vocabulary development and comprehension [12]. Meta-cognitive instructions have intermediate and delayed effects, which can manifest in improved mathematical achievement and improved cognitive regulation among students [13]. Various studies have suggested that knowledge about meta-memory can be acquired and may directly benefit the learning process in students [14]. Metacognitive abilities related to concentration i.e., meta-concentration, is one of the most important non-intellective psychological factor which can influence students’ performance, as indicated by grade point average [15]. At the present time, there is no questionnaire designed to measure these metacognitive aspects, either separately or in terms of their interactive effects, in student populations. It was thus felt that a brief, easily administered, and valid questionnaire would be of use to campus counselors, psychologists, and others. It was also felt that such a tool could help in the routine screening of the students.

We therefore investigated the literature on this subject for useful examples of instruments that could be adapted for use with students. A number of excellent psychometric instruments currently exist for diagnosing meta-memory and meta-concentration. These include commonly used questionnaires for metacognition such as the Metamemory in Adulthood (MIA) scale [16], which has 108 items, the Metacognition Questionnaire (MCQ), which has 65 items [17], and the Metacognition Questionnaire-30 (MCQ-30), which has 30 items [17]. These instruments, however, are primarily designed for use in medical or psychiatric settings, and while they tend to be exhaustively comprehensive, they can be cumbersome and time-consuming to administer. An exception to this generalization is a brief metacognition questionnaire,which was recently developed for use at the Charité - University Medicine Berlin [10]. The present investigators reviewed this questionnaire and used it as a guide for developing the questionnaire that is reported on here, although it has been modified to make it more appropriate for students. In this study, we present the psychometric properties of this adapted version of a brief meta-memory and meta-concentration questionnaire, which has been designed to suit the daily activities of university student populations.

Methods

The study presents findings of data taken (Fig. 1) from a cross-sectional study using simple random sampling method regarding psychological health and associated factors among university students carried out at the Mizan campus of the Mizan-Tepi University (MTU), Mizan-Aman, Bench Maji Zone, South Nation Nationalities Peoples Region, Ethiopia.

Fig. 1
figure 1

Schematic of study sample

Full size image

Participants

Three hundred and eighty-three university students with an age range of 18–35 years and a body mass index of 21.2 ± 3.4 kg/m2 completed this study. Students with self-reported mental illness difficulties, such as a previous diagnosis of depression or psychosis that might have compromised the data quality were excluded. Similarly, those under the age of 18 years were not included because in such cases consent would have to have been obtained from their parents as well, a difficult requirement to fulfill inasmuch as many students were from remote regions of the country.

Procedures

The Institutional Ethics Committee, College of Medicine and Health Sciences, Mizan-Tepi University approved the research. Guidelines for Good Clinical Practice and the norms of the 2002 Declaration of Helsinki (DoH) were followed. Informed written consent was provided by the participants after the objective and procedures of the study were explained to them. The Mizan meta-memory and meta-concentration scale for students (MMSS), a semi-structured socio-demographics questionnaire, plus the Epworth sleepiness scale (ESS) were employed. The questionnaire packages were administered in English because participating students belonged to different linguistic groups and had differing levels of proficiency for reading Amharic. Moreover, the study participants were students of a university in Ethiopia, where the medium of instruction is English. The instruments were administered to the participants at the university premises by those members of the team of investigators who were also part of the MTU faculty.

The Mizan meta-memory and meta-concentration scale for students (MMSS)

Background and questionnaire conceptualization

As a first step for developing the scale, a panel of experts was brought together to discuss the objective of constructing a new tool for assessing meta-memory and meta-concentration in the target audience of university students. Panel members, who were drawn from the fields of psychometrics, physiology, medicine, statistics, and languages, were asked to develop scale items according to several criteria. Among these criteria priority consideration was given to the scale’s potential usability in survey research, response rate maximization, conciseness, and appropriateness as a preliminary screening tool. Following a detailed search of the literature which sought to gather previous experience regarding scale readability as well as comprehensibility, twelve items were generated. Some of the items were adapted from a metacognition assessment instrument developed by Klusmann and colleagues at the Department of Psychiatry, Charité - University Medicine Berlin [10]. The items measuring meta-memory were based on the questionnaire of meta-memory in adulthood developed by Dixon and colleagues [16]. The items measuring meta-concentration were based on the EURO-D, which was developed by Prince and colleagues [18]. The items of the original instrument were adapted to suit the metacognitive functions associated with the daily activities of students. None of items were reverse scored in our preliminary questionnaire.

Format and content validity

The panel of experts assessed and revised these items for relevance, comprehension and clarity. It was agreed to delete one of the items, ‘I am good at reasoning, planning activities, or solving problems’, after discussions because experts did not find it relevant to meta-memory and meta-concentration.

Field testing

An 11-item scale was finally developed and employed in an initial field test. This testing led to a decision to delete two items due to their significantly adverse effect on the overall internal consistency as determined by the Cronbach’s alpha test. These items were, ‘I have no issues of memory losses’ and ‘I have no difficulties related to concentration’.

Final tool: MMSS

The preliminary testing of the MMSS produced a brief questionnaire with nine items that assess two aspects of metacognition, i.e., meta-memory (five items) and meta-concentration (four items). The MMSS used in this study is shown in Additional file 1: Appendix I. The items are scored in the range of 1–5, where, ‘1’ stands for ‘strongly disagree’ and ‘5’ denotes ‘strongly agree’. Individual scores of the 9-items of the MMSS are linearly added to get the global score of the MMSS in the range of 9 and 45, where higher scores imply good meta-cognitive ability in areas of meta-memory and meta-concentration. Individual scores of the five items of the meta-memory subscale are linearly added to obtain the total score for this dimension. Similarly, scores for the four items of the meta-concentration subscale are added to get total score for this dimension.

Epworth sleepiness scale

The ESS is an eight item questionnaire which is used to assess daytime sleepiness [19]. These 8-items have a four-point scale, where, ‘0’ indicates ‘would never nod off’, while, ‘3’ indicates a high chance of nodding off in eight different situations encountered in daily lives [19]. The scores of individual item scores are added to get the ESS total score in the range of 0 to 24. Increasing levels of daytime sleepiness are indicated by higher ESS scores [19].

Socio-demographics questionnaire

A semi-structured socio-demographics questionnaire with nine items, one open ended and eight close ended, were used. Information concerning the respondent’s age, gender, ethnicity, alcohol use, khat use, smoking, use of tea/coffee, use of other beverages such as soft drinks and other fermented/non-fermented non-alcoholic indigenous drinks and presence of chronic conditions were collected. Height and weight were taken for assessing body mass index.

Statistical analysis

Data analysis was performed by SPSS version 23.0, an add-on module called AMOS, and two plugins [20, 21]. Participants’ characteristics were examined using the mean (±SD), frequency, and percentage. Item analysis was performed by mean (±SD), skewness, kurtosis, percentage, Spearman’s item-Factor correlations, and the Cronbach’s alpha (if the item were deleted). The internal consistency of the responses was assessed by the application of the Cronbach’s alpha and the McDonald’s Omega test. Nunnally and Bernstein have suggested that during the initial stage of research, as in the case of questionnaire development, a Cronbach’s alpha of 0.70 is sufficient. However, the experimental research where emphasis is on quantitative aspect of correlation as well as the differences in mean, a Cronbach’s alpha of 0.80 may be desirable [22]. The internal homogeneity and divergent construct validity were evaluated by the Spearman’s correlation coefficient test.

Three multivariate outliers were identified, and hence deleted, for factor analysis following application of Mahalanobis distance testing (criterion of a = .001 with 9df, the critical χ2 = 33.72) (Fig. 1) [23]. Six of the MMSS items were skewed (Z score of Skewness≥ ± 3.29) (Table 1). All the items were retained without transformation inasmuch as a related instrument was found to be valid in German and Portuguese samples [10, 24].

Table 1 Descriptive statistics of the Mizan meta-memory and meta-concentration scale for students (MMSS) in university students
Full size table

In view of the fact that six item scores were skewed a confirmatory factor analysis (CFA) using maximum likelihood extraction with bootstrapping was carried out. Modification indices (co-varying error terms) were employed to increase the fit during confirmatory factor analysis (CFA). The standardized loadings of the MMSS item scores on the respective factors were estimated. CFA was used to screen two 2-Factor models; model-A: a 2-Factor model based on theoretical considerations [10], and model-B: a 2-Factor model with incorporation of modification indices (co-varying error terms) (Table 2, Fig. 2). Multiple fit indices from different categories were employed according to recommended norms [23, 25, 26]. Analyses based on discrepancy functions, such as χ2, χ2/df and standardized root mean square residual (SRMR), absolute fit index, the goodness of fit index (GFI), tests comparing target model with the null model (such as the comparative fit index [CFI]), non-centrality indices (such as the root mean square error of approximation [RMSEA]), and PClose were employed [23, 27]. The findings for various tests, e.g., RMSEA (≤ .08), RMR (≤ 0.05) and χ2/df (≤3) indicated an acceptable fit [28]. For CFI and GFI a value greater than 0.95 implied an excellent fit [28]. A non-zero value of the PClose also indicated an acceptable fit [28]. Tests for evaluation of configural, metric/weak, scalar/strong and strict measurement invariance for the model validated by CFA were performed.

Table 2 Discriminant or divergent validity: Correlation of the Mizan meta-cognition scale for students (MMSS) scores with Epworth sleepiness scale (ESS) scores in university students
Full size table
Fig. 2
figure 2

Confirmatory factor analysis models of the Mizan meta-memory and meta-concentration scale for students (MMSS) in university students. A: 2-Factor, B: 2-Factor model with incorporation of modification indices (correlated error terms). BMMS: Brief Meta-memory sub-scale; BMCS: Brief Meta-concentration sub-scale, bmms_1 to bmms_5: items of BMMS, bmcs_1 to bmcs_4: items of BMCS. All coefficients are standardized. Ovals latent variables, rectangles measured variables, circles error terms, single-headed arrows between ovals and rectangles factor loadings, single-headed arrows between circles and rectangles error terms. Amos does not display standardized values of uniqueness on the models; therefore models were manually edited to put numerical values taken from the Amos text output (Estimates→Scalars→Variances)

Full size image

Results

Participants’ characteristics

Participants’ characteristics are shown in Table 3. The mean age was 21.2 ± 3.4 years, and students with normal BMI’s formed the largest subgroup, making up 66.1% of the sample (Table 3). Amhara and Oromo ethnicities together comprised the majority (59%) of the study population (Table 3). The self-reported prevalence of the use of alcohol, Khat and cigarettes were 10.2, 9.9 and 5.7%, respectively (Table 3). Nearly 1/10th, i.e., 11.5% of the sample, reported having chronic medical conditions, including AIDS, hepatitis-A, hepatitis-B, hypertension, diabetes mellitus I/II, and tuberculosis (Table 3). It was observed that a high mean MMSS global score of 30.71 ± 7.29 occurred in the study population (Table 3).

Table 3 Participant characteristics
Full size table

Preliminary item analysis

The descriptive analysis of the MMSS scores is shown in Table 1. There were 0–1% missing values for the MMSS item scores in the final study sample. Little’s test [χ2 = 65.98 (df = 62), p < 0.34] indicated that the missing values for MMSS scores were completely random. Missing values were dealt with by adding in the expected maximization because it is a method of choice irrespective of sample size, the proportion of data missing, and distribution characteristics [29]. None of the MMSS item scores showed a floor effect; the lowest score occurred in less than 15% of the sample [30, 31]. However, five items, i.e., BMMS-2, BMMS-3, BMMS-4, BMCS-3,and BMCS-4 demonstrated a ceiling effect, i.e., the highest scores were achieved by more than 15% of the respondents [30, 31]. The MMSS global score did not demonstrate any significant problems in terms of ceiling/floor effects, with 0.5% reporting the lowest score of 9 and 0.8% reporting the highest score of 45. The meta-memory score did not demonstrate any significant problems in terms of ceiling/floor effects, with 1.0% reporting the lowest score of 5 and 1.8% reporting the highest score of 25. The meta-concentration score did not demonstrate any significant problems in terms of ceiling/floor effects, with 0.8% reporting the lowest score of 4 and 4.7% reporting the highest score of 20.

Factor analysis

Measures assessing adequacy, suitability and factorability of the MMSS scores

The diagonal elements of the anti-image correlation matrix of the MMSS item scores were either 0.89 or above, satisfying the condition for factor analysis (Table 4) [32]. The MMSS item scores had an excellent degree of shared variance, as indicated by a Kaiser-Meyer-Olkin Test of sampling adequacy of 0.91 (Table 4) [32]. The MMSS item scores had linear combinations necessary for factor analysis, as suggested by a significant Bartlett’s test of sphericity (Table 4) [32]. There was neither an issue of singularity nor of the multicollinearity as required for factor analysis in the MMSS item score, because the determinant of the correlation matrix was greater than 0.00001 and less than 1 (Table 4) [32]. A threshold for variance was derived from the common factors as determined by a range of 0.34 to 0.65 for the communality (Table 4), therefore all the MMSS items were retained for the factor analysis [33]. None of the inter-item correlations were less than 0.3 (r = 0.37–0.71, p < 0.01), therefore ideal conditions were found for the factorability of the MMSS item score correlation matrix [34] (Table 5).

Table 4 Sample size adequacy measures of the Mizan meta-cognition scale for students (MMSS) in university students
Full size table
Table 5 Inter-item Correlation matrix of the Mizan meta-memory and meta-concentration scale for students (MMSS) in university students
Full size table

Confirmatory factor analysis (CFA)

Table 6 shows the goodness of fit statistics of the models screened in the CFA of the MMSS scores in the university students. Both models had either an excellent or an acceptable fit, i.e., CFI and GFI > .95, SRMR and RMSEA<.08 and χ2/df < 3 and PClose> 0 [28].

Table 6 Fit statistics of the Mizan meta-memory and meta-concentration scale for students (MMSS) models in university students
Full size table

Measurement invariance of model-B among gender groups

The configural invariance of Model-B was excellent as indicated by values of the fit indices (χ2/df < 2, CFI > .95, RMSEA (CI) < .05, when groups were estimated without constraints (Table 7). Chi-square testing did not reveal significant differences ([Δχ2(df) = 10.988 (7), p = .139] and ΔCFI <.01) between the model constrained for loadings and the fully unconstrained model, thus supporting metric or weak invariance of the Model-B, across gender groups (Table 7) [35]. Strong or scalar invariance of model-B was indicated by a finding of non-significance following chi-square testing ([Δχ2(df) = 14.234 (9), p = .114] and ΔCFI <.01) between models constrained for loadings and models constrained for intercepts (Table 7) [35]. Models constrained for residuals and models constrained for intercepts showed significant chi-square differences ([Δχ2(df) = 53.024 (15), p < .001] but ΔCFI <.01) (Table 7) [35].

Table 7 Measurement invariance of the 2-Factor model among gender groups of the Mizan meta-memory and meta-concentration scale for students (MMSS) in university students
Full size table

Internal consistency and homogeneity

The Cronbach’s alpha for the meta-memory and meta-concentration subscales were 0.84 and 0.80, respectively (Table 8). The McDonald’s Omega for the meta-memory and meta-concentration subscales were 0.84 and 0.82, respectively (Table 8). Item-Factor score correlations for the meta-memory subscale ranged between r = 0.75 (p < .01) and r = 0.80 (p < .01) (Table 1). Item-Factor score correlations for meta-concentration subscale ranged between r = 0.75 (p < .01) and r = 0.82 (p < .01) (Table 1). Inter-item correlations ranged between r = 0.32 (p < .01) and r = 0.68 (p < .01) (Table 5).

Table 8 Internal consistency: Cronbach’s alpha and McDonald’s Omega of the 2-Factor model of the Mizan meta-memory and meta-concentration scale for students (MMSS) in Ethiopian university students
Full size table

Divergent construct validity

There was no significant correlation between the ESS score and MMSS scores.

Discussion

This is the first study to carry out a psychometric validation of an instrument for measuring two important aspects of meta-cognition i.e., meta-memory and meta-concentration, in a student population. The study found sufficient psychometric validation of the MMSS to support the conclusion that this instrument measures what it is intended to measure. This was evidenced by the absence of findings of major issues in terms of ceiling/floor effect, favorable item discrimination, factorial validity and measurement invariance across gender groups, internal consistency, and divergent validity.

Preliminary item analysis

There was some concern about the ceiling effect in five item scores of the MMSS; the presence of this phenomenon could possibly affect the responsiveness and discriminative validity of this instrument for the highest score of these items [30]. The MMSS items are scored in such a way that normal behavior, i.e., of metacognitive functioning, is indicated by higher scores, therefore, the presence of the ceiling effect is possibly explained by the non-clinical nature of the study population. Indeed, a scale for assessing affective disorders, i.e., the Hospital Anxiety and Depression Scale (HADS) was reported to show a floor effect when validated in a normal elderly Swedish population [36]. This situation is similar to the one we encountered for the MMSS, because in the case of the HADS, the lower score denotes normal behavior, while for the MMSS it is the higher score [36]. However, the absence of the ceiling/floor effect in the MMSS global score and factor scores, as well as the absence of the floor effect for all the MMSS item scores, are further evidence of its applicability in student populations [36]. Additionally, findings which were similar to our own with respect to the ceiling/floor effect were confirmed for the brief Meta-Cognition Questionnaire, of which the MMSS is an adapted version, thus providing concurrent evidence for the presently studied instrument’s overall validity [10, 24]. The Cronbach’s alpha if item deleted (all above 0.72) and item-Factor correlations (all above 0.75) indicate that the all items scores of the MMSS had favorable ability to discriminate between low and high scorers [37].

Factor analysis

Though it is desirable to perform both exploratory factor analysis (EFA) and CFA for establishing factorial validity, it is also an acceptable practice to present findings from CFA for constructs based on theoretical considerations [10, 38]. Therefore, we employed CFA along with measurement invariance analysis across gender groups to evaluate the validity of the 2-Factor model of the MMSS.

Measures assessing adequacy, suitability and factorability of the MMSS scores

Factor analysis was employed to investigate the scale’s dimensionality because the MMSS scores satisfied the conditions of sample adequacy, sample suitability, and factorability. Evidence for this conclusion came from findings such as the diagonal elements of the correlation anti-image matrix, Bartlett’s test of Sphericity, determinant, Kaiser-Meyer-Olkin Test of sampling adequacy (KMO), communality and inter-item correlations, all of which were within normal limits [32].

Confirmatory factor analysis (CFA)

CFA was employed to establish the dimensionality conditions, though the instrument was expected to produce a 2-Factor model based on theoretical considerations [10]. Both models, i.e., the model-A, a2-Factor model and model-B, a 2-Factor model with incorporation of modification indices (correlated error terms) performed very similarly with excellent to acceptable values for the fit indices [28]. However, model-B was favored because of the higher value of the PClose and lower value of χ2/df. Furthermore, the very good to excellent level of correlations between the MMSS item scores and its factors for the model-B favor its validity [39].

Measurement invariance of model-B among gender groups

Gender specific differences in metacognitive abilities are common in adolescents [40]. Moreover, gender dependent relationships between metacognitive dysfunctions and affective conditions such as anxiety and depression are also found among adults [41]. Given this background, it was imperative to assess that the MMSS construct comparability is not confounded by gender. Therefore, measurement invariance of the MMSS across gender groups was evaluated in the study population. The validity of the model-B, a 2-Factor model with incorporation of error terms was further evidenced by the establishment of its measurement invariance, i.e., configural, metric, scalar and strict invariance among two gender groups. For metric and scalar invariance, conditions for both, i.e., non-significant differences were found following chi-square testing and ΔCFI<.01 were met [35]. Even though the chi-square test of difference was significant the finding that ΔCFI<.01 still supports the strict invariance condition [35]. This is because ΔCFI is a more robust measure than chi-square test of difference [35].

Internal consistency and homogeneity

According to the “rule of thumb” of [42] George and Mallery (2003), the MMSS and its subscale internal consistency were good, as implied by the Cronbach’s alpha and Mcdonald’s omega [42]. Furthermore, according to the criteria of Nunnaly and Bernstein, the Cronbach’s alpha of the factors of the MMSS suggest that it may have a potentially viable application in experimental research as well [22]. The Cronbach’s alpha of the MMSS was higher than that reported for the related instrument in a German elderly population (0.61–0.67) [10]. The internal homogeneity of the MMSS was supported by the strong item-total correlations in this student population. Here again, the item-total correlations were higher for the MMSS than that of the brief meta-cognition questionnaire in the German population (r = 0.26–0.52) [10]. Inter-item correlations indicated a moderate to a strong relationship, thus reinforcing the internal homogeneity of the MMSS in the study population.

Divergent construct validity

Daytime sleepiness is an important defining feature of insomnia [43]. Furthermore, metacognition is associated with mental activity in primary insomnia [39, 40]. Therefore, ESS, which is a measure of sleepiness, was employed to assess the divergent validity of the MMSS. No correlation between the MMSS scores and the self-reported measure of daytime sleepiness support the divergent construct validity of the scale in the study population. This is because even though sleepiness and sleep are associated with meta-cognition in some populations but these represent non-overlapping constructs [44, 45]. In summary, the present findings of an absence of ceiling/floor effect for the MMSS global and factor scores, sufficient item discrimination, factorial validity, measurement invariance across gender groups for the factor structure of the MMSS, good internal consistency, strong internal homogeneity, and sufficient divergent validity favored psychometric validation of the MMSS in university students.

Some of the limitations of the study were that assessments of test-retest reliability, convergent validity, and concurrent validity were not carried out. The sample had a biased gender ratio. Therefore, the generalizations are more likely to be applicable for male students, who outnumbered females in the present study. Even though simple random sampling was used, fewer females completed the study, thereby causing the gender representation to be unbalanced. Future efforts to investigate the psychometric properties of the MMSS should accordingly anticipate and plan for a higher drop-out rate among female students, which could occur at any time from the stage of enrollment to the completion of the study. The scale was designed to assess to two important dimensions of the metacognition, i.e., meta-memory and meta-concentration. Future work should build on the current findings to incorporate brief subscales for other dimensions of metacognition to get a comprehensive yet brief tool to assess this function in students.

Conclusion

Despite these qualifications, the findings of the present study are generally supportive of the value and applicability of this instrument. The MMSS, which is the first measure of meta-memory and meta-concentration to be evaluated in a sample of university students, thus has relevance for use in student populations. This conclusion is supported by psychometric measures of its ceiling/floor effect, internal consistency, internal homogeneity, divergent validity, factorial validity and measurement invariance of the validated factor structure across gender groups.