Practice makes proficient: teaching undergraduate students to understand published research

Abstract

Scientific knowledge, including the critical evaluation and comprehension of empirical articles, is a key skill valued by most undergraduate institutions for students within the sciences. Students often find it difficult to not only summarize empirical journal articles, but moreover to successfully grasp the quality and rigor of investigation behind the source. In this paper, we use instructional scaffolds (reading worksheets, RWs, with tutorials) to aid students in being able to comprehend, and ultimately transfer, the skills necessary in critically evaluating primary sources of research. We assess students’ learning of these skills on a multiple-choice assessment of Journal Article Comprehension (JAC). Students in experimental classes, who received instructional scaffolds, improved on the JAC post-test compared with students in control classes. This result shows that students are acquiring fundamental research skills such as understanding the components of research articles. We also showed that improvement on the JAC post-test for the experimental class extended to a written summary test. This result suggests that students in the experimental group are developing discipline-specific science process skills that allow them to apply JAC skills to a near-transfer task of writing a summary.

Appendices

Appendix 1: Method, analysis, and results details, pilot studies

Pilot study 1

The participants of the first pilot study included 78 students from an undergraduate cognitive psychology course in the experimental group who completed RWs, and 78 students from a different undergraduate cognitive psychology course in the control group who only used a textbook as their instructional materials. Students completed one form of the JAC as a pre-test at the beginning of the semester and one form as a post-test as the end of the semester with order of forms counterbalanced. After completion of the JAC pre-test, but before completion of the first RW assignment, the experimental group received a 1-h tutorial covering strategies for locating information within journal articles needed to complete a RW assignment. This tutorial used a research article that was not part of the JAC assessment. The control group did not receive a tutorial.

The first pilot study showed an unexpected effect of order, in that there was a larger increase between pre- and post-test when students had the Beilock and Carr (2005) JAC for the post-test. To account for these order effects, we included order as a covariate in a factorial ANCOVA with a repeated measures factor of time (pre-test vs. post-test) and a between-subjects factor of group (experimental vs. control). All main effects were significant as well as the interactions, as shown in the table below.

Source	SS	MS	F	p	$\eta_{\text{p}}^{2}$
Time	.46	.46	27.42	.0001	.15
Group	.61	.61	28.33	.0001	.16
Order	.24	.24	11.28	.001	.07
Time × group	.13	.13	7.85	.01	.05
Time × order	.41	.41	24.55	.0001	.14
Within cells	2.56	.02
Between cells	3.27	.02

1. df = 1, 153

To further explore the time x group interaction, paired-samples t-tests were conducted within the experimental and control groups. The experimental group showed significant improvement between their pre-test (M = .73, SD= .14) and post-test proportion scores (M = .79, SD= .14), t (77) = − 2.53, p = .01, d = .32, while the control group’s scores did not change significantly between pre-test (M = .68, SD= .14) and post-test (M = .68, SD= .16), t (77) = .24, p = .81, d = .05.

Pilot study 2

In the second pilot study there was no control group. Participants included 97 undergraduate psychology students in a cognitive psychology course who completed RWs. Similar to the experimental group of the first pilot study, all participants received a tutorial after the pretest, but unlike the first pilot study, the article used for the tutorial was either Lee et al. (2006) or Beilock and Carr (2005).

Performance on the JAC forms was not correlated at post-test (r = .15, p = .15), so each form was analyzed separately. Using a repeated measures ANOVA on the Beilock and Carr JAC, we found that all students improved from pre-test (M = .73, SD = .14) to post-test (M = .84, SD= .11) no matter which article they reviewed during the tutorial, F (1, 95) = 47.73, p = .0001, $\upeta_{\text{p}}^{2}$ = .32. There was no main effect of tutorial type, and no interaction, as shown in the table below.

Source	SS	MS	F	p	$\upeta_{\text{p}}^{2}$
Time	.65	.65	47.73	.0001	.32
Review type	.01	.01	.69	.41	.01
Time × review type	.03	.03	2.07	.15	.02
Within cells	1.29	.01
Between cells	1.77	.02

1. df = 1, 95

For the Lee et al. JAC, we found no main effect of time and no main effect of review content (see following table). There was, however, a significant interaction between time and review content, F (1, 95) = 4.06, p = .05, $\upeta_{\text{p}}^{2}$ = .04. Specifically, if students received the Lee et al. tutorial, their proportion scores did not significantly increase between pre-test (M = .72, SD= .13) and post-test (M = .75, SD= .13), t (51) = − 1.26, p = .21, d = .16. If students received the Beilock and Carr tutorial, their proportion scores decreased between pre-test (M = .71, SD = .13) and post-test (M = .68, SD = .12), although not significantly, t (44) = 1.76, p = .09, d = .23.

Source	SS	MS	F	p	$\eta_{\text{p}}^{2}$
Time	.00	.00	.002	.97	.00
Review type	.07	.07	3.73	.06	.04
Time × review type	.05	.05	4.06	.05	.05
Within cells	1.26	.01
Between cells	1.79	.02

1. df = 1, 95

Appendix 2: Beilock and Carr (2005) JAC questions and scoring rubric

• 1. Which of these is the best statement of the purpose of this research?

  • a. The authors’ purpose was to examine why some people choke under pressure but others don’t. isn’t as specific as answer C partial credit

  • b. The authors’ purpose was to explain how math anxiety negatively affects performance on math tests. although the authors discuss math anxiety in their intro, this is not the purpose of the study no credit

  • c. The authors’ purpose was to explain how individual differences in WM capacity affect susceptibility to choking under pressure in mathematical problem solving. full credit

  • d. The authors’ purpose was to examine how individuals with high WM capacity excel in testing situations. this is the purpose of many WM articles, but not this one no credit

• 2. Which of the following best describes the participants and what they did?

  • a. 93 undergraduate students completed the operation span and reading span tests and then were split into low and high working memory groups. this is true but not as good of an answer as B partial credit

  • b. 93 undergraduate students were screened for their WM capacity and then performed low- and high-demand modular arithmetic problems under low- and high-pressure conditions. full credit

  • c. 48 undergraduate students completed 93 modular arithmetic problems. this is wrong no credit

  • d. 93 undergraduate students completed a series of math problems that were adopted from the SAT. this isn’t true at all no credit

• 3. Which of the following is/are the independent variable(s)?

  • a. Level of math anxiety not measured at all no credit

  • b. Accuracy on modular arithmetic problems and RT on correct problems these are the DVs no credit

  • c. Whether the modular arithmetic problem had a large number or required a borrow operation or not this is the definition of problem demand partial credit

  • d. Problem demand (low vs. high), pressure (low vs. high), and working memory capacity (low vs. high) full credit

• 4. Which of the following is/are the dependent variable(s)?

  • a. Accuracy on modular arithmetic problems and RT on correct problems full credit

  • b. Score on the operation span and reading span tests these scores are used to create an IV no credit

  • c. Performance on the modular arithmetic problems students need to be more specific partial credit

  • d. Problem demand (low vs. high), pressure (low vs. high), and working memory capacity (low vs. high) these are the IVs no credit

• 5. Which of the following best summarizes the important results?

  • a. Low WM subjects were slower than high WM subjects to solve the modular arithmetic problems. while the LWMs were slower on high-demand problems, this is not the best summary partial credit

  • b. High WM subjects showed higher accuracy and were faster in the high-pressure condition. prediction that was not supported no credit

  • c. High WM subjects showed lower accuracy on high-demand problems in the high-pressure condition (low WMs not affected by pressure) and all subjects were slower on high-demand problems and under high-pressure. full credit

  • d. High WM subjects showed higher accuracy on high-demand problems in the high-pressure condition (low WMs not affected by pressure) and all subjects were faster on high-demand problems and under high-pressure. this is wrong, just trying to have an answer that is the same length as the correct answer no credit

• 6. Which of the following is/are valid criticism(s) of the research?

  • a. There is demographic information missing about the participants this is an ok, but not great criticism—the authors do skip a lot of demo. info. partial credit

  • b. Math test and high-pressure condition are unlike what one would experience in the real world both of these are valid criticisms full credit

  • c. Subjects’ baseline math ability was not tested. this is a nuisance variable and also no Ss would have experience with the modular arithmetic task partial credit

  • d. Only undergraduates were tested. although WM might change with age, this isn’t really a valid criticism for the current study no credit

• 7. Which of the following statements is most likely to be true, based on this research?

  • a. People only do well on math problems when they are extremely anxious. wrong no credit

  • b. People choke under pressure because they are worried about what others think about them. this could be true for the HWM group but not as good as C partial credit

  • c. People with high WM capacity are more likely than people with low WM capacity to choke under pressure in high-demand situations. full credit

  • d. People with high WM capacity outperform people with low WM capacity on math problems. this is true only under low pressure partial credit

Appendix 3: Sample coded summaries

Subject 1048, experimental group, pre-test

The most important points of study in this article are those claiming that the presence of anxiety interferes with a person’s ability to think as diligently as it normally would under normal circumstances. The article is saying that although a person may have the ability to solve math problems and perform well, if the working memory is interrupted by the negative emotion of anxiety, its ability to work at its best is decreased. When anxiety poses a threat to the working memory, it cannot put all its efforts into thinking and solving a problem, it needs to also put effort into figuring out a way to deal with the anxious feeling that is clouding the person’s mind. The article says that for people high in working memory, the presence of anxiety poses a threat because so much of their thought processes is used for solving the specific problem and not used for coping with anxious nerves in order to focus on the specific problem.

Subject 1048, experimental group, post-test

The goal of this study was to see if individual differences in WMC may have something to do with a person choking under pressure. 93 undergraduate students were divided into two groups, LWM and HWM, each person was tested on MA problems and used a computer to do so. Each person was given a partner and was told that they could win an award of $5, but that their partner had already completed the task and that it was a team effort. Their scores were dependent on time and accuracy. Results revealed that HWM was not affected by pressure on low-demand problems, however HWM’s performance on high-demand started to decline under pressure. In addition, all groups were slower in the low-pressure test than in the high-pressure test. The implications from this study are that HWM does not have an advantage over LWM during high-pressure demand situations.

Subject 5002, control group, pre-test

The research study performed by Beilock and Carr was about whether or not pressure and anxiety during a situation like answering arithmetic problems would affect High Working Memory (HWM) individuals and Low Working Memory (LWM) individuals. It was hypothesized that participants with low working memory are more susceptible to crack under pressure than participants with high working memory due to limited capacity to obtain information and figure out problem solutions. It was also hypothesized that participants with HWM are more susceptible to failure under pressure while answering arithmetic problems than LWMs because the pressure of the situation may deny them the resources/working memory that they are used to relying on while not in an anxiety provoked situation. The important parts to this study include the findings and results on how participants with HWM did in fact perform worse under pressure during the tests than they would without having pressure. On the other hand, participants with LWM performed equally worse on both the tests with or without added pressure to the situation. So for the LWM group it did not matter if there was added pressure in the situation. Under normal conditions HWM performs better than the LWM group as well because they have high levels of attentional capacities. However, when the attentional capacity was affected, from the pressure induced situation, the HWS advantage disappears. The most important finding is basically that those who have the highest capacity for success or HWM are the individuals who are more susceptible to failing while under pressure.

Subject 5002, control group, post-test

In this study, researchers are interested in whether individuals who rely more on their working memory are influenced by performance pressure while solving mathematics problems than individuals low in working memory. The findings suggest that there was no significant difference in individuals with low capacity working memory and being influenced by performance pressure. The findings further suggest that there was a significant relationship between individuals with high capacity working memory and performance pressure.

Scores received for assessment areas, sample coded summaries

Assessment area	Summary
	Subject 1048 (experimental)		Subject 5002 (control)
	Pre	Post	Pre	Post
Overall accuracy	1.5	4.5	3	2
Hypothesis/goals	.5	.5	1	1
Sample	0	1	0	0
Procedure	0	.5	0	0
IVs	0	.75	.5	.5
DVs	0	.75	0	0
Results	0	1	.5	.5
Interpretation	1	0	1	0

Appendix 4

Mean accuracy on each question of the JAC assessment at pre- and post-test by group

Question	Experimental group		Control group
	Pre	Post	Pre	Post
Purpose	.89 (.27)	.95 (.19)	.79 (.35)	.82 (.34)
Participants and procedure	.84 (.25)	.87 (.22)	.84 (.24)	.84 (.31)
Independent variables (IVs)	.72 (.31)	.84 (.22)	.67 (.38)	.60 (.43)
Dependent variables (DVs)	.52 (.30)	.73 (.28)	.51 (.32)	.45 (.35)
Results	.89 (.31)	.89 (.31)	.79 (.42)	.79 (.40)
Conclusions	.95 (.17)	.95 (.15)	.89 (.25)	.84 (.27)
Criticisms	.46 (.21)	.45 (.21)	.46 (.21)	.46 (.23)
Total JAC score	.75 (.11)	.81 (.10)	.71 (.13)	.68 (.15)

1. Values are proportion scores represented in the form M (SD). Experimental group, n = 86. Control group, n = 28

Appendix 5

Mean accuracy on each item of the summary task at pre- and post-test by group

Item	Experimental group		Control group
	Pre	Post	Pre	Post
Hypothesis/goals	.80 (.31)	.70 (.38)	.75 (.29)	.79 (.35)
Sample	.37 (.46)	.68 (.41)	.45 (.46)	.46 (.47)
Procedure	.52 (.29)	.61 (.29)	.41 (.27)	.38 (.32)
Independent variables (IVs)	.59 (.22)	.65 (.22)	.52 (.24)	.54 (.26)
Dependent variables (DVs)	.11 (.27)	.30 (.39)	.05 (.20)	.08 (.24)
Results	.35 (.31)	.45 (.32)	.29 (.29)	.30 (.25)
Interpretation	.17 (.29)	.24 (.37)	.21 (.37)	.18 (.34)
Total summary task score	.42 (.16)	.52 (.18)	.38 (.18)	.39 (.16)

1. Values are proportion scores represented in the form M (SD). Experimental group, n = 86. Control group, n = 28