American Journal of Community Psychology

, Volume 45, Issue 3, pp 310–324

Does the Amount of Participation in Afterschool Programs Relate to Developmental Outcomes? A Review of the Literature

Authors

    • National Center for Children and Families, Teachers CollegeColumbia University
  • Lizabeth M. Malone
    • National Center for Children and Families, Teachers CollegeColumbia University
  • Jeanne Brooks-Gunn
    • National Center for Children and Families, Teachers CollegeColumbia University
Original Paper

DOI: 10.1007/s10464-010-9303-3

Cite this article as:
Roth, J.L., Malone, L.M. & Brooks-Gunn, J. Am J Community Psychol (2010) 45: 310. doi:10.1007/s10464-010-9303-3

Abstract

Contrary to the findings from previous reviews we found little support for the general notion that greater amounts of participation in afterschool programs was related to academic, behavioral, or socio-emotional outcomes. However, some relationships did emerge depending on how participation was conceptualized and measured, and the methodology used to assess the relationship between participation and outcomes. For example, some benefits occurred when participants with high levels of participation were compared to non-participants, not when they were compared to other program participants. Several suggestions are offered to improve future research on the relationship between aspects of participation and developmental outcomes.

Keywords

Afterschool programsParticipationAttendanceOutcomesYouth

Introduction

American school-age youth spend a larger proportion of their weekly waking hours in discretionary activities than in school. The developmental consequences, both positive and negative, following from how they spend this time concern parents, policy-makers, and educators. Participation in unsupervised and unconstructive activities during the afterschool hours typically is associated with risky choices and poor adjustment, whereas participation in supervised, organized activities often results in increased educational attainment and achievement, fewer problem behaviors, and better psychosocial adjustment (Mahoney et al. 2005a). Funding, acceptance, and provision of afterschool programs, one of the many types of organized activities, blossomed in the 1990s (Halpern 2002; Roth and Brooks-Gunn 2003). Currently approximately 10% of American school-age youth—roughly 6 million—attend afterschool programs (Afterschool Alliance 2004; Carver and Iruka 2006).

One consequence of the heightened interest in afterschool programs has been an ever-increasing number of studies investigating whether participation in formal afterschool programs leads to academic improvements and, to a lesser extent, other types of developmental gains (e.g., increased self-esteem, reduced problem behaviors). Recent reviews indicate that youth participating in afterschool programs show greater academic and developmental gains when compared to non-participating youth (e.g., Durlak and Weissberg 2007; Lauer et al. 2006; Redd et al. 2002). Yet these reviews also find that program effectiveness varies greatly across programs and outcomes. Positive effects emerge on average across studies, but the majority of studies in each review do not find that program participants show higher academic performance than non-participants (Granger 2008).

Most of the research on the effectiveness of afterschool programs, and thus the reviews of this research, utilizes a dichotomous measure of youths’ absolute participation (yes or no) to compare participants to non-participants. This approach does not take into account the amount of participation. Attendance at afterschool programs is often sporadic, averaging one to two days per week, for the typical participants across studies in one review (Kane 2004). This raises questions about the amount of exposure to the program participants actually receive (i.e., level of treatment or intervention; Durlak and Weissberg 2007). Low levels of attendance among program participants may explain some of the non-significant findings (Bodilly and Beckett 2005; Kane 2004).

Amount of Participation in Afterschool Programs

Participation includes more than just attendance (Simpkins et al. 2004; Weiss et al. 2005). Participation, a complicated, multidimensional construct, involves at least five different aspects: intensity (i.e., frequency of attendance during one program year), duration (i.e., years of attendance), total exposure (i.e., frequency of attendance over multiple years), breadth (i.e., involvement in different types of program activities), and engagement (i.e., effort and interest in program activities). Evidence from different afterschool programs indicates substantial variability in all five aspects of participation. For example, intensity of participation averaged 58 days per year in one program for elementary school students (Dynarski et al. 2003), 80 days per year in another (Lodestar Management/Research 2005), and 126 days per year in yet another (Chang-Rios 2007). The percentage of elementary school students attending an afterschool program for two or more years ranged as well, from 15% (Leake and Gardner 2006) to 53% (James-Burdumy et al. 2005). Not surprisingly, participants’ total exposure to a program also varied. One study tracking participation over 5 years reported an average total exposure of 194 days (Goldschmidt and Huang 2007), while another reported an average of 324 days of attendance over 3 or more years (Center for Prevention Research and Development 2004). Breadth and engagement, although less studied across programs, appear to vary within programs as well. In one study of breadth, 26% of participants attended fewer than one-quarter of available activities while 34% attended more than three-quarters of them (Texas Education Agency 2005). In a study measuring engagement, 45% of participants agreed that the program offered interesting activities “a little”, and 49% thought it did so “a lot” (Walker and Arbreton 2004).

Age Differences in Amount of Participation

Nationally, the percentage of youth attending afterschool programs drops in half after fifth grade (Afterschool Alliance 2004; Carver and Iruka 2006). The intensity and duration of their participation likewise declines with age. Elementary school students attend afterschool programs for more days per week than middle school students (e.g., two to three versus one to two days; Dynarski et al. 2003). Duration can also decline in relation to youths’ age. In one study, for example, 75% of kindergartners continued to attend the afterschool program the following year, but only 33% of fifth graders did so (Leake and Gardner 2006). Data on age differences in breadth and engagement are not available.

The decline in participation likely reflects the greater independence and choice parents give older youth (Guavain and Perez 2005) as well as an increase in the availability and time spent in extracurricular activities beginning in middle school (Marsh and Kleitman 2002). Youth, regardless of their age, can benefit from the supportive relationships with adults and supervised creative outlets for constructive time use afterschool programs provide (e.g., Search Institute 2006a, b). Still, attending an afterschool program, and the level of participation, likely means something different for a middle school student, when program attendance is less normative and choices are greater, than it does for an elementary school student.

Amount of Participation and Developmental Outcomes

The growing body of evidence documenting the positive developmental benefits from participation in afterschool programs, compared to no participation, coupled with the wide variability in participation among participants, poises the field to move beyond the simplistic yes–no distinction to investigate more nuanced questions of participation. Researchers only recently began to empirically investigate how different aspects of participation (e.g., intensity, breadth) contribute to changes in developmental outcomes. Conventional wisdom holds that youth who experience greater amounts of participation in an afterschool program that offers structure, adult supervision, and a range of enriching activities should demonstrate improved academic, behavioral, and socio-emotional outcomes. The answer from existing research, however, is less clear. In this paper we synthesize the empirical studies to determine if the existing evidence supports this belief. Finding many limitations in the current research literature, we also offer suggestions on how to improve subsequent research on aspects of participation.

Extant Reviews

We know of only two other reviews of the literature on the association between the amount of participation and outcomes (Lauer et al. 2006; Simpkins et al. 2004). Neither one, however, focused solely on formal afterschool programs. Lauer et al. (2006) used meta-analytic techniques to determine the effectiveness of out-of-school time programs (i.e., formal afterschool, tutoring, and summer school programs) for improving reading and math achievement. The total number of available program hours was a significant moderator of program effectiveness. The largest effect sizes for improving achievement came from programs lasting approximately 45 or more hours. This study, however, did not answer how youths’ actual time in the programs affected their achievement. Actual intensity, duration, or total exposure were not evaluated.

In the other review, Simpkins et al. (2004) presented a narrative review of the link between intensity, duration, and breadth of participation in out-of-school time activities (i.e., afterschool programs and extracurricular activities) and developmental outcomes. They concluded that greater participation was associated with better outcomes. Two aspects of the review, however, call into question this positive conclusion. First, they included studies utilizing single group pre-post designs without statistical controls, increasing the likelihood of positive findings, and thereby reducing confidence in the cross-study conclusions (Shadish et al. 2002). They also included extracurricular activities (e.g., sports teams) which differ from formal afterschool programs in, among other things, intensity, duration and breadth of participation (Roth et al. 2008). As a result, it is possible that the positive findings regarding participation were influenced by study design or the type of out-of-school time activities examined. Furthermore, despite potential differences in the amount of afterschool program participation for elementary and middle school students (e.g., Carver and Iruka 2006; Leake and Gardner 2006), neither review explored the possibility of differential effects from the amount of participation due to participants’ grade level.

Methodological Critique

Four methodological issues common to research on afterschool program participation complicate efforts to understand the associations between the aspects of participation and outcomes: (1) how the aspect of participation is measured; (2) what type of comparison groups are used; (3) when the outcomes are measured; and (4) the variety of program models studied. An additional goal of this review, unlike the previous reviews, is to determine the potential influence of these issues on the association between aspects of participation and outcomes.

The first methodological issue involves the divergent metrics used for each aspect of participation. Participation varies within and across programs. Accurately capturing this variability, however, presents a challenge to researchers. Part of the problem stems from the unavailability of accurate and detailed daily attendance records (Fiester 2004). A lack of consistent definitions and metrics for measuring participation poses another obstacle, particularly for synthesizing findings across studies. In reviewing the studies for this paper we encountered a dizzying array of terms and metrics used to describe the amount and type of participation. For example, researchers have measured intensity of participation as the hours per day, or days per week, month, or year youth attend the program. Duration of participation has been measured in months, semesters, or years. Similarly, measures of total program exposure include the number of days of attendance over two or more years as well as a dichotomous indicator of participants who do and do not attend for a minimum period of time over two years (e.g., 60 days or 60% of the possible days). The variety of activities youth participate in while at the program, their breadth of participation, has been measured as a count of activities, a percentage of available activities, or by the mix of specific types of activities (e.g., educational only, educational and recreational). Measurement of youths’ level of engagement in program activities has been less variable, likely due to its rarity.

Additionally, variation exists in how the measures are used in analysis. Some studies of intensity, duration, and total exposure treat participation as a continuous predictor of the outcome. These studies provide information on the influence of more or less participation but do not help with questions about optimal amounts of participation, such as the minimal level of participation necessary to improve outcomes. Other studies of these aspects investigate the amount of participation categorically, thereby providing information on the benefits of participating for a certain amount of time, such as 60 days per year or more, or 60% of the available days. The findings from these types of studies provide a starting point from which to answer to the question of optimal amounts of participation.

The second methodological issue pertains to the comparison group. The literature has compared participants with varying amounts of participation within the same program (e.g., high vs. low) and/or compared program participants with high levels of participation to youth not attending the program. The first provides information on the benefits or risks of greater participation compared to less. The second type of comparison only tells us if youth with participation levels above a certain level (e.g., 60 days) outperform youth not attending the program.

The third methodological issue involves the study time frame for measuring the outcomes. Most studies in this literature measured developmental outcomes concurrently with program participation, typically at the end of the school year. These studies tell us whether or not participation was associated with short-term changes in outcomes. A few studies collect follow-up data months or years after the youth left the program, providing information about the potential for lasting effects from participation.

The fourth issue concerns the variety of program models evaluated. Several studies in the field examine the same program models (e.g., Citizens Schools), often in the same sites. Additionally, within a single study, the researchers may conduct analysis on different cohorts of students attending the same programs (e.g., 6th or 8th graders attending Citizen Schools). Treating each study and each association equally may weight a review’s results toward certain program models rather than provide answers for the broader set of afterschool programs.

Current Review

In the present review of the literature, we consolidate the growing number of empirical studies investigating aspects of participation to examine if youth benefit from greater intensity, duration, total exposure, breadth, or engagement of participation in afterschool programs. Consistent with the evidence that afterschool programs hold promise for improving the lives of youth, not merely their academic performance, we examine a range of developmental outcomes. Thus, the primary goal of this review was a synthesis of the empirical literature. As part of the synthesis, we sought to address questions raised in our review of the literature, including differences due to age of participant and the implications of the literature’s methodological issues on the synthesis.

Method

Literature Search

We located the studies included in this review using a multi-pronged search for empirical literature on afterschool programs (as defined below) and developmental outcomes. We conducted searches of 13 electronic databases (e.g., PsychInfo, ERIC) and 26 websites (e.g., Public/Private Ventures, Child Trends) using keywords and phrases such as afterschool programs and program evaluations and their variations. We also used more informal methods, inspecting the references in articles and obtaining work brought to our attention through colleagues, email alerts, and electronic mailing lists. Studies eligible for review had to meet the following criteria: (1) examine a formal afterschool program serving elementary or middle school students; (2) appear in English by 2007; (3) report the results of at least one statistical analysis relating one or more aspect of participation to at least one developmental outcome; and (4) include pre-test measures.

As specified by the first criteria, this paper focuses on formal afterschool programs. Although formal afterschool programs themselves vary on many dimensions, including their primary focus (e.g., academic improvement vs. youth development opportunities), staff qualifications, and hours of operation, to name a few, all formal afterschool programs share the general goal of providing youth with regular access to a safe and enriching environment during the non-school hours. Specifically, formal afterschool programs consist of the following features: they (1) meet on a regular basis throughout the school year; (2) are supervised by adults; (3) offer more than one type of activity (e.g., homework help, recreation, arts and crafts); and (4) are structured around group-based activities. This criterion resulted in the exclusion of extracurricular activities (e.g., sports teams, dance lessons) as well as programs with a single focus (e.g., drug prevention programs), exclusive use of one-to-one mentoring or tutoring, or an intensive case management approach. Additionally, we excluded studies of programs for high school students due to the unique nature of programs (e.g., apprenticeships; Goerge et al. 2007). We identified 142 potential studies meeting these features that were published in English by 2007.

Next, included studies had to match the focus of our review—understanding the association between aspects of participation and developmental outcomes. We eliminated studies containing only comparisons of youth based on their absolute attendance (yes versus no) as well as studies of only participants that did not examine differences in an aspect of participation, reducing the pool of studies to 51. We also required that studies include pre and post assessments of the outcomes. The issue of self-selection looms large in research on the effects of afterschool programs: Youth who attend afterschool programs more often or for longer, or who are more involved once at the program, can differ from both their peers who do not participate at all and their peers with lower levels of participation in observable (e.g., socioeconomic status) and unobservable (e.g., motivation) ways (see Bodilly and Beckett 2005). Researchers cannot manipulate youths’ levels of participation through random assignment. Instead, they must rely on elements of quasi-experimental research designs, such as pretests and statistical controls for pre-existing differences, to reduce the plausibility of alternative explanations, including self-selection (Shadish et al. 2002). We, therefore, only included studies utilizing these techniques. On rare occasions pre-test scores for problem behaviors were not used, but in these instances, analyses included other demographic characteristics and pre-test scores for other areas of development (e.g., academic scores). This pre-test criterion excluded an additional 16 studies, leaving 35 studies meeting all four criteria for inclusion. The studies used for the synthesis are designated with an asterisk in the reference list. A table detailing the characteristics and findings of each study is available from the first author upon request.

Synthesizing the Literature

We conducted a narrative review of the literature to synthesize the research. Narrative reviews are often criticized for their selectivity in describing studies that support a particular view. In an effort to avoid the common pitfalls of narrative reviews, we adopted a strategy that determines, for each aspect of participation and outcome, the number of significant findings across studies in relation to the overall number of relevant statistical analyses conducted. Our discussion of the extant literature thus revolves around the percentage of significant findings. We believe this approach is appropriate given the status of current research on participation.1 The first step in calculating the percentage of significant findings was to determine the number of significant associations between each aspect of participation and each outcome for a given study.

Aspects of Participation

Definitions for the five aspects of participation we examined—intensity, duration, total exposure, breadth and engagement—reflect the prevailing approach to measuring participation evident in the literature. Intensity of participation refers to the frequency of attendance during one program year. Participation measures coded as intensity included mostly the number of days per year of attendance but also the number of hours per week, days per week, days per month, and the length of the program. Duration of participation refers to the number of years of attendance (e.g., more than one year, two years). Total exposure measures the frequency of attendance over multiple years. The number of days of attendance over two or more years was coded as a measure of total exposure as were measures indicating youth who met minimum exposure criteria (e.g., 60 days, 60% of possible days) and measures tallying the number of days of attendance during the second year of participation. Breadth of participation refers to involvement in different types of program activities while at the afterschool program. We coded measures of the number of activities, percentage of available activities attended, or the mix of activities (e.g., academic plus recreational versus recreational only) as capturing breadth. Engagement of participation refers to the process of active involvement, evident in participants’ behaviors, such as effort and attention, and their emotions, such as interest and enjoyment.

Outcomes

We grouped the many outcomes investigated across the studies into seven categories. Four were academic outcomes: academic performance (e.g., grades, test scores, and progression in school), academically-related attitudes and beliefs (e.g., feelings about school, educational expectations, and academic self-perceptions), learning behaviors that display a positive approach or commitment to learning (e.g., effort and work habits) and attendance at school (e.g., absenteeism and tardiness). Problem behaviors included school-related problem behaviors (e.g., suspensions) as well as substance use, misconduct, or delinquency (e.g., stealing, aggressive behavior, acting out). Two aspects of socio-emotional functioning were included: peer relations (e.g., number of positive peers and peer interactions) and self-concept.

Coding Findings Across Studies

We coded the associations between each aspect of participation and each outcome for a given study as significant and positive, significant and negative, or not significant at the p < 0.05 level. When a study reported findings for multiple measures fitting a single outcome category, such as test scores and grades for academic performance, we required that at least one half of those multiple measures be significant for the category to be coded as having a significant association for that study to avoid potentially giving undue weight to a single study. After scoring each of the associations in each study, we next calculated the percentage of significant positive findings across the studies for each aspect of participation and outcome category. Thus, the percentage of significant findings refers to the number of significant and positive associations occurring at the p < 0.05 level divided by the total number of associations assessing the same relationship between a particular aspect of participation and an outcome. For example, across studies we found 21 tests of the association between intensity of participation and academic performance. Five, or 24%, were significant and positive.

We recalculated the percentage of significant positive findings across studies by subgroups to explore differences due to participants’ age group and the four methodological issues on the synthesis of the findings. We calculated the percentages separately for elementary and middle school students. The tests of association for the mixed-aged samples were not included in these recalculations. To investigate the first methodological issue, measurement, we recalculated the percentages for each aspect of participation to include only tests using similar ways of measuring that aspect of participation. The choice of the specific methods for each recalculation was driven largely by the existence of enough tests using that method. More specifically, the recalculation of the percentage of significant findings for the intensity of participation was limited to only those tests where intensity was measured as the number of days per year of attendance. The recalculation for duration included only tests comparing 2 year participants to non-participants. There were not enough tests to recalculate the percentages for other ways of measuring intensity or duration. For total exposure, we recalculated the percentages both as the number of days across three or more years and as a minimum number of days over 2 years. The small number of associations with breadth and engagement did not permit recalculations.

For the second methodological issue, type of comparison group, we recalculated the percentages for two subgroups of tests. One included only tests comparing participants with higher and lower amounts of participation. The second subgroup included only tests comparing participants with higher levels of participation to non-participants.2 For the study time frame we recalculated percentages separately for studies measuring post-test outcomes concurrently with participation (i.e., at the end of the program year) and for those collecting any follow-up data (i.e., after their time in the program).

The fourth methodological consideration addressed the existence of multiple tests of associations of the same aspect of participation within afterschool programs using the same program model. This typically occurred when a report examined multiple cohorts or age groups attending the same program sites or when a multi-phased evaluation resulted in a series of reports. In an effort to reduce the potential influence of findings from a particularly strong or weak program model, we recalculated the percentages by afterschool program model by collapsing the findings across studies to code one summative association per program model, using the 50% or more rule to designate the program’s association as significant or not. For example, three tests of the association between intensity of participation and academic performance were reported as part of the Citizen Schools evaluation. In only one of these tests was the association significant; thus, we recoded the category as not significant and reduced the denominator (i.e., the number of tests) by two when calculating the percentage of significant findings between intensity and academic performance. By collapsing the findings this way, the findings from the Citizen Schools evaluation contributed one finding rather than three. We collapsed findings only when enough information was provided to determine that the same program models were used.

Table 1 presents the total number of associations across studies and the percentage of significant and positive findings in bold for each aspect of participation and developmental outcome. The rows below the overall findings show the recalculated percentages based on age group and methodological considerations. We do not show the percentages for, or discuss in the text, sets of associations based on fewer than four associations to avoid placing undue weight on the findings from a limited number of analyses. This criterion eliminated any discussion of the findings pertaining to breadth or engagement. We interpret categories where at least half of the associations across studies were significant and positive as evidence of a positive link between that aspect of participation and outcome.
Table 1

Number and percentage of significant findings between aspects of participation and developmental outcomes

 

Intensity

Duration

Total exposure

Breadth

Engagement

N

% Sig.

N

% Sig.

N

% Sig.

N

% Sig.

N

% Sig.

Academic performance

21

24

12

42

12a

33

3a

 

2

 

 Elementary school

11

36

8

50

5

40

1

 

2

 

 Middle school

10

10

4

25

6

17

1

 

0

 

 Measurementb

18

28

10

40

6 (5)

33(40)

 

 

 High vs. low participation

17

18

4

25

8

38

3

 

2

 

 High vs. non-participants

4

50

8

50

4

25

0

 

0

 

 Concurrent outcomes

21

24

10

40

9

33

3

 

2

 

 Follow-up outcomes

0

 

2

 

3

 

0

 

0

 

 Programs

15

27

5

60

6

50

3

 

1

 

Academic attitudes/beliefs

6

0

2

 

3

 

1

 

0

 

 Elementary school

3

 

1

 

1

 

1

 

0

 

 Middle school

3

 

1

 

2

 

0

 

0

 

 Measurementb

5

0

2

 

0 (2)

 

 

 

 High vs. low participation

6

0

0

 

3

 

1

 

0

 

 High vs. no participation

0

 

2

 

0

 

0

 

0

 

 Concurrent outcomesc

6

0

2

 

3

 

1

 

0

 

 Programs

4

0

1

 

2

 

1

 

0

 

Learning behaviors

8

0

5

40

3

 

1

 

0

 

 Elementary school

4

0

2

 

1

 

0

 

0

 

 Middle school

4

0

3

 

2

 

1

 

0

 

 Measurementb

7

0

4

25

1 (2)

 

 

 

 High vs. low participation

8

0

1

 

3

 

1

 

0

 

 High vs. no participation

0

 

4

25

0

 

0

 

0

 

 Concurrent outcomesc

8

0

5

40

3

 

1

 

0

 

 Programs

5

0

3

 

2

 

1

 

0

 

Attendance

11

64

5

0

8

75

3a

 

0

 

 Elementary school

3

 

3

 

2

 

1

 

0

 

 Middle school

8

75

2

 

5

60

1

 

0

 

 Measurementb

9

78

5

0

4 (3)

75

 

 

 High vs. low participation

8

50

3

 

4

75

3

 

0

 

 High vs. no participation

3

 

2

 

4

75

0

 

0

 

 Concurrent outcomes

11

64

5

0

7

86

    

 Follow-up

0

 

0

 

1

 

3

 

0

 

 Programs

7

71

3

 

5

80

3

 

0

 

Problem behaviors

13

38

6

33

4

25

1

 

0

 

 Elementary school

7

43

4

25

0

 

1

 

0

 

 Middle school

6

33

2

 

4

25

0

 

0

 

 Measurementb

10

40

5

40

3

 

 

 

 High vs. low participation

9

22

2

 

1

 

1

 

0

 

 High vs. no participation

4

75

4

50

3

 

0

 

0

 

 Concurrent outcomes

12

33

5

40

3

 

1

 

0

 

 Follow-up outcomes

1

 

1

 

1

 

0

 

0

 

 Programs

9

44

4

25

2

 

1

 

0

 

Peer relations

13

31

5

20

4

25

1

 

1

 

 Elementary school

9

33

3

 

2

 

0

 

1

 

 Middle school

4

25

2

 

2

 

1

 

0

 

 Measurementb

10

20

5

20

1 (2)

 

 

 

 High vs. low participation

12

25

1

 

4

25

1

 

0

 

 High vs. no participation

1

 

4

25

0

 

0

 

0

 

 Concurrent outcomesc

13

31

5

20

4

25

1

 

0

 

 Programs

9

44

3

 

3

 

1

 

1

 

Self-concept

6

0

3

 

2

 

2

 

0

 

 Elementary school

3

 

1

 

1

 

1

 

2

 

 Middle school

3

 

2

 

1

 

1

 

0

 

 Measurementb

5

0

2

 

0 (2)

 

 

 

 High vs. low participation

6

0

1

 

2

 

2

 

0

 

 High vs. no participation

0

 

2

 

0

 

0

 

0

 

 Concurrent outcomesc

6

0

3

 

2

 

2

 

0

 

 Programs

4

0

2

 

1

 

2

 

0

 

Percentage of significant findings shown only for associations examined in four or more studies

–, not assessed in this paper. Significant associations at p < 0.05. Percentages are within cell for significant and positive associations out of tested associations for that cell (e.g., intensity X academics)

aTotal does not equal the sum of the age groups because one study examined a mixed age group (e.g., K-8)

bIncludes only associations from studies using similar measurement. This was the number of days per year for intensity, comparison of two year participants to non-participants for duration; and for total exposure, both as a minimum number of days over two years and as the number of days across three or more years (shown in parentheses)

cAll of the associations were measured with concurrent outcomes

Results

The data in Table 1 show two main patterns of findings. First, there are few findings indicating a positive link between any aspect of participation and outcomes. When such links do appear, they tend to be qualified in some way by either the age of the participants or the methodological approach taken in the studies. Second, the blank cells in the table highlight the lack of research on certain outcomes and aspects of participation.

Only 2 of the 35 bolded comparisons (i.e., 5 aspects of participation multiplied by 7 developmental outcomes) show evidence of a positive link (i.e., 50% or more of the statistical associations were significant); greater amounts of participation were positively linked with improved attendance at school. The improvements emerged when youth attended afterschool programs more frequently during a single year (i.e., intensity) or more frequently over multiple years (i.e., total exposure). The recalculations by age group, however, qualified these findings. The positive links were evident for middle school students only. There were not enough studies involving elementary school students to interpret.

At least half of the findings remained significant when we recalculated the percentages based on the four methodological critiques. These recalculations provided greater specificity in understanding how participation relates to attendance at school. For example, how participation was measured did not alter the findings of a positive link. Greater intensity of participation was positively linked to improved attendance at school when either a specific intensity measure (i.e., more days per year of attendance) or a more general measure (i.e., any measure of the frequency of participation within a given year) was used. Greater total exposure was positively linked to improved attendance at school when one of the specific total exposure measures (i.e., a minimum number of days over 2 years) or the more general measure (i.e., any measure of the amount of attendance over multiple years) was used. There were not enough tests of association to interpret the findings from studies employing the other specific total exposure measure, the number of days over three or more years. The findings from recalculations for the second methodological issue, the type of comparison group, demonstrate that the linkages remained when youth with higher levels of intensity or total exposure were compared to program participants with lower levels. Thus the positive linkages were not the result of comparisons between youth with higher levels of participation and non-participating youth. Additionally, since only one of the associations collected follow-up data, the recalculations by study time frame show that greater intensity or greater total exposure was positively linked to higher attendance at school only while the youth were attending the program. The positive linkages remained when findings were collapsed by program models, indicating that the linkages were evident across different afterschool program models.

While we found only two overall linkages, more emerged when we considered separate age groups and the four methodological issues. First, considering age, greater duration of participation was positively linked to better academic performance among elementary school students, but not middle school students. Second, focusing on the type of comparison group, greater intensity or duration was linked to improved academic performance and fewer problem behaviors, but only when youth with high levels of participation were compared to youth not attending the afterschool program. Third, although overall differences were not detected, greater duration or total exposure was linked to improved academic performance when findings from the same program model were collapsed.

More than half of the 35 bolded percentages in Table 1 are blank, indicating that not enough associations met our minimum (i.e., four or more) to synthesize the research across studies. The small number of associations for two of the five aspects of participation—breadth and engagement—hindered our ability to draw conclusions on their contribution to improved outcomes among participants. Intensity of participation was the most often studied aspect of participation; it was also the only aspect examined in relation to each of the seven outcomes. The links between duration of participation and two outcomes, academic attitudes and beliefs and self-concept, could not be interpreted due to the small number of associations. Similarly, there were too few associations between total exposure and three outcomes, academic attitudes and beliefs, self-concept, and learning behaviors. Table 1 clearly shows the lack of participation research pertaining to certain outcomes, particularly academic attitudes and beliefs and self-concept. Furthermore, the relatively few tests for many of the associations precluded interpreting findings based on participants’ age group.

With regard to the methodological critiques, the blanks in Table 1 show further gaps in the literature. Few studies used the same measurement strategies to assess total exposure or compared youth with higher versus lower levels of participation duration or total exposure. Few studies collected any follow-up data.

Discussion

The present study aimed to synthesize the extant research on the associations between the amount of program participation (i.e., intensity, duration, total exposure, breadth, and engagement) and program participants’ development. Contrary to previous findings, we found little support for the benefits of greater amounts of participation. In most cases, the majority of findings examining the link between greater amounts of participation and improved developmental outcomes were not significant. More specifically, greater intensity, the most commonly studied aspect of participation, was significantly associated with only one outcome, greater attendance at school, in the majority of findings, but was not significantly associated with other improvements in academic outcomes, or with behavioral, or socio-emotional outcomes. Similarly, greater total exposure was significantly related to greater attendance at school in the majority of findings, but was also not linked to any of the other outcomes across studies. The majority of findings failed to find a significant association between longer duration of participation and any of the outcomes. Too few studies examined breadth or engagement to synthesize at this point.

Previous reviews have offered more positive conclusions about the influence of participation on outcomes, finding that greater intensity or duration in out-of-school-time activities was positively associated with a variety of academic and non-academic outcomes (Lauer et al. 2006; Simpkins et al. 2004). The contrast between our findings and the more positive conclusions offered in previous reviews likely stems from differences in the inclusion criteria. Lauer et al.’s meta-analysis measured possible, not actual, hours of participation and included afterschool and summer school programs in their analysis. They found that summer school programs, however, were more effective at improving reading achievement than afterschool programs, tempering the differences between our current review and their work. Although Simpkins et al. examined youth’s actual participation, they included extracurricular activities along with afterschool programs. Extracurricular activities are more often linked to better developmental outcomes than afterschool programs, at least for adolescents (Roth et al. 2008). Furthermore, they included studies utilizing non-rigorous designs (e.g., pre-post designs) without statistical controls, increasing the likelihood of positive findings (Shadish et al. 2002). Thus, our review provides a more direct assessment of the implications of varying amounts of participation in afterschool programs.

Our findings further extend previous reviews by examining the possibility of differential associations due to the age of the participants. Although the small number of studies examining certain aspects of participation (i.e., breadth and engagement), and outcomes (i.e., academic attitudes and beliefs and self-concept) limited our conclusions, we found little evidence of differential associations due to age group. In only two of the 10 sets of associations that we could examine did the results vary. Middle school students improved their attendance at school with greater intensity of participation in the majority of findings. There were not enough studies to interpret the association for elementary school students. The academic performance of elementary school students, but not middle school students, improved with greater duration (i.e., more years of participation) in half of the findings.

Increasingly, afterschool programs are judged on their effectiveness in improving academic performance (e.g., Kane 2004). These findings lend support to current efforts to increase participation. Afterschool programs have a harder time recruiting and retaining youth as they move from elementary to middle school (Herrera and Arbreton 2003; Lauver et al. 2004). The findings from this review suggest that increasing young adolescents’ attendance, while difficult, could be worthwhile as school-day attendance is necessary for academic gains (Lamdin 1996). One suggested strategy for improving the attendance of middle school age adolescents is to offer flexible attendance options, such as signing up for certain days or time, or to allow youth to attend on a drop-in basis. These efforts, however, need to be balanced with the benefit of more intensive involvement. Flexible schedules coupled with clear messages about expected attendance, appealing programming, and staff who can connect with adolescents can increase participation intensity (Herrera and Arbreton 2003; Lauver et al. 2004).

Researchers have also sought to identify and address barriers to continued participation in afterschool programs (i.e., duration), including structural issues such as transportation and cost as well as programmatic issues such as a lack of interesting activities or good rapport with staff (e.g., Lauver et al. 2004). In one study of 550 afterschool programs in New York City, the researchers compared structural and programmatic elements of programs with higher and lower retention rates. They found that elementary school programs with higher retention rates the following year employed directors with more advanced educational credentials, utilized a volunteer parent liaison, offered a strong academic component along with enrichment activities (i.e., breadth), and had higher program attendance rates (i.e., intensity; Pearson et al. 2007).

Bolstering Studies of Breadth and Engagement

Our review also highlights many limitations in the extant research, most notably the small number of studies investigating breadth and engagement, crucially important aspects of participation that play central roles in programs’ appeal and quality. Program participants often give the availability of a variety of activities as a reason for attending (Borden et al. 2005), and a breadth of activities likely facilitates greater amount of participation (Fiester 2004). A recent review of nine youth program quality instruments identified engagement as a core concept in each (Yohalem and Wilson-Ahlstrom 2007). As these should be burgeoning areas of research, we review and critique the methods used in the few studies we found. We then propose directions for future measurement of each construct in an effort to establish a stronger empirical base.

Breadth

The variety of activities provided within an afterschool program differs across programs, but typically includes academic activities (usually homework help), enrichment activities, and structured and/or unstructured recreation options. Offerings vary by day of week, time of year, and year. Although many program evaluations describe the percentage of youth who participate in particular activities, researchers do not often report the overlap or combination of activities for an individual attendee (i.e., breadth) or use it in outcome analyses (e.g., Dynarski et al. 2003; Reisner et al. 2004; Mahoney et al. 2007). The three studies in this review that did seek to test the association between breadth of participation and outcomes exemplify the diversity of methods researchers use to capture breadth. Two measured breadth by the quantity (number or percentage) of activities youth participated in (Baker and Witt 1996; Texas Education Agency 2005). This method, however, fails to indicate the content of the activity, and whether youth experienced a variety of content types. It is the variety, rather than the sheer number of activities, however, that is hypothesized to lead to better outcomes for participants; a greater variety of activities offers support and skill-building opportunities in more domains (e.g., social, academic; Eccles and Gootman 2002).

The third study of breadth classified the activities youth attended into five categories—education (homework help/tutoring and education enrichment), arts and recreation, leadership, career, and health—that were then collapsed into education only, non-education only, or a combination of the two for the outcome analyses (Walker and Arbreton 2004). Future research needs to capture, at a minimum, participation in these two categories of activities, in order to inform afterschool program policy debates and practices regarding the most advantageous balance of academic and recreational activities. For example, Walker and Arbreton (2004) found lower grades and test scores among the program participants who attended only non-academic activities. In contrast, preliminary findings suggest that reading grades did not improve when students participated in many academics activities and few non-academic activities but did show improvement among students who participated in both types of activities (Van Egeren et al. 2006).

Breadth also refers to the total array of activities in which youth participate, not just those within an afterschool program (Simpkins et al. 2004). Few program evaluations collect data on breadth of participation outside of the program (e.g., other organized activities youth attend when not at the program). Yet many program participants report participating in at least one other organized activity, such as sports or scouts (Gambone and Arbreton 1997; Vandell et al. 2005a, b). Some youth, therefore, acquire the diversity of activities outside of their afterschool program. Measures of breadth that gather the full spectrum of activity participation can be used to assess the relative benefits of different profiles of participation. For example, the study by Vandell et al. (2005a, b, 2006, 2007) did examine the full spectrum of activity participation, although we did not include these findings in the synthesis. They found more benefits among elementary school youth who showed high program participation and low participation in other activities than among the youth with high participation in both the program and other activities. The reverse was true for middle school students; adolescents who combined high program participation with high extracurricular activity participation showed more benefits. Additional studies measuring the gamut of breadth would provide needed advice to practitioners and policy-makers regarding the most advantageous combinations both within an afterschool program and across a community’s out-of-school time options.

Engagement

Without youth demonstrating effort or interest in the afterschool program, the frequency of attendance or the breadth of activities may have little influence on improving outcomes. Engagement refers to the cognitive, behavioral, and emotional attributes necessary to connect to the people and activities at the afterschool program (Weiss et al. 2005). In program research, engagement is typically measured by such constructs as youths’ sense of belonging to the program or the effort, enjoyment, and interest youth express during activities (Bartko 2005; Vandell et al. 2005a, b). These measures, however, are rarely examined in association with outcomes. In the two studies included in the current review, engagement was measured at both the individual and program level using staff ratings and independent observations. For individual youth, Mahoney et al. (2005c) had staff rate youth for enjoyment, effort, and interest in academic and non-academic activities. Staff ratings, however, may be biased. For example, a halo effect is possible for regular attendees. Observations were used to rate engagement at the program level; observers rated the program based on how focused, engaged, and interested in program activities the participants were (Mahoney et al. 2007). Observations, however, are more labor-intensive, and only provide a snapshot for a given day and given set of attendees. Program evaluations conducted by Public/Private Ventures typically measure participants’ engagement by surveying youth directly about the availability of challenging and interesting activities and their sense of belonging (see Gambone and Arbreton 1997; Walker and Arbreton 2004), but only recently have the researchers reported the associations between engagement and outcomes (Arbreton et al. 2008b). Youth surveys, however, may be unreliable for younger participants.

Although each method of data collection has its drawbacks, the existing studies suggest that engagement should be measured in relation to specific activities. Across methods, engagement levels are higher for enrichment activities (Mahoney et al. 2007; Walker and Arbreton 2004). The existing studies further suggest that engagement is linked to aspects of quality, such as the social climate, adult support, and safety (Arbreton et al. 2008b; Mahoney et al. 2007; Walker and Arbreton 2004). Research to date cannot determine the directionality of this link. Repeated measurement of engagement and quality over time will be necessary to determine if quality lays the groundwork for engagement or visa-versa (Mahoney et al. 2007). Thus, we recommend that, at a minimum, questions about belonging, effort, enjoyment and interest in relation to specific activities be incorporated into youth surveys.

Improving the Methodology in Studies of Aspects of Participation

A second goal of this review was to determine how four methodological issues common to the evaluation literature might influence the findings. The first methodological issue pertained to the lack of consistency in measurement across studies. Recalculating the findings to include only associations in which participation was measured in the same metric did not alter our findings. The small size of the existing literature base did not allow us to fully explore the influence of measurement however. Researchers are often constrained by the type of attendance information programs routinely collect. Yet more knowledge about the amount of time (e.g., intensity, duration, and total exposure) participants spend in afterschool programs and how it relates to program outcomes is necessary before the research can provide guidance on the optimal number days and/or years of participation. To remedy this problem, we recommend that programs collect detailed daily activity attendance data that includes the specific number of days and hours of attendance. Such detailed daily information on participating youths’ level of participation would benefit both programs and research (Fiester 2004). This type of information would provide useful information for practice, such as the ability to evaluate how different activities attract and retain youth, or to calculate cost-benefit analysis in times of budget cuts (Arbreton et al. 2008a; Walker and Arbreton 2004). This type of information would allow researchers to score participants on a variety of aspects of participation, including intensity, breadth, and when collected over time, duration and total exposure. The establishment of detailed daily attendance records would assist in longitudinal investigations of duration and total exposure by providing data on participants who left the program.

The type of comparison group was another confounding factor across studies. A larger percentage of significant findings emerged from studies comparing youth with high levels of participation intensity, duration, or total exposure to non-participants than from studies comparing participants with varying levels. Most of the literature on aspects of participation comes from evaluation studies seeking to establish program effectiveness and, thus, relies on experimental and quasi-experimental designs. Comparison with non-participants, however, does not provide as useful of information with regard to the benefits of greater amounts of participation. Thus, even in evaluations of program effectiveness, daily attendance data can be used to better understand the influence of varying levels of participation within the treatment group (see Moore 2008).

We strongly recommend that future studies empirically compare participants with participation levels above and below a specific threshold. Such analyses would provide needed information about the minimum amount of participation necessary to expect developmental gains. The thresholds for the intensity of participation tested in the existing studies varied greatly, precluding recommendations about a minimal level of participation. Recent recommendations for programs serving preteens suggest that youth need to attend for at least three days per week during the school year (Metz et al. 2008). This level of intensity, however, is higher than the high intensity groups in the extant research. For the total exposure, the thresholds used in the studies ranged from 49 to 60 days per year for two years, between once and twice a week.

The study time frame was the third methodological issue we examined. The existing body of research contains few studies of outcomes beyond the end of the program. It is feasible to expect that greater levels of participation could have lasting impacts on participants. Longer longitudinal studies are necessary to assess this possibility. Studies that extend past the school year also would provide researchers with more opportunity to assess duration and total exposure.

The final methodological issue pertained to the variety of program models studied. The findings suggest that in the few instances where the majority of the associations were significant, the positive findings cannot be attributed to a single program model. That is, evaluations from one particular program model did not account for all the positive findings. Rather, the findings emerged across a variety of program models. Repeated study of a particular program model can provide needed information for program improvement, but studies of a variety of program models is necessary to gain a better understanding of how the aspects of participation relate to outcomes across programs.

Limitations of this Review

Although the present review offers a unique narrative summary of the strength of the associations between greater amounts of participation and developmental outcomes, some methodological limitations deserve mention. We coded categories as having a significant association when at least half of the tests were significant, an admittedly liberal criterion. Adopting a more conservative benchmark, however, would not substantially alter our findings given how few associations were significant with the liberal cut-off.

The narrowness of our definitions of the aspects of participation poses another set of limitations. Our definitions for intensity and duration, adapted from a previous review of the literature (Simpkins et al. 2004), distinguish between frequency and length of participation. Neither alone, however, accurately captures the reality of youths’ participation in programs. Studies of intensity measure participation during a discrete amount of time and, therefore, do not measure the full extent of participation, such as the number of years of participation prior to the beginning of the study (Arbreton et al. 2008a). Studies of duration, on the other hand, do not take into account the intensity of participation during the years of participation and, thus, mix youth with frequent and infrequent attendance. Although the construct of total exposure captures some of this complexity, at least with regard to duration and intensity, fewer researchers measure participation this way. Further, a participant can have high total exposure, but partake in only a small fraction of the available activities. Researchers typically do not include the possibility of interrelations in their empirical studies of the associations between the amount of participation and outcomes, and thus our review cannot.

Conclusions

This review of the extant literature found few links between the amount of participation and developmental outcomes. This evidence suggests that general statements proclaiming that greater participation in formal afterschool programs leads to improved outcomes are premature and inaccurate. Instead, these findings illustrate that different developmental gains are associated with different aspects of participation. They further show that the links between outcomes and aspects of participation are qualified by age group or study methodology.

The dearth of findings pertaining to any aspect of participation does not permit us to make firm conclusions regarding the relative strength of the different aspects of participation. It is our belief, supported by the theoretical research on positive settings for youth (e.g., Eccles and Gootman 2002), that the complex construct of participation is best viewed as one component of intersecting influences affecting the association between afterschool programs and developmental outcomes. Existing evidence does not support favoring one aspect of participation over another. It is too early in the development of the knowledge base on afterschool programs to ascertain if a specific aspect of participation can be responsible for developmental gains. Better methods of collecting data on each aspect of participation will allow future research to attempt to disentangle how youth can benefit from greater amounts of time in a quality afterschool program that offers a variety of engaging activities.

Many programs already collect daily attendance information, either via paper-and-pencil records, web-based databases, or swipe cards with a Management Information System (MIS) processing and storing the information (Fiester 2004). The information includes daily attendance, and ideally the exact time youth arrive and leave. Time spent at activities within the program is increasingly being collected as well (Arbreton et al. 2008a; Fiester 2004; Walker and Arbreton 2004). The importance of considering participants’ engagement is growing, helped by recent efforts to measure and improve the quality of afterschool programs (Yohalem and Wilson-Ahlstrom 2007). We are hopeful that these trends, coupled with the growing awareness of the complex nature of participation, will increase the number of empirical investigations on aspects of participation.

Footnotes
1

Although meta-analysis offers a quantitative alternative to narrative reviews, methods for calculating effect sizes from observational studies, while possible using inferential test statistics, are controversial because of the unresolved issues of causality (Rosenthal and DiMatteo 2001; Glass 2000). Since observational studies are best suited to answering the main question posed in this review—do participants benefit from greater amounts of participation in an afterschool program—we opted against conducting a meta-analysis.

 
2

Higher amounts of participation were defined by the researchers within each study. For example, in categorical comparisons, higher intensity was defined as 36 or more days in one study (Chappel 2004) but as the top third of attenders in another study (Dynarski et al. 2003). In studies analyzing participation as a continuous variable, higher intensity was not defined in terms of a specific number of days of attendance, but instead as more days than those with lower intensity.

 

Acknowledgments

This research was supported by a grant from the William T. Grant Foundation. We would like to thank Christina Borbely and Yvonne McNair for their help in conducting the literature search. We would also like to thank the helpful comments of the editor and anonymous reviewers.

Copyright information

© Society for Community Research and Action 2010