American Journal of Community Psychology

, Volume 45, Issue 3, pp 358–369

Quality at the Point of Service: Profiles of Practice in After-School Settings

Authors

    • Weikart Center for Youth Program Quality
  • Stephen C. Peck
    • University of Michigan
  • Anne-Sophie Denault
    • University of Michigan
  • Juliane Blazevski
    • Weikart Center for Youth Program Quality
  • Tom Akiva
    • University of Michigan
Original Paper

DOI: 10.1007/s10464-010-9315-z

Cite this article as:
Smith, C., Peck, S.C., Denault, A. et al. Am J Community Psychol (2010) 45: 358. doi:10.1007/s10464-010-9315-z

Abstract

A unique observational data set was used to explore quality at the point of service in after-school programs. Staff practices in after-school settings were represented on a series of unidimensional scales closely indexed to staff behavior. In order to account for heterogeneity of staff performances, pattern-centered methods were used to construct profiles of common staff practices. Results revealed six pedagogy profiles that were classified in terms of three broad types of performances delivered by after-school staff: (1) positive youth development, (2) staff-centered, and (3) low-quality. Staff membership in these profiles was not related to youth-staff ratio. However, results revealed significant differences between the profiles on the content of the offering and the age of youth in the setting.

Keywords

After schoolYouth developmentCluster analysisQualityQuality assessment

Introduction

A growing body of evidence suggests that high-quality after-school programs can provide developmentally powerful contexts for youth (e.g., Bodilly and Beckett 2005; Eccles and Gootman 2002; Larson 2000). However, the experimental evidence also suggests that whereas some program experiences produce positive outcomes, others do not (Durlak and Weisberg 2007; Lauer et al. 2006). How can we raise the quality of programs to increase the probability of positive outcomes? What are the necessary levels of program quality to assure return on investments? Unfortunately, there is little evidence about how and for whom after-school programs work.

Although many studies have examined out-of-school time activity involvement at the individual level (e.g., Bartko and Eccles 2003; Mahoney 2000; Peck et al. 2008; Shanahan and Flaherty 2001), few of these studies have examined directly the setting-level features hypothesized to mediate between participation and outcomes (cf. Eccles and Gootman 2002; Fletcher et al. 2003; Hansen et al. 2003; Tseng and Seidman 2007). A recent summary of research on organized activity settings concluded that research on relations between contextual features and youth outcomes is rare and that even less is known about how contextual features interact to produce developmental change (Mahoney et al. 2005). However, a consensus is emerging on both the features that constitute high-quality after-school programming (Eccles and Gootman 2002; Durlak and Weisberg 2007) and a set of measures to assess setting-level features in after-school programs (Granger et al. 2007; Yohalem and Wilson-Ahlstrom 2007).

Staff practices and their combinations are arguably one the most important features of education and human service settings (Blyth 2006; Pianta 2008). In this paper, we advance theory and research focused specifically on the quality of staff performances with youth. First, we develop a theoretical rationale for our interest in staff practices delivered at the point of service and identify a specific type of after-school micro-setting as an appropriate frame for sampling these practices. Next, we present empirical evidence of construct validity for an observational quality measure calibrated to these after-school micro-settings. Finally, we define a small number of widely used staff pedagogy profiles that have relevance for both research and policy. As a source of validation for the pedagogy profiles, we test hypotheses regarding relations between the profiles and three additional program features. Youth-adult ratio and offering content represent key structural features of after-school programs whereas grade-level represents a marker for developmental characteristics of youth.

Defining Quality at the Point of Service

Although program quality can be defined and operationalized in many ways, quality at the point of service focuses on the coexistence and correspondence between staff practices and youth experience that is likely to produce positive developmental change. For the after-school field, our conceptual definition consists of several elements. First, key developmental experiences must be available in a setting, minimally including (a) positive relationships with adults that provide a context for (b) engagement with content that (c) becomes increasingly complex over time. Further, these key experiences should include initiation and response from both youth and their environments (e.g., people, materials, ideas) to optimally stimulate “attention, exploration, manipulations, elaboration, and imagination” (Bronfenbrenner 1999, p. 6). Although these core dynamics are described in terms of youth experience, the intentional structures and processes necessary to initiate and sustain these experiences are, by definition, the products of staff practice.

A second element of definition involves where access to key developmental experiences (e.g., relationship, engaging content, and increasing complexity over time) might occur in an after-school setting. We believe that these developmentally dynamic youth experiences are most frequently available in micro-settings where consistent groupings of adults and youth meet over multiple sessions for the same learning purpose. We refer to these micro-settings as program offerings (e.g., an 8th grade poetry workshop that meets for a set time each week after school). Importantly, our conception of quality at the point of service is content-independent in the sense that no explicit academic or other content areas are mentioned as integral elements of high-quality practice. Rather, staff practices provide a foundation for scaffolding many types of content in ways that optimize youth experience.

With a working definition of key developmental experiences and relevant micro-settings where they are most likely to occur, we are left to specify how staff practices map onto developmental experience for youth. This effort is explicitly about youth work pedagogy as a coherent and purposeful approach to child development in group learning contexts and can be contrasted with “kitchen sink” approaches to defining after-school quality that include a wide and atheoretical variety of best practices. In this study we describe point-of-service quality in terms of the extent to which staff behaviors create opportunities for developmentally powerful youth experiences. Although we do not measure or analyze youth-level outcome data, our measures of staff-level behavior were designed to correspond to three general domains of youth experience: emotional supports; structured interaction with people, ideas, and materials; and cognitive engagement with environmental contents and processes. These domains (a) reflect widely shared ideas about developmentally important learning experiences (cf. Eccles and Midgley 1989; Maslow 1943; Marzano 2001; Bransford et al. 1999), (b) refer to both youth experience and staff behavior, and (c) are usefully described with a hierarchal metaphor: High quality emotional and instructional supports provide a foundation for deeper forms of youth engagement with program content. These domains are useful because they represent staff performance and youth experience together in a way that is focused on pedagogy designed to create youth experiences that will have developmental consequences.

A supportive environment provides a sense of inclusion and belonging for youth through program traditions that cultivate inclusion and youth ownership. Youth in supportive environments are welcomed with warmth and learn well-defined conflict resolutions methods. Youth experience a positive social climate by, for example, seeing that staff care about the ideas and feelings of all youth. We operationalize quality in this domain with measures of welcoming and inclusion practices used by program staff.

Structured interaction supports youths’ experiential learning with both concrete materials and abstract ideas. Youth experience cooperative learning, collaboration with staff and other youth, and divergent questions that provoke substantive verbal exchange. Interactive experiences are contrasted with those available during the “traditional” school-day: full-group activities, individual seat work, few hands-on learning props, and few discussions. We operationalize quality in this domain with measures of active learning and grouping practices used by program staff.

Opportunities for purposeful engagement involve higher-order decision-making and evaluation experiences that extend over tasks and time. This model of engagement presses youth to use self-reflection to plan, set goals, and make choices about program content and processes. Environments with high levels of choice, planning, and reflection prioritize deep thinking and promote a shared construction of processes and purposes. We operationalize quality in this domain with measures of choice, planning, and reflection practices used by program staff.

Although there is evidence that the observed quality of staff performances in after-school programs is associated with youth development and learning outcomes (e.g., INCRE and NIOST 2005; Russell and Reisner 2005), more evidence is needed. In several validation studies for the observational measures used in this study, the quality of observed staff performance was positively related to: student self reports of personal growth, community giveback, and decision-making (Smith and Hohmann 2005); after-school attendance, school-day reading scores, and school-day behavior (Blazevski and Smith 2007); and youth reports of challenge and interest in the programs (Smith et al. 2008).

Pattern-Centered Approaches and Pedagogies

Pattern-centered theory and methods are ideally suited to address issues of behavioral complexity and holistic representation because they have been developed for examining hypotheses involving multilevel data and functionally interconnected variables that combine differently within different people both within and across time (Bergman et al. 2003; Peck 2007; von Eye and Bogat 2006). For example, pattern-centered theory indicates that after-school program staff function as integrated wholes, and pattern-centered methods allow us to treat patterns of values on staff practice variables as integrated wholes (i.e., profiles). Treating individual staff members (instead of variables) as the unit of analysis allows us to identify a small number of relatively homogeneous subgroups characterized by distinct pedagogy profiles. Using this combination of theory and method allows us to better understand how individual staff members translate their training and background into specific styles of behavioral interaction with youth during program offerings. Further, the identification of a small number of pedagogy profiles, and the assignment of each staff member to one of these profiles, allows for subsequent analysis (not reported here) of how different practice styles relate to, for example, different program management styles and different experiences of youth within specific program offerings.

Pattern-centered approaches almost always involve the integration of variable- and pattern-centered methods (cf. Cairns and Rodkin 1998; Peck et al. 2008). One of our goals is to demonstrate the effectiveness of this integration for both researchers and policymakers. By using variable-centered methods to determine dimensions relevant to staff performance in after-school programs, and then pattern-centered methods to identify homogenous subgroups, we move the unit of analysis from unidimensional measures (or aggregated global measures) to qualitatively distinct styles of practice. Such pedagogy profiles highlight specific patterns of staff behavior as targets for policy change (e.g., training programs). For example, if a particular pedagogy profile is associated with especially poor youth outcomes, then substantial cost savings and training effectiveness may be achieved by designing intervention strategies that are particular to staff characterized by this low-quality practice style.

Research Questions & Hypotheses

Although several studies have examined profiles of youth out-of-school activity involvement (e.g., Peck et al. 2008) and profiles of staff practices in early education (e.g., LoCasale-Crouch et al. 2007), few if any studies have examined profiles of staff practices in youth-serving programs. Here we address two primary research questions: First, within after-school staff, how are practice sets bundled together as coherent pedagogies? Second, are specific pedagogies associated with key variables like youth-staff ratio, offering content, or age of the youth in accordance with theory and prior research? Given the constraints imposed by the socio-cultural context in which after-school programs are embedded (cf. Magnusson 2003), we expect to find a relatively small number of pedagogy profiles. Given the dominance of traditional staff-centered approaches to instruction in schools (Hamre and Pianta 2005) and the short supply of youth leadership opportunities in after-school settings (Smith et al. 2006), we expect that one of the primary differences between the profiles will be the frequency with which staff use the higher-order instructional practices of choice, planning, and reflection. Further, we expect that some of the pedagogy profiles will be characterized by high levels of staff support (welcoming atmosphere, inclusion practices) and low levels of instructional support (active learning, grouping strategies) reflecting profile patterns found in early education settings (LoCasale-Crouch et al. 2007).

Several relationships with other setting characteristics are also anticipated as validation evidence for the pedagogy profiles. First, given age-graded changes in the salience of autonomy needs (Eccles et al. 1997), we expect pedagogy profiles marked by higher levels of support for planning, choice, and reflection to be more evident for older than younger after-school program participants. Second, because arts and enrichment content is often selected for after-school programming precisely because of flexibility for exploration and expression, we expect pedagogy profiles associated with these offerings to be marked by higher levels of choice. Finally, we do not expect pedagogy profiles to vary systematically with the youth-staff ratio. This unconventional expectation comes from our field experiences and recent early education research that presents mixed evidence about relationships between ratio and the observed quality of instructional performances (Karoly et al. 2008; Pianta et al. 2005).

Method

Sample

The sample includes observational data for 599 unique after-school offerings and offering leaders nested within 165 different after-school organizations in six states. In organizations with multiple offerings, site directors were asked to select staff that spent the most face-time with youth. We used data from nine studies conducted between 2005 and 2007 as part of the Youth PQA Validation Study (Smith and Hohmann 2005) and several contract evaluations. This sample was drawn from a universe of organizations that delivered year-round programming, had full-time administrators, and could produce a weekly schedule of offerings. Approximately 15% of the program offerings observed were nationally affiliated (e.g., Boys and Girls Club, Campfire USA, YMCA) and 26% were twenty-first Century Community Learning Centers. Approximately 60% of the offerings were school-based and 40% were community-based. The offerings’ content were primarily arts/enrichment and academics although homework, life skills, sports, informal time, and technology were also present. On average, program sessions involved approximately 15 youth with one to two staff members present.

Nearly all offerings involved youth in grades 4 through 12 and, for the 3,362 youth surveyed across 350 of the offerings, 58% were female. Approximately 74% of the offerings were drawn from organizations located in large central cites. For the two studies with information on staff education level, 41% of the observed staff had a high school diploma or less in one study (160 offerings), and 70% had a high school diploma or less in the other study (29 offerings).

Procedures

The Youth PQA, Form A was developed to assess the quality of staff practices in after-school program offerings (Smith and Hohmann 2005). Completion of the instrument requires observation during one entire session for an offering, usually 1–2 h. A running record of events that occur during the offering, centered on the actions of staff, is generated by an outside rater. After the observation period, the rater uses the written record to score items on a 3-level scale where “1” indicates that the staff practice did not occur, “3” indicates that the practice occurred informally or for only some youth, and “5” indicates that the practice occurred formally for nearly all youth (items not rated are coded “NR”). Training required at least 75% perfect agreement at the item level with a set of “gold standard” scores when using video or an expert rater during a paired observation (Blazevski and Smith 2007).

Two findings from prior research support use of an observation for a single session of an offering. First, test–retest coefficients (separated by at least 2 weeks) for all scales listed in Table 1 (except inclusion practices) for observations by the same rater during two sessions of the same offering (N = 26) ranged between .63 and .89 suggesting that the quality of staff performances are stable over short intervals. Similar patterns of short-term stability have been demonstrated with other observation-based measures of staff practices (Yohalem and Wilson-Ahlstrom 2007; NICHD 2005). Second, prior research also reveals that staff scores vary substantially across staff within the same organization, suggesting that individual staff performances are a meaningful unit for quality assessment. Similar to research on teacher performance in school settings (Nye et al. 2004), intraclass correlations for unconditional HLM models using various samples of Youth PQA data suggest that between 20 and 60% of quality score variance occurs within organizations (Smith et al. 2006).
Table 1

Scale and item-level descriptive statistics for the seven youth PQA scales

 

M

SD

% Scoring 1

Welcoming atmosphere (α = .82)

4.48

.88

 

 1. Staff use a warm tone of voice and respectful language

4.48

.93

1

 2. Staff smile, use friendly gestures, and make eye contact

4.49

.97

2

Inclusion practices (α = .62)

3.66

1.19

 

 3. Inclusive rather than exclusive climate among youth

3.60

1.42

13

 4. Evidence of shared traditions or youth-owned climate

3.73

1.28

8

Active learning (basic best practices) (α = .69)

3.67

.95

 

 5. Staff use active learning tasks (e.g., create/reformulate materials or ideas)

4.01

1.38

11

 6. Staff use activities that balance concrete experiences and abstract concepts

3.53

1.35

13

 7. Staff encourage youth to try new skills/improve

3.63

1.62

22

 8. Staff are actively involved with youth

4.43

1.08

4

 9. Staff use open-ended questions throughout the activity

2.73

1.62

40

Support for group participation (α = .67)

2.55

1.45

 

 10. Activities carried out in different groupings

2.49

1.56

44

 11. Groups have purpose/goal and members cooperate to accomplish it

2.59

1.78

51

Opportunities to make choices (α = .66)

2.87

1.46

 

 12. Opportunities to make content choices

2.90

1.69

38

 13. Opportunities to make process choices

2.85

1.69

40

Opportunities for planning (α = .83)

2.24

1.41

 

 14. Opportunities to make plans for projects and activities

2.38

1.57

51

 15. Opportunities to use multiple planning strategies

1.93

1.29

44

Opportunities to reflect (α = .70)

2.35

1.28

 

 16. Opportunities to reflect on work in progress or completed work

2.31

1.64

57

 17. Opportunities to reflect on work in multiple ways

2.16

1.38

53

 18. Opportunities to make presentations to the whole group

2.59

1.79

52

“% Scoring 1” in column three refers to the percentage of 599 observed staff who received a score of 1 on the Youth PQA for that item, indicating that the practice named in the item did not occur

Measures

The staff performance scales selected for this study capture the extent to which after-school staff provide: (a) a welcoming atmosphere, (b) inclusion practices, (c) support for active learning, (d) support for group participation, (e) opportunities for youth planning, (f) opportunities for youth to make choices, and (g) opportunities for youth to reflect (see Table 1 for reliability coefficients). We also used categorical variables representing youth-staff ratio (Ratio), offering grade level (Grade), and offering content (Content). Ratio was coded as 1 = 1 staff to 8 or fewer youth, 2 = 1 staff to 9–16 youth, and 3 = 1 staff to 17 or more youth. Grade level was coded as 1 = mostly elementary students, 2 = mostly middle school students, 3 = mostly high school students. Content was coded as 1 = homework & tutoring, 2 = academics, 3 = arts & enrichment, 4 = life skills, character, health, 5 = sports, and 7 = not applicable to categories. Content codes were developed from offering names, for example, offerings named “lyrists lounge,” “acting class—arts and crafts,” “peace camp,” and “peer rap” were all coded into the enrichment/arts category.

Data Analytic Strategy

Our analysis strategy consisted of three steps. First, we used confirmatory factor analysis (CFA) to test the theoretically derived structure of our staff performance data at the scale level. Scale level confirmation was critical for these analyses because higher-order factors are not well indexed to specific aspects of staff practice and are less useful in the construction of profiles. Next, we used cluster analysis to create profiles of staff practices based on the scale scores derived in step one. Finally, we examined the validity of the profiles by examining how three after-school setting characteristics (Ratio, Grade, and Content) vary across profiles.

Results

Scale and item-level descriptive statistics for seven staff practice scales are shown in Table 1 (cf. Smith and Hohmann 2005). In general, mean scores decrease and standard deviations increase moving from the top to the bottom of Table 1, following a pattern established in other samples of data collected using the Youth PQA (Smith et al. 2006, 2008). For example, whereas staff warmth and positive body language (items 1 & 2) were common, in over 50% of all offerings the staff person did not provide an opportunity for youth to reflect on the session’s activities or products (items 16, 17, & 18).

Confirmatory Factor Analysis

The proposed factor structure for our measure of staff practice was evaluated with CFA using LISREL 8.53 (Jöreskog and Sörbom 1996). Parameters were estimated using maximum likelihood and the full-information maximum likelihood method of handling missing data (percentage of missing values = 3%). Table 2 summarizes the factor patterns emerging from the seven-factor CFA model of program quality as well as correlations between the resulting scales. Results provided some support for our proposed seven-factor model but also indicated that there is room for improvement: χ2 (df = 114, N = 599) = 343.04, p < .000; RMSEA = .058, 90% CI (.051; .065).
Table 2

Standardized maximum likelihood estimates and correlation coefficient for the measurement model (N = 599)

Item

Welcoming atmosphere

Inclusion practices

Active learning

Group participation

Opp. for choice

Opp. for planning

Opp. for reflection

1

.77

      

2

.89

      

3

 

.62

     

4

 

.77

     

5

  

.51

    

6

  

.62

    

7

  

.62

    

8

  

.49

    

9

  

.55

    

10

   

.76

   

11

   

.67

   

12

    

.70

  

13

    

.70

  

14

     

.86

 

15

     

.83

 

16

      

.73

17

      

.78

18

      

.56

Welcoming

      

Inclusion

.27*

     

Active Lrn

.35*

.36*

    

Grouping

.14*

.23*

.29*

   

Choices

.17*

.23*

.32*

.15*

  

Planning

.18*

.24*

.37*

.13*

.43*

 

Reflect

.20*

.21*

.52*

.21*

.43*

.29*

Completely standardized CFA solution. Chi-square = 343.04, p < .000, df = 114; RMSEA = .058, 90% CI (.051; .065), p value test of close fit: p = .03. The lower part of the table includes Pearson correlation coefficients. * p < .01

Overall, the scale reliabilities, scale correlation coefficients, factor loadings, and fit indices indicate that the seven-factor model is an adequate representation of the data. These are generic practice sets that can be discerned across a wide variety of after-school settings. Accordingly, these seven practice sets (factors) served as the basis for subsequent investigation of pedagogy profiles within after-school programs.

Cluster Analysis of Practice Sets

After creating the seven “practice set” scales, we subjected these unstandardized variables to cluster analyses using the Sleipner (version 2.1) statistical package for pattern-oriented analyses (Bergman and El-Khouri 2002; Bergman et al. 2003). Four modules were used in the analysis: Impute, Residue, Cluster, and Relocate. We began by using the Impute module to assign valid variable values to 58 after-school offerings that were missing data on no more than three of the seven practice-set variables (the imputed values were taken from the staff person with the closest matching profile). Two cases were deleted after this analysis (one with too much missing data; one with no matching twin). After imputing the data, the Residue module was used to remove seven multivariate outliers (i.e., staff whose pattern of values on the practice-set variables matched no more than two other staff).

We then used the Cluster module to obtain initial cluster solutions ranging from 2 to 20 groups (using Ward’s method and squared Euclidian distances as the dissimilarity measure). For each level of complexity, an index of the increase in the error sums of squares is produced (ESS). The explained and increased ESS from the 2 to 20-cluster solutions can then be plotted (see Fig. 1) to determine the statistically justifiable upper and lower number of cluster groups that provide unique information (Bergman et al. 2003). As shown in Fig. 1, the results provided statistical justification for selecting as few as four or as many as 12 cluster groups. We selected the 6-cluster solution because it was relatively parsimonious and contained sufficient variability in instructional styles (e.g., profiles were differentiated by the presence or absence of opportunities for reflection and grouping practices). Finally, we used the Relocate module to conduct a k-means relocation analysis of the 6-cluster Ward’s solution. This procedure re-assigned 176 staff to cluster groups that best matched their individual profile, thereby correcting for premature classification by the hierarchical algorithm and further increasing within-group homogeneity. The relocation procedure resulted in a point-biserial correlation change from .37 to .42 and a total explained ESS change from 46.58 to 52.68. The centroids, standard deviations, and homogeneity coefficients for each cluster group are shown in Table 3.
https://static-content.springer.com/image/art%3A10.1007%2Fs10464-010-9315-z/MediaObjects/10464_2010_9315_Fig1_HTML.gif
Fig. 1

Change in the explained sum of squares for the 2–20-cluster solutions. Note: IESS increase in error sum of squares; EESS explained error sum of squares

Table 3

Centroids, standard deviations, and homogeneity coefficients for the cluster groups of practices (n = 590)

Cluster

I

II

III

IV

V

VI

Label

PYD I

PYD II

Staff Cent I

Staff Cent II

Low Qual I

Low Qual II

n = 97

n = 69

n = 132

n = 99

n = 95

n = 98

Homogeneity coefficient

1.46

1.66

1.50

1.50

1.62

1.29

Practices

 Welcoming atmosphere

4.78 (.50)

4.78 (.59)

4.65 (.68)

4.53 (.86)

4.56 (.81)

3.78 (1.11)

 Inclusion practices

4.24 (.88)

4.12 (.99)

3.98 (.95)

4.03 (.92)

3.28 (1.11)

2.26 (.98)

 Active learning

4.41 (.65)

4.34 (.65)

3.78 (.72)

3.83 (.73)

3.32 (.85)

2.54 (.72)

 Support for group participation

4.16 (.69)

1.46 (.68)

4.10 (.73)

1.31 (.55)

1.81 (.90)

1.56 (.83)

 Opportunities for choices

4.38 (.87)

3.43 (1.32)

2.18 (1.10)

1.95 (.87)

4.17 (.75)

1.56 (.80)

 Opportunities for planning

3.77 (1.19)

4.00 (.92)

1.55 (.85)

1.54 (.84)

2.21 (1.10)

1.08 (.34)

 Opportunities to reflect

3.63 (1.01)

3.74 (.98)

1.90 (.95)

2.43 (1.17)

1.59 (.69)

1.36 (.60)

PYD positive youth development; Staff Cent staff-centered; Low Qual low quality. Lower homogeneity coefficients indicate more homogeneous subgroups (the homogeneity coefficient corresponding to the sample as a whole is 3.13)

The six clusters are independently interpretable but can also be collapsed into three related pairs: two Positive Youth Development pedagogy profiles, two Staff Centered pedagogy profiles, and two Low Quality pedagogy profiles. Positive Youth Development pedagogy includes the PYD I and PYD II profiles with high levels of staff practices in nearly all areas. These profiles represent 28% of all offerings in the sample, providing youth with a supportive environment (welcoming atmosphere, inclusion practices); active learning (mix of hands-on and abstract materials and ideas, encouragement for skill practice, divergent questions, staff involvement); and opportunities for engagement through planning, goal setting, and reflection. Based on visual inspection, only support for group participation (activities carried out in different groupings, group members’ work toward purposes/goals) distinguishes the two PYD profiles.

Staff Centered pedagogy profiles represent 39% of the sample. These profiles include offerings with youth workers who provide a supportive environment for learning and active learning but give youth few opportunities for choice, planning, and reflection. Again, support for group participation is the predominant difference observed between the two Staff Centered profiles. The term “staff centered” is used because these profiles lack practices that invite youth to make their own choices and plans.

The Low Quality profiles constitute 33% of the sample. The Low Quality I profile is characterized by lower levels of basic support and active learning than the other four pedagogies, low levels of grouping practices, and fewer opportunities for youth to make plans and reflect. In addition to having the lowest scores on every indicator, the Low Quality II profile differs from the Low Quality I profile by having lower levels of welcoming atmosphere and opportunities for choice.

Differences Between the Profiles with Respect to Ratio, Grade-Level, and Content Area

In order to explore the distribution of offerings by Ratio, Grade, and Content, we estimated the standardized differences between expected and observed cell counts for distributions of pedagogy profiles within each level of a program characteristic (see Table 4). In our sample, 22% of offerings had youth-staff ratios of eight youth or fewer to one staff, 16% had ratios of between 9 and 16 youth to one staff, and 5% had ratios of 17 or more youth to one staff. Fifty-seven percent of the offerings were missing ratio information (accurate ratios cannot always be determined when some support staff float between offerings at a particular program site). Our results revealed no significant relation between Ratio and Pedagogy Profile (χ2(N = 252) = 7.03, ns).
Table 4

Counts and adjusted standardized residuals for crosstabulations of staff pedagogy profiles by youth grade levels and offering content areas

 

Youth grade level

Offering content area

Elementary

Middle

High

Total

Hwk/tut

Acad

Arts/enr

Lifesk

Sports

Total

1. PYD I

16 (−3.2)**

37 (.9)

22 (3.0)**

75

5 (−1.4)

8 (−1.6)

53 (2.3)*

6 (−.8)

9 (.5)

81

2. PYD II

12 (−1.9)

22 (.2)

14 (2.3)*

48

4 (−1.3)

6 (−1.6)

42 (1.7)

9 (1.2)

5 (−.6)

66

3. Staff Cent I

54 (5.2)***

27 (−2.9)*

6 (−2.9)**

87

11 (.0)

24 (2.3)

38 (−3.9)***

13 (1.1)

17 (2.6)*

103

4. Staff Cent II

21 (−.8)

31 (.8)

11 (.0)

63

16 (2.3)*

17 (.7)

41 (−2.0)

13 (1.6)

5 (−1.5)

92

5. Low Qual I

30 (.5)

33 (.0)

11 (−.6)

74

9 (.2)

7 (−1.9)

56 (3.3)***

2 (−2.3)*

5 (−1.1)

79

6. Low Qual II

25 (−.5)

37 (1.4)

9 (−1.2)

71

9 (.0)

19 (1.7)

43 (−.7)

6 (−.9)

8 (−.1)

85

Total

158

187

73

418

54

81

273

49

49

506

Adjusted standardized residuals are in brackets. These values can be interpreted as z-scores (absolute values above 1.96, 2.58, and 3.29 are significant at the two-tailed .05, .01, and .001 levels, respectively). Positive residuals indicate that the observed counts are higher than expected by chance; negative residuals indicate that the observed counts are lower than expected by chance. Hwk/tut homework/tutoring; Acad academic; Arts/enr arts or enrichment; Lifesk life skills

p < .05, ** p < .01, *** p < .001

With respect to grade-level, the sample included 27% elementary school (primarily grades 4 or 5), 32% middle school, and 12% high school. Twenty-nine percent of the offering ratings were missing this information (due to confusion regarding how to score mixed age groups). Results revealed significant differences in the distribution of profiles in relation to Grade (χ2(N = 418) = 43.38, p < .001). For high school aged youth, the PYD I and PYD II profiles were both more prevalent while Staff Centered I was less prevalent. For middle school aged youth, a pattern similar to the older youth was present with PYD II being more prevalent and Staff Centered I less prevalent. Elementary school offerings demonstrated a contrasting pattern with Staff Centered I being more prevalent and both PYD profiles being less prevalent. None of the age groups were significantly more likely than expected by chance to experience either of the Low Quality profiles.

With respect to content area, 46% of offerings were Arts/Enrichment (e.g., theater, chess club), 14% were Academics (e.g., science club, math adventures), 9% were Homework/Tutoring, 8% were Life Skills (e.g., conflict resolution, healthy eating), and 8% were Sports; 7% of offerings did not fit neatly into any of the categories described above, and 7% of offerings were missing content information (both counted as missing in subsequent analyses). Results revealed significant relations between Content and Pedagogy Profiles (χ2(N = 506) = 50.79, p < .001). Staff in arts/enrichment programs were more likely to be characterized by the PYD I, PYD II, or Low Quality I pedagogies and less likely than expected by chance to use either of the Staff Centered approaches. Homework/Tutoring and Academics followed an inverse pattern, with each being more likely to have staff using one of the Staff-Centered approaches and with all PYD coefficients carrying a negative sign (non-significant). Sports offerings were more likely to have staff using the Staff Centered I approach. Finally, Life Skills offerings exhibited no relationships to any of the PYD or Staff Centered approaches but were significantly less likely than expected by chance to have staff providing a Low Quality I approach.

Discussion

Staff practices were measured and represented on a series of unidimensional scales closely indexed to specific staff practices. Factor analysis supported the use of seven dimensions. Pattern-centered methods were used to reveal pedagogy profiles representing holistic styles of staff performance during after-school offerings. Three broad classes of pedagogy profiles emerged from these analyses: (a) Positive Youth Development, (b) Staff Centered, and (c) Low Quality. As hypothesized, pedagogy profiles were related to the age of youth in the setting, the content of the offering, but not to youth-staff ratio.

Our derivation of six pedagogy profiles was driven by both the empirical evidence (see Fig. 1) and theoretical parsimony. For both the PYD I & II and Staff Centered I & II profiles, only the presence or absence of the grouping practice set differentiates each pairing. The PYD and Staff Centered profiles are clearly differentiated by the presence or absence of choice, planning, and reflection, a suite of practice sets focused on autonomy and higher-order metacognitive functions. The Low Quality I & II profiles were not as parallel but reflect a general absence of practice sets and may reflect the absence of any intentional pedagogy at all.

Empirical Profiles of Quality at the Point of Service

Across 599 after-school offerings, staff tended to provide emotional warmth and opportunities for active learning. However, in roughly half of these offerings, access to developmentally important experiences such as small group work, project planning, reflection, or group presentation were not available. In roughly 40% of all settings, youth were not provided with opportunities to make choices about either the processes or content of offerings. Similar sample-level patterns have been found in the few large rigorous observational studies that exist for early childhood care (Peisner-Feinberg et al. 2000) and elementary schooling (Pianta 2008). Our findings suggest that many after-school settings have not advanced far beyond a “child care” model where safety and fun are part of the program model but where motivation and deeper cognitive engagement with content is lacking.

Whereas studies from early education (e.g., LoCasale-Crouch et al. 2007) suggest two primary factors that represent staff practices (i.e., warmth and instructional support), our data suggest the presence of a third developmentally relevant domain for older youth: choice, planning, and opportunities to evaluate and provide feedback about program processes and products. The engagement domain was in greater supply for older youth indicating, we suspect, both attention to the developmental needs of older youth and the fact that a greater proportion of older youth likely make their own decisions about attendance.

The PYD I and PYD II profiles suggest the presence of developmental intentionality on the part of staff; that is, an awareness that delivery of key developmental and learning experiences is a focal purpose for an after-school program (Walker et al. 2005). Further, in these settings, participants are likely engaged in youth-led or project-based learning (Larson et al. 2005) to a greater extent than settings characterized by the Staff Centered or Low Quality approaches. While the PYD profiles were present for all age groups and in all content areas, they were most prevalent in the arts/enrichment content area, the most prevalent type of content in the after-school field. It is possible that the arts and enrichment vision for after-school may provide a platform for the most developmentally intentional staff. However, offerings focused on arts/enrichment content were concentrated in both high and low quality profiles with the highest levels of choice (i.e., PYD I & Low Quality I) suggesting that the causal pathway may flow both ways. Arts and enrichment offerings are more conducive to a simple “hands off” choice-based pedagogy and attract staff with skills for intentionally leveraging this content-driven opportunity into delivery of planning and reflection experiences. It is worth noting that the arts/enrichment content area is associated in a negative direction (although non-significant) with all of the profiles with very low levels of the choice practices set (i.e., Staff Centered I & II and Low Quality II).

Contrasted with the PYD profiles that focus on arts/enrichment programming for older youth, the Staff Centered profiles present a second pedagogy more frequently adopted (a) for delivery of academics, homework, and sports and (b) with elementary aged children. It may be important to see these profiles as not necessarily of lower quality than the PYD profiles but perhaps better suited for different uses. For example, the Staff Centered profiles parallel approaches used in many school-day classrooms that demonstrate qualities of higher warmth, lower autonomy, and infrequent opportunities for higher-order cognitive experiences like planning and reflection (Hamre and Pianta 2005). However, the desire for after-school settings to provide an alternative to school day experience may cause dissatisfaction with the prevalence of the Staff Centered pedagogy, particularly for adolescents.

Finally, the Low Quality pedagogy includes two profiles that do not represent well-designed approaches to after-school instruction. Although youth frequently make choices in these settings, this may be due more to an excessive lack of structure. This “hands off” approach may overlap with approaches identified in the early childhood literature, including custodial care and Laissez faire (Weikart et al. 1978). The Low Quality II profile shows low levels of staff practices in all areas, perhaps indicating disorganization or a state of disconnectedness between staff and youth.

Point of Service Theory

The after-school (and the larger out-of-school time) field is progressing toward more broadly applicable models and measures of quality at the point of service. This study suggests two ways to support this progress. First, definitions of quality should incorporate strong theory about how, when, and at what level of intensity staff practices are likely to affect youth development. Otherwise, we run the risk of measuring things that do not matter. Second, quality measures should exhibit dimensionality that supports indexing of measures to practices. Otherwise, we run the risk of producing data that do not support either understanding or action.

The theory of point-of-service quality rests upon a few empirically grounded or testable assumptions. Like others, we argue that staff practices are the setting feature with the greatest potential to add value in terms of gains to youth development and learning (Granger et al. 2007; Tseng and Seidman 2007). Although structural features such as staff education and pay, program accreditation/licensure, and youth-staff ratio have been seen as dominant levers of change in the past, a growing body of evidence suggests that these issues may be less important than staff practices in the prediction of child outcomes (Early et al. 2007; Mashburn et al. 2008).

We also suggest that individual staff are a naturally occurring unit of quality and that the best frame from which to sample individual staff practice is a micro-setting defined by continuity of participants and purposes. While youth may attend an after-school program every day, it is their task- and project-focused relationships that are likely to support experiences of interaction and engagement. Further, we suggest that a hierarchal metaphor, paralleling Maslow’s (1943) hierarchy of needs, is a useful way to think about point-of-service quality. Staff practices that address basic physiological, emotional, and esteem needs are likely to provide a context where youth attention is available for direction toward higher-order concerns. All of the pedagogy profiles that exhibited high scores on choice, planning, and reflection also had high scores on the “lower” practice sets having to do with support and interaction.

Finally, the three domains of quality at the point of service which we used to group the practice sets hint at rich areas of future research with implications for both measurement and modeling relationships between staff performance, youth experiences in the setting and, ultimately, youth outcomes. As an example, in Table 3 the grouping practices set did not co-occur with the active learning practice set, although these measures were listed as members in the same domain of youth experience focused on interaction with people, ideas, and materials. It is possible that these practice sets may indeed both produce similarly powerful youth experiences of active instruction but may not be related in the sense of forming a correlated higher-order factor. Two equally plausible interpretations follow. One hypothesis is that the practice sets are “formative,” not “reflective,” meaning that either can achieve the same effect but that the presence of one does not imply the presence of the other (Diamantopoulos and Siguaw 2006). Conversely, each practice set may be necessary for effective practice, but they often do not co-occur due to deficiencies of staff training or circumstance. In either case, it may be questionable to assume that the lack of strong correlation between these practices indicates that they are not properly considered within the same domain.

Limitations of the Study

There are several limitations to this study. First, although our sample approximates the mix of after-school setting types that may represent after-school institutions and workforces in the aggregate, we did not use a representative sample of after-school programs. A second weakness in this work is the absence of youth-level measures. Staff performances are influenced by youth behavior and background and these attributes are not explicitly reflected in any of the findings presented here. Further, this study reflects the challenges of conducting research in applied settings in that our validation analyses were produced despite large amounts of missing data for age and youth-adult ratio. Finally, the Youth PQA items represent a minor threat to the efficacy of the confirmatory analyses due to the fact that items do not name the same referent—the lead staff—in all cases.

Conclusions and Future Directions

For the after-school field, our findings suggest that for youth development programs to deliver on their promise and public investment, many after-school staff could use more intentional youth work pedagogies, building from relationships to interactions with people and materials and finally to higher order cognitive engagement with program content. If our profiles of staff practices reflect access to key developmental experiences, then the experiences of youth in many after-school programs represent missed opportunities. For example, 33% of staff in our sample failed to cultivate a sense of warmth and inclusion during the offerings that they led.

Despite these missed opportunities, 28% of staff in our study demonstrated use of an identifiable positive youth development pedagogy, representing a substantial professional skill base in a field frequently singled out for its high rates of transience and lack of professional norms. Further, almost all of the offerings sampled included welcoming and inclusive staff who delivered the basic characteristics of active learning and got involved with youth during the offering. These are clear signs of a youth work pedagogy that is intentionally designed to deliver key developmental experiences during staff led program offerings, the developmental crucible of the after-school field.

We hope this study will advance research agendas focused on measuring and modeling relationships between staff performance and other levels of action and experience in education and human service systems. Our current work is focused in both directions, seeking to test cross-level relationships between management practices and more effective staff performances at the point of service, while also working within the offering level to model relationships between staff performance and the parts and processes that constitute youth motivation and skill building.

Acknowledgments

This research was funded by grants from the William T. Grant Foundation and the Picower Foundation awarded to the David P. Weikart Center for Youth Program Quality. We would also like to thank the following people for their contributions to various parts of this research: Monica Jones, Linda Snedacar-Horne, Samantha Sugar, Kynisha Johnson, Joe Bertoletti, and Tom Devaney.

Copyright information

© Society for Community Research and Action 2010