Contents

Evaluation of the New York City Out-of-School Time Initiative: Implementation of Programs for High School Youth

Click here to download the full report:
Evaluation of the New York City Out-of-School Time Initiative: Implementation of Programs for High School Youth

Program enrollment. In the second year, DYCD more strictly enforced standards for program enrollment and participation, with a policy of retaining funds for programs that did not achieve established targets. On average, Option I programs successfully scaled up to meet and exceed their targeted enrollment levels. Option I programs had a total target enrollment of approximately 50,000 youth, based on the contracts awarded by DYCD. In practice, programs actually served more than 70,000 students from September 2006 through June 2007. Elementary-grades and middle-grades programs were especially successful at meeting or exceeding their enrollment targets (90 percent and 79 percent of programs, respectively), while high school programs fell slightly short of their goal (57 percent of programs reached the target). Overall, 80 percent of Option I programs met or exceeded their enrollment targets.

Program participation. Across all grade levels, Option I programs successfully increased their participant-level attendance over Year 1 levels, although elementary-grades programs continued to struggle to achieve attendance goals. As noted earlier, elementary-grades offered more program service hours and expected more hours of youth attendance; high school programs offered the fewest hours with the fewest hours of attendance expected. On average, elementary-grade Option I participants attended an average of 359 hours during the year, compared to the 432 hours they were expected to attend. This represented an average of 83 percent of targeted hours, which is an increase from the 72 percent of targeted hours attended by elementary-grades participants in the first year. Overall, middle-grades participants nearly achieved their targeted number of hours of participation: on average, middle-grades participants attended 213 hours of the 216 hours expected at the middle-grades level.1 This was a substantial increase over the 159 hours attended, on average, by middle-grades participants in Year 1. Finally, high school participants exceeded the targeted number of hours of participation, attending on average 105 hours in the second year of the initiative, 29 hours above their target of 76 hours and several hours higher than during the first year (97 hours).

Association between enrollment duration and participation. OST programs typically enroll participants on a first-come, first-serve basis. Returning participants are not necessarily guaranteed enrollment for a second year. However, evaluation data reveal especially high levels of program engagement by returning participants. On average, two-year participants attended their OST program more regularly than did participants who enrolled for the first time in Year 2. Two-year elementary-grades participants attended an average of 399 hours during the 2006-07 school year, compared to 359 hours for one-year participants (d=.20). Middle-grades participants enrolled for a second year attended 253 program hours in Year 2 (exceeding their target by 37 hours), compared to 213 hours for one-year participants (d=.26). Two-year high school participants also attended substantially more hours than did one-year participants (135 hours, compared to 105 hours; d=.29). (Neither these associations nor others presented in this report should be interpreted to imply causality, however.)

As part of its efforts to provide comprehensive out-of-school time services to families throughout New York City, DYCD also funded certain OST programs to provide services to youth during the summer of 2006. Participation in OST summer programming was associated with higher rates of participation during the 2006-07 school year. Students in elementary-grades programs who participated in OST services in the summer of 2006 and the following school year attended, on average, 441 hours in Year 2 (d=.40). Middle-grades summer participants attended an average of 294 hours in Year 2, on average (d=.52). However, programming was not offered in all programs, and summer enrollment was not guaranteed to school-year participants.

Associations between program features and program participation. The evaluation found significant associations between certain staffing patterns and attendance rates in Option I OST programs. In particular, programs that hired at least some young staff members had higher program attendance rates than those without these young staff members. This was true for college student staff members (66 percent, compared to 57 percent; d=.49) as well as high-school age staff (67 percent, compared with 61 percent; d=.35). In addition, programs with school-day paraprofessionals or aides on staff had higher attendance rates than those without (69 percent, compared with 61 percent; d=.41). One explanation for this pattern might be that the presence of these younger staff members in addition to more experienced educators appealed to youth, who could identify and forge connections with staff closer to their age. Another possibility is that lower wages for less experienced staff allowed programs to increase the number of staff employed in the program and thus increase the amount of staff attention to each program participant.

Year 2 program attendance was also positively associated with programs' efforts to engage families, supporting a finding of the importance of family outreach. In particular, programs with a parent liaison on staff had a higher attendance rate than programs without a parent liaison (68 percent, compared to 62 percent; d=.32). The intensity of program communication with parents was positively correlated with attendance rates (r=.29). Although each form of communication with parents was positively associated with program attendance rates, certain types of parent outreach occurring at least a few times a month were associated with particularly strong attendance effects: holding individual meetings with parents (d=.68), sending materials home to parents (d=.55), and having conversations with parents over the phone (d=.49).

Based on program directors' reports of the intensity of certain types of activities in their program offerings, evaluators found evidence of positive associations between program attendance and a program's focus on academics, arts, and physical recreation:

  • Programs that reported a higher relative intensity of academic activities tended to have higher attendance rates (r=.26).
  • Analyses found a positive relationship between the degree to which a program focused on activities in the arts and the attendance rate (r=.19).
  • Programs that engaged participants in physical activity more frequently tended to have higher attendance rates (r=.19).

In addition, for elementary-grades programs, there was a positive association between the intensity of activities focused on youth development (e.g., conflict resolution) and program attendance (r=.19). This relationship was not significant for middle-grades or high school OST programs. There were no notable associations between the intensity of career development or civic engagement activities and program attendance.

Social Development Outcomes of Youth

OST programs encourage positive youth development, in part by giving participants opportunities to interact in relaxed settings and fostering positive interactions among youth and between youth and adults. In the second year of the OST initiative, participants reported a strong sense of connection to their OST program as well as a moderate level of engagement in prosocial behaviors.

Program-level analyses revealed specific program features that were associated with positive social development outcomes among participants. In general, when program content included a strong focus on civic or social development programming, participants experienced measurable benefits in their sense of belonging and prosocial behaviors, such as helping or complimenting other youth.

  • The extent to which a program focused on youth development (e.g., conflict resolution, peer discussion, socializing) was positively correlated with participants' reports of engaging in prosocial behaviors (rs =.38).
  • There was also a positive relationship between a program's level of civic programming (e.g., discussion of current events, service projects) and participants' sense of belonging (rs =.32).
  • A program's focus on career development activities (e.g., career exploration, field trips to businesses) was positively correlated with participants' reported sense of belonging (rs =.20).

In addition, the extent to which programs provided opportunities for youth leadership was positively correlated with youth reports of their sense of belonging within the program and of engagement in prosocial behaviors. Participant-level analyses found a positive relationship between the number of leadership opportunities in which a youth participated in the OST program and their sense of belonging in their program, for both middle-grades participants (rs =.24) and for high school participants (rs =.52). For middle-grades students, there was a positive association between participation in leadership opportunities and reports of engagement in prosocial behavior (rs =.38).

Across all grade levels, analyses found a positive relationship between participant reports of the quality of their interactions with peers and their reports of their sense of belonging in the program (rs =.61). This association was significant and strong for elementary-grades (rs=.65), middle-grades (rs =.56) and high school (rs =.72) programs. Among middle-grades participants, there was a significant positive relationship between reported interactions with peers and engagement in prosocial behaviors (rs =.23).

The evaluation also revealed a positive correlation between a participant's reports of interactions with program staff members and their sense of belonging in the OST program (rs =.68). This correlation was significant and strong at all grade levels, but particularly for high school youth (rs =.79) and elementary grades youth (rs =.71). For middle-grades youth, reports of positive interactions with staff members were also positively correlated with reports of engagement in prosocial behaviors (rs =.22).

Youth Academic Outcomes

OST programs contribute to improving academic outcomes for youth by offering activities geared towards building the skills and knowledge that can contribute to school success and also by reinforcing students' perceptions of themselves as individuals capable of academic success. In Year 2, participant survey responses indicated a moderately high level of self-reported academic benefits due to OST participation. Elementary-grades participants were more likely than their middle-grades and high school counterparts to report academic benefits as a result of OST programming (r=.15 for the difference between elementary and middle; r=.14 for the difference between elementary and high). Elementary-grades participants recorded an average score of 3.10 on a four-point scale, while middle school participants averaged a 2.84 and high school participants averaged a 2.86.

Participants in the second year recorded a relatively high mean on the academic motivation scale. Again, there were notable differences in the level of academic motivation based on grade level, with elementary-grades participants reporting the highest levels of academic motivation, with a mean of 3.48 on the four-point scale, followed by middle school participants (3.20; r=.22) and high school participants (3.11; r=.29).

Analyses revealed specific program components that were associated with positive academic outcomes among participants, including staffing patterns, participation in technical assistance, and program content or focus. Staffing patterns were especially well associated with student reports of academic outcomes. For example, participants in programs that hired some high-school staff reported greater academic benefits than participants in programs that did not (r =.22). Participants in programs with some high-school staff also reported higher levels of academic motivation (r=.24). As noted earlier, programs typically hired staff members with diverse qualifications, and high-school staff were supported by more experienced staff members. Among programs that employed high-school staff, 89 percent also employed college students, 73 percent hired activity specialists, and 65 percent employed certified teachers. In program observations, high-school staff were frequently observed serving as support staff and tutors: a possible explanation for the correlation between the presence of such staff and participants' reported academic benefits is that programs with high-school staff were able to provide more individual attention to youth, such as one-on-one or small group tutoring.

Academic motivation was somewhat higher in programs without a master teacher on staff (r=.22) and in programs without specialist staff (r=.30). This finding suggests that programs were more likely to engage professional support when they served populations who were struggling academically or most in need of additional support services. Similarly, participants in programs without specialist staff reported greater academic benefits (r =.30).

Program efforts to develop staff skills and to engage families were also positively associated with academic outcomes. In particular, evaluators found a positive relationship between a program director's self-reported level of participation in technical assistance and participants' academic motivation (rs =.36). There was also a positive relationship between the frequency with which program directors communicated with parents and both academic benefits (rs =.24) and academic motivation (rs =.27).

Not surprisingly, evaluators also found a positive relationship between the extent to which a program focused on academics and participants' self-reported academic benefits (rs= .25). In addition, participants in programs that used a published or externally developed curriculum reported higher levels of academic motivation than participants in programs that did not (r=.42).

Middle-grades participants who reported taking on more leadership roles in their OST programs tended to report greater academic benefits (rs =.29) and academic motivation (rs =.21).

Programs that fostered positive relationships among youth and between youth and staff also demonstrated positive academic outcomes. Across all grades, analyses found a positive relationship between participants' reports of their interactions with peers and academic benefits (rs =.54). There was also a positive association between interactions with peers and academic motivation for elementary (rs =.56) and middle-grades participants (rs =.38).

Systems Outcomes

Opportunities for provider organizations. Overall, the majority of executive directors reported that the DYCD initiative had increased their organization's capacity to serve more youth and families either to a great extent (53 percent) or somewhat (27 percent). Executive directors' reports of the opportunities the DYCD initiative offered their organization reflected the same patterns as in the first year of the initiative. In Year 2, executive directors most frequently reported that the OST initiative had "to a great extent" or "somewhat" increased opportunities for training and technical assistance for their staff (75 percent).

More than half of directors also reported that the initiative had increased opportunities to partner with city agencies (64 percent), cultural organizations (61 percent), and a public school (60 percent). This finding of increased partnerships with public schools was greater in Year 2 than Year 1: 32 percent of executive directors reported that, compared to other out-of-school time programs, their DYCD OST programs established linkages with surrounding schools much more or somewhat more, while 25 percent reported this in Year 1 (V=.19).

As earlier noted, in the second year, DYCD began enforcing attendance and enrollment requirements by withholding a percent of funding from programs that did not meet participation targets. Perhaps related to this policy, executive directors' survey responses reflected increased focus on participation tracking. In Year 2, 39 percent of directors reported that their OST-funded programs tracked student program attendance more than their programs funded through other sources, compared to 17 percent who reported this experience in Year 1 (V=.38). While the DYCD policy may have contributed to increased participation rates-as reflected by higher enrollment numbers and higher rates of daily attendance- executive directors reported that it was a challenge to meet these standards. Fiftyone percent of executive directors reported that meeting DYCD's enrollment and attendance requirements was a challenge, significantly more than the 40 percent who reported this challenge in Year 1 (V =.12). In general, more than half of executive directors (55 percent) reported that the administrative burden associated with the initiative presented a challenge for their organization.

Not surprisingly, evaluators found notable differences in the capacity of organizations with out-of-school time budgets greater than $500,000 and organizations with smaller budgets devoted to out-of-school time programming. In particular, compared to organizations with smaller out-of-school time budgets, provider organizations with at least $500,000 devoted to out-of-school time were more likely to provide their staff with employment benefits, paid professional development, and opportunities for promotion.

However, there was also evidence that the OST initiative is helping to increase the capacity of provider organizations with small OST budgets to offer high-quality programming. Executive directors were asked to compare their organization's DYCD-funded OST programs to other out-of-school time programs sponsored by the organization. Directors of organizations with small OST budgets were more likely to report notable differences between their DYCD-funded programs and their other programs. In particular, organizations with small OST budgets were more likely than providers with large OST budgets to report that their DYCD OST programs complied with city and state child care regulations somewhat more or much more (30 percent, compared to 8 percent; V =.29). Organizations with small OST budgets were also more likely to report that their DYCD OST programs adhered somewhat or much more to strict standards about hiring and screening qualified staff (32 percent, compared to 10 percent; V =.28).

Meeting the needs of working parents. Parent survey responses indicated that the OST initiative is filling a need for structured after-school opportunities in New York City. Seventy-three percent of parents reported that the OST program was the only structured program their child attends after school.

Overall, parents were satisfied with the quality of the OST program in Year 2. Sixty-one percent of parents rated the program as "excellent" and an additional 20 percent said it was "very good." However, the survey responses also demonstrated that parents felt that there was room for improvement in the quality of OST programming. Overall, slightly less than half (46 percent) of parents strongly agreed that the OST program helped their child academically, and this opinion differed significantly by grade level.

Parents of elementary- and middle-grades OST participants also reported that the availability of the OST program improved their own opportunities. Eighty-four percent of parents reported that they work outside the home, including 64 percent who work 35 hours or more per week. Sixty-five percent of parents strongly agreed that the program hours fit their needs, and more than half strongly agreed that the OST program met their needs by making it easier for them to keep their job, work more hours, or attend school.

« Previous | Next »

References

DYCD expects middle-grades participants to attend 75 percent of the required 288 hours, which equals 216 hours. Similarly, high school participants are expected to attend 70 percent of the 108 required hours, which equals 76 hours.