Evaluation of the New York City Out-of-School Time Initiative: Implementation of Programs for High School Youth
Click here to download the full report:
Evaluation of the New York City Out-of-School Time Initiative: Implementation of Programs for High School Youth
The New York City Department of Youth and Community Development (DYCD) launched the Out-of-School Time Programs for Youth (OST) initiative in September 2005. At the same time, it contracted with Policy Studies Associates (PSA) to conduct a comprehensive evaluation of OST. This report summarizes evaluation findings from the second year of the OST initiative, 2006-07.
In its 2005 Request for Proposals under the initiative, DYCD described its OST vision as follows: "A quality OST system offers safe and developmentally appropriate environments for children and youth when they are not in school. OST programs support the academic, civic, creative, social, physical, and emotional development of young people and serve the needs of the city's families and their communities. Government, service providers, and funders are partners in supporting an accountable and sustainable OST system." Accordingly, DYCD designed the components of the OST program to reflect this vision. The largest program component, known as Option I, funded OST programs for youth in elementary, middle, and high schools in neighborhoods throughout New York City. Expectations for Option I programs varied by the grade level served, with programs for younger youth expected to provide more programming hours and hence more comprehensive services to youth attending programs on a more frequent basis, compared to programs serving older youth. Option II was designed to support OST programs that would use private match funds to subsidize at least 30 percent of their OST budgets; these programs would serve youth at any grade level. Option III programs were to be operated in collaboration with the Department of Parks and Recreation and offered at Parks sites; these programs would also serve youth at any grade level.
Based on the first year of OST operation, evaluation findings identified avenues for improving the effectiveness of OST programming as the initiative matured and programs became more fully established in their schools and communities. Although programs successfully enrolled students in the first year, they struggled to maintain high youth participation rates, suggesting a need to establish program policies and activity offerings that encouraged regular participation. While programs in the first year consistently provided safe and structured environments for participants in the out-of-school hours, they experienced challenges in delivering innovative, content-based learning opportunities that engaged youth. Programs also experienced challenges in recruiting well-qualified staff members, pointing to a need to develop effective practices for staff recruitment and for development of staff skills.
In the second year, the evaluation examined recent evidence of programs' efforts to improve program quality and scale. In particular, evaluators looked for evidence that programs had established structures to support high-quality staffing and effective partnerships, were delivering rich program content through activities that also fostered positive interpersonal relationships, and were engaging youth in programming to develop their skills in both social and content-based areas. Evaluators also examined the extent to which programs increased the number of youth served and their level of program attendance.
Scope and Extent of OST Programming in Year 2
During the 2006-07 school year, evaluators examined data from a total of 536 OST programs that offered services throughout New York City and reached more than 69,000 participants, up from 51,000 participants in the preceding school year. Approximately two-thirds of these youth were enrolled in school-based OST programs and the remaining third in center-based programs. These programs served youth of all ages. More than 40,000 participants were served by OST programs located in zip codes identified as in high need of services for youth.
This increase in the scale of services reflected a considerable additional investment in OST programming throughout the city. According to data on funding levels presented in the initiative's online management information system, DYCD awarded more than $66 million to OST programs in Year 2, a substantial increase over the $44 million awarded in Year 1. The median second year award was $100,000, compared to $73,000 in the first year, and grant awards ranged from $3,100 to $514,000.
Evaluators identified a representative sample of 15 OST programs from which to collect additional data. New York City Department of Education (DOE) data on participants in these 15 in-depth study sites confirmed that the OST initiative reached New York City students who could benefit from the support of high-quality programs. Across these programs, 85 percent of youth were eligible for free or reduced-price lunch (compared to approximately 82 percent of youth citywide), 21 percent were classified as eligible for English Language Learner services (compared to 13 percent citywide), and 14 percent received special education or related services (equivalent to the 14 percent rate citywide).
In addition, OST participants' prior performance on the English Language Arts (ELA) and math tests (administered to all New York City students in grades 3-8) indicated that participants were at risk academically compared to students citywide. Forty-nine percent of OST participants scored at performance levels 3 or 4 on the ELA test, indicating that they were performing at or above grade level, compared to approximately 57 percent citywide. Fifty-four percent performed at this level in math, compared to 57 percent citywide.
In contrast, the data also suggested that OST participants were at least as engaged in school as were students citywide. The average school attendance rate of elementary-grades participants in the year prior to OST participation was 92 percent, equivalent to the citywide rate. Middle-grades participants, however, had a higher school attendance rate than did their citywide counterparts (95 percent, compared to 90 percent), as did high school participants also (90 percent, compared to 81 percent).
Structural and Institutional Program Features
In the second year of the OST initiative, programs continued to develop policies, strengthen partnerships with schools and families, and build on their efforts in the first year to establish strong foundations for high-quality programming. In particular, OST programs and DYCD both worked to improve staff recruitment and training, addressing one of the primary challenges reported in Year 1.
Program focus. Program directors almost unanimously reported that providing a safe environment for youth was a major objective of their OST program (97 percent) and that they aimed to help youth develop socially (93 percent).
Highlighting a possible tension between social and academic goals, evaluators found a small but notable decline from Year 1 to Year 2 in the percent of program directors who identified academic improvement as a major objective of their program. In the second year, 80 percent of program directors reported that one of their major objectives was to help youth improve their academic performance, compared to 88 percent in the first year (V=.11). In contrast, more program directors reported in Year 2 that a challenge in providing high-quality programming was that the principals of the schools their participants attend would like the program to be more academically focused (47 percent, compared to 39 percent in the first year; V=.10). Despite this challenge, more than half of program directors (59 percent) believed the quality of their relationships with schools had improved in the second year.
Strategies for participant recruitment. Effectively recruiting students continued to pose a challenge to program directors. Forty-five percent reported youth dropping out because they lost interest as a challenge, and 43 percent identified youth not attending the OST program regularly enough to have enriching experiences as a challenge.
Overall, strategies for participant recruitment remained much the same in Years 1 and 2. As in their first year of operation, an overwhelming majority of Option I program directors (90 percent) reported that they offered open enrollment to all youth who were interested in attending the program. In addition, at least half of program directors reported that they targeted youth who were recommended by school-day teachers or counselors (55 percent) and youth with siblings already attending the program (50 percent).
Program director and staff qualifications and responsibilities. In Year 2, DYCD required any elementary- or middle-grade OST program that served at least 100 youth to hire a full-time program director, reflecting the need for one person working full-time to coordinate the multi-faceted components of the OST program in each site. Survey findings confirm that OST programs generally complied with this expectation. In addition, program directors had strong educational qualifications. Eighty-six percent of program directors had completed a four-year college degree or higher, and 37 percent had a master's degree or higher. Seventeen percent of program directors reported that they were certified to teach.
Program directors reported that finding qualified staff was a lesser challenge in the second year of the OST initiative than in the first. In Year 2, only 16 percent of program directors reported that finding qualified staff to hire was a major challenge, compared to 48 percent in the first year of the initiative. In fact, 42 percent of program directors reported that finding qualified staff to hire was not a challenge at all, while in the first year all program directors reported that this was at least a minor challenge (V =.54). One reason for programs' improved capacity for staffing may be that most programs were able to retain at least half of their staff from the first year of the initiative (68 percent of directors reported that at least half of their staff worked in the program in the previous year).
OST programs employed staff members with varied qualifications and prior experiences to carry out roles in implementing program content and supervision. By hiring staff with a mix of experience levels, programs worked within their budgetary constraints. Although a large proportion of programs employed certified teachers and activity specialists to provide targeted programming on a more limited basis, in general, programs relied most heavily on non-certified and non-specialist staff.
Technical assistance through the OST initiative. In the second year of the OST initiative, DYCD made efforts to more closely align the focus of technical assistance workshops and consultations to program needs. In addition, DYCD improved its methods of tracking program referrals to technical assistance services to ensure that programs received relevant assistance. Perhaps reflecting these improvements in communications and targeting of technical assistance, during the second year of the OST initiative program directors reported higher levels of satisfaction with the technical assistance that DYCD provided than they did in Year 1.
Program staff members (other than directors) also took advantage of technical assistance opportunities offered in Year 2. Eighty percent of staff members surveyed reported attending a workshop, 56 percent participated in an on-site consultation, and 42 percent attended an institute or conference. Only 9 percent of staff members reported that they did not participate in any technical assistance in Year 2. Program staff members were also generally satisfied with the technical assistance they received through DYCD. Forty-four percent felt the training served their purposes completely, while another 46 percent felt it was a good start.
Overall, when asked to compare the technical assistance they received during the first and second years of the initiative, 62 percent of program directors either agreed or strongly agreed that the professional development they received through the OST initiative was more useful in the second year. Fifty-five percent reported that the professional development their staff received was more useful. Program directors were also more satisfied with DYCD's approach to program monitoring in the second year: 64 percent either agreed or strongly agreed that DYCD's monitoring approach was more effective in the second year.
Implementation of Process and Content Features
Program activities. Achieving positive outcomes for participants requires that program activities be reasonably diverse and capable of promoting personal development. Reflecting program objectives, activities varied somewhat by grade level. In Year 2, elementary-grades programs most often offered homework help, visual arts and crafts, group academic instruction, organized reading activities, and learning games. Middle-grades programs offered a similar roster of activities, with an additional emphasis on organized team sports. High school programs were more specialized, and each targeted a smaller set of program activities. These activities tended to be more civic-oriented than in the elementary- and middle-grades programs and to be more focused on social development.
Content delivery strategies. Overall, youth participants expressed a strong degree of satisfaction in the extent to which participation in OST program activities exposed them to new experiences. However, through activity observations, the evaluation concluded that programs often struggled to design and implement activities that provided youth with the opportunity to actively engage in learning through hands-on activities, discussion, or meaningful choices and roles.
Fostering positive relationships. OST programs consistently developed positive relationships among youth and between youth and staff in Year 2 of the initiative. In surveys, participants overwhelmingly reported positive relationships with program staff members and with their peers. Observer ratings of activities confirmed that programs were developing participants' personal and social skills.