Hours of Opportunity
Click here to download the full report:
Hours of Opportunity
High-quality out-of-school-time (OST) programs, which for our purposes include both after-school and summer learning programs, have been shown to positively affect youth development and reduce negative behaviors. At the same time, the provision of OST programming in urban centers has been criticized for poor quality and lack of access for those most in need of services. In response, The Wallace Foundation sponsored an initiative in 2003 to help five cities develop better coordinating mechanisms to reduce OST fragmentation, redundancy, and inefficiency and to increase OST access and quality. The Wallace Foundation first provided each site with a planning grant to support the development of a business plan. After The Foundation approved a site’s plan, the site received its implementation grant. The initiative began with a planning grant to Providence, Rhode Island, in 2003, followed by grants to New York City, Boston, Chicago, and Washington, D.C. The Foundation’s funds were to be used for cross-agency and within-agency planning and coordination to meet the initiative’s goals. In 2008, The Foundation asked RAND to assess the progress of the five sites.
Purpose of This Study
The RAND study had two interrelated parts. The first, reported here, was to describe the sites’ work under the grant and to analyze the conditions and activities that contributed to their progress in building a coordinated system of services that would meet the initiative’s goals: increasing access, improving quality, developing and using information for decisionmaking, and planning for sustainability. The second part of the study, reported in
Hours of Opportunity, Volume 2: The Power of Data to Improve After-School Programs Citywide (McCombs, Orr, et al., 2010), involved a detailed analysis of the cities’ progress in building and implementing management information (MI) systems to track student enrollment and attendance, including—but not limited to— The Wallace Foundation grantees. In this monograph, we specifically answer the following research questions:
- What decisions did the sites make about approaches to improving OST systems during the early phases of the initiative? What drove these decisions?
- What progress did sites make toward increasing access, improving quality, using data-based decisionmaking, and improving sustainability?
- How did collaboration and coordination enable progress? What other enablers were important?
The analysis provides interesting examples of what Wallace grantees did and why, as well as the proximate result—the immediate effect on OST provision, structure, access, quality assurance, and information for planning and sustainability. In-depth case studies of the individual cities are presented in
Hours of Opportunity, Volume 3: Profiles of Five Cities Improving After-School Programs Through a Systems Approach (McCombs, Bodilly, et al., 2010).
To address the research questions, we used a qualitative, replicated case-study approach in which the unit of analysis was the citywide, multiorganizational initiative funded by The Wallace Foundation. Our literature review of collaborative interagency reform efforts in the OST and other social service sectors further guided the development of our research questions, data-collection instruments, review of the literature, and interviews with key city leaders, leaders of communitybased organizations, principals, providers, and staff at The Wallace Foundation. These data were developed into case studies constructed around key descriptive and analytic categories that factored in evidence from key individuals and reports about the history of the OST sector in that site. We then conducted a cross-site analysis to identify patterns of activities that led to greater coordination across agencies and organizations. To further guide other cities in their improvement efforts, we examined and include examples of the types of activities that the five cities used in their attempts to improve service provision. The study did not measure the effects of these efforts on students and families.
Variation in Starting Conditions Among the Sites
The Wallace Foundation chose five cities, each with its own context, demographics, and organizational characteristics that influenced the development and implementation of the initiative. The variation afforded the opportunity to examine coordination and system building in multiple contexts, highlighting both city-specific and shared factors that appeared to contribute to or inhibit progress.
The sites varied in size. New York City had more than 8 million residents, while Providence had approximately 175,000. Each city had a high-need student population, with more than 60 percent of students eligible under federal guidelines for free or reduced-price lunch. Sites varied in their start dates. Providence began its planning grant early in 2003, followed by New York City later that year; the three other sites started in 2005. The sites varied in their existing infrastructure for OST provision. For example, the city of Providence did not contribute strongly to after-school programming prior to the grant, while the city governments in New York City and Chicago provided significant funding across an array of city agencies.
The Wallace Foundation provided planning grants to sites to encourage them to develop concrete ideas, with the expectation that solid plans would lead to implementation grants. It encouraged sites to use market research, needs assessments, and other information-gathering and analysis methods to identify specific targets for their work. The market research and gap analyses proved to be crucial starting points for several cities: This work identified areas of the city without provision, age groups that lacked accessible programs, and issues of concern to parents and students that acted as barriers to participation. Plans were developed to target these specific issues.
The different histories and structures of the cities led to different coordination structures. New York City and Chicago were led by city agencies and concentrated on improving services funded by the city. In Boston, Washington, D.C., and Providence, intermediaries led the work initially.
What Was Attempted and Progress Toward Goals
The Foundation set four broad goals for the sites. We tracked the specific activities that the sites undertook in each of these broad areas through the spring of 2009. While we do not describe all of these activities in detail in this monograph, we do provide several examples of what the sites tried to accomplish.
Goal 1: Increase Access and Participation. Providence, New York City, Washington, D.C., and Boston focused on access and addressed issues related to convenience and lack of access by opening additional programs in underserved neighborhoods and schools and, in one site, by providing transportation. Many sites also developed online program locators that parents and students could use to identify programming in their local area. These methods of improving access, combined with greater funding, increased the number of program locations and slots available to students in four of the five cities (Boston, Washington, D.C., New York City, and Providence).
Goal 2: Improve Quality. All five sites focused some energy on improving quality of programming. The mechanisms used varied but included the development and widespread use of quality standards, quality-assessment systems for providers, and incentives and contractual mechanisms to encourage improvement. Several sites offered professional development programs for OST providers and the coordinators who managed OST programming in schools.
Goal 3: Develop Information Systems for Decisionmaking. All the cities devoted considerable energy to developing web-based MI systems to track enrollment, attendance, and demographic data. These systems also collected information about providers and their programs. Data from these systems were used to determine which programs were attracting students, and, for the first time, cities were able to make use of data for planning. Interviewees reported that analytic capability was limited in some sites but that the data-based decisionmaking and communication strategies improved the agencies’ ability to plan. In addition, Washington D.C., New York City, and Providence reported using data from these systems to produce the evidence needed to argue for greater funding based on both need and improved effectiveness.
Goal 4: Plan for Financial Sustainability. Under the grant, the cities were asked to plan and develop sustainable funding strategies. While some cities had diversified funding sources, all sites were struggling with issues of financial sustainment when the study ended, exacerbated by the downturn in the national economy.
Collaboration and Other Enablers of Coordinated System Building
The cities varied in their use of mechanisms for collaboration, and this affected their progress. New York City and Providence, the sites with more longevity, used collaborative approaches to make significant progress toward the larger goal of a more coordinated system. In its first 18 months, Chicago concentrated its effort almost exclusively on the development of MI systems to enable further collaboration in building a better OST system. Boston struggled with collaboration in the early years of the initiative, which impeded its progress in creating a more coordinated system. Washington, D.C., simultaneously encouraged collaboration through a city-level coordinating structure while the school district pushed forward with a major initiative mostly on its own. Interviewees in all sites noted that there was still more work to be done in this regard.
The sites used a variety of collaborative mechanisms, including data collection and analysis to identify gaps in provision; consolidating functions within specific agencies; establishing a coordination structure, such as a steering committee, to ensure regular meetings; vesting a special adviser with the power and authority to ensure interagency cooperation; establishing memoranda of understanding (MOUs) across agencies to document specific agreements on the sharing of resources and interpretation of policy; creating structures for cross-agency information sharing used in joint decisionmaking; and providing incentives and supports for coordination.
The adept use of these and other mechanisms by some sites inspired a shared vision among the collaborators, which paid off significantly in the later years of the initiative. And, clearly, the funding provided by The Wallace Foundation acted as a catalyst for collaboration.
Mayors and their representatives proved to be crucial enablers of collaboration and system building. Actions by mayors, including restructuring agencies, increasing funding in the city budget, and demanding progress reports, positively affected the efforts. A recession, with its related drop in city and state budgets, was under way by the end of the study, and it significantly challenged the cities’ efforts to expand access, in particular.
Themes for Other Cities
The findings of the study suggest some themes that other cities working to improve OST provision might consider as they move forward.
Coordinated system-building efforts can improve access and quality. Four of the five sites successfully increased the number of students served. At the end of the study, all of them were in the process of building quality-assessment systems to help identify poorly performing providers and offer training and incentives for improvement. Four of the cities were using newly developed information to improve decisions regarding access and quality. However, all were struggling with the financial sustainment goal. Thus, The Wallace Foundation investment provided some proof that city organizations could work in a cooperative fashion to promote better OST services and programming.
Each city has a unique context that should drive goals. Based on their unique conditions, each city selected a slightly different focus, such as a targeted age group, targeted locations, or an emphasis on quality versus access. Other cities considering how to improve provision should not simply adopt one of these specific approaches, but should examine their own circumstances to identify how to best propel their efforts forward.
Investments in early planning and management information system development paid off. The sites deliberately considered the specific assets in place, the organizations involved, the challenges faced, and the funding available, which helped them identify targets for improvements. Collaborative early planning efforts also supported shared goals among the organizations and agencies that later proved useful as the efforts unfolded and inevitably faced challenges. By collectively going through early planning processes, organizations at sites such as Providence and New York City had the ability to effectively face challenges together. Sites that did not stress the development of shared goals, especially Boston, did not fare as well in meeting their OST goals. In addition, early collaboration on needs assessments paved the way in several instances for the cooperation needed to develop an MI system that provided the data necessary to further improve access and quality. Collection and analysis of data focused on specific improvements, allowed the sites to assess whether progress was being made, and supported arguments for additional funding. In addition, shared data enabled some sites to maintain their shared goals across organizations. New York City and Providence, with more years of experience, had pushed farther than the other sites in this direction by the end of the study.
Cities can consider an array of approaches to improving access and quality. The study sites adopted an array of ways to improve access and quality. Improving access involved identifying underserved populations and using appropriate mechanisms to increase enrollment, such as placement of programs in neighborhood schools, providing transportation to and from the programs, and providing programs at no cost to participants. Each city addressed quality through the adoption of standards, the use of the standards to assess program quality, provision of professional development, and evaluation of their efforts.
Cities can consider an array of mechanisms to increase coordination. The sites used an array of collaborative mechanisms to improve coordination—including restructuring, consolidating roles, establishing coordinating committees or steering committees, appointing mayoral envoys to ensure interagency cooperation, developing interagency MOUs, sharing information, and changing rules and incentives. Putting such mechanisms in place ensured that some sites kept moving forward toward shared goals.
Several enablers were important. Interviewees agreed on several important enablers of collaborative efforts: building a common vision among stakeholders, an early assessment of needs, development of an MI system, an actively supportive mayor, the buy-in of schools, and investment funding. The major constraint on progress cited was lack of funding and stovepiped funding that prohibited integrated services. Most sites rated the mayor’s support as essential, and, in three sites, mayoral involvement went beyond simple encouragement or “bully pulpit” statements. Active mayors crucially supported efforts in their cities by restructuring the organizational landscape, realigning funding sources, creating special adviser positions to ensure cooperation across agencies, chairing forums and overseeing intermediaries, and demanding analysis of outcomes for consideration in funding decisions. Rather than waiting for such mayors to emerge, it might be possible for other cities to educate their mayor early in the process about how he or she can positively participate in such an initiative.
Thus, other cities should consider what actions they can take within the confines of their specific environment. Small steps forward can add up over time to significant improvements for underserved children. This document provides important ideas and concepts to help inform those considerations.
Next > >