Contents

• Preface
• Summary
• Acknowledgments
• Abbreviations
• Chapter One: Introduction
• Foundation Goals and Expectations
• Purpose of This Monograph
• Methods
• Themes from the Literature on Coordination and Collaboration Across Agencies
• Data Sources
• Analysis
• Study Caveats
• Organization of This Monograph
• Chapter Two: The Early Phases of the Initiative and Decisions Made: The Importance of Context
• Chapter Three: How Sites Attempted to Improve Access, Quality, Information, and Sustainability and the Progress They Made
• Chapter Four: Enabling Coordinated System-Building Efforts
• Chapter Five: Lessons for Other Cities
• References

Click here to download the full report:
 Hours of Opportunity

Youth (grades kindergarten through 12) across the United States participate in publicly supported out-of-school-time (OST) programs in group settings after school and in summertime. Such programs include simple after-school care services to support working parents, programs specifically structured to help reduce problem behaviors, programs that reinforce academic achievement, and programs that offer access to sports, arts, crafts, and other activities. Local service providers may be a combination of community-based organizations (CBOs), city agencies, and intermediary organizations. The collection of OST providers and funders in a city can often be fragmented and uncoordinated, however (Bodilly and Beckett, 2005; Halpern, 2006).

Recent studies indicate that high-quality, well-managed and -structured OST opportunities can help youth develop critical academic, social, and emotional attributes and skills, especially if offered consistently and persistently over time (Lauer et al., 2006; Bodilly and Beckett, 2005). This research has drawn attention to whether publicly supported programs meet these conditions and whether they are effective avenues for youth development. In particular, cities are attempting to improve the access and quality of programs to ensure that more youth have the opportunity to achieve the results associated with the most effective programs.

Foundation Goals and Expectations

To further promote effective provision, The Wallace Foundation decided to fund an out-of-school-time learning initiative to help five cities (Providence, New York City, Boston, Chicago, and Washington, D.C.) develop and test ways to plan and implement coordinated OST programming that, ideally, would achieve four goals: increased access, improved quality, better use of data for decisionmaking, and increased sustainability.

Increased Access to and Participation in OST Programs. The Foundation expected sites to ascertain the demand for services from different age groups, how to increase demand among certain groups, and the most effective locations in which to meet demand with supply in order to develop plans to improve participation. To increase access, the sites could more systematically address such issues as safety (in transit and at the program location), access to transportation, affordability, and convenience (hours of operation amenable to children’s and parents’ schedules). They could build program locator systems or otherwise work to ensure that parents and children knew about the programs and how to access them. In addition, cities could conduct marketing activities to appeal to underparticipating groups, such as teens. Finally, the cities could open more slots at more locations to increase enrollment.

Improved Quality of OST Programs. While high-quality OST programs can produce positive outcomes for participating students, the quality of programming within a city is typically mixed. The Foundation expected sites to create mechanisms to support high-quality programs and ensure strong enrollment, attendance, and desired student outcomes. Activities could involve developing standards, using standards to assess program quality, monitoring improvement over time, and vetting providers upon entry into the field with common criteria. Performance incentives could be offered to programs. In addition, the cities had to ensure that a supply of professional providers was available to meet expansion and quality goals simultaneously, implying that some professional development and training might be needed.

Finally, sites could undertake evaluations of the effort to ensure that the changes resulted in improved outcomes.

Better Use of Information Systems for Improved Decisionmaking. Cities have not traditionally invested in developing data systems to support improvements. As a result, many cities across the United States are unable to accurately report the enrollment and participation of youth in OST programs. To support access and quality, cities needed to track program activities and monitor participation and attendance rates. This required the adoption of management information (MI) systems to track programs and participation, if they did not already exist.

Improved Financial Sustainability. Finally, The Foundation was interested in making a large investment in system-building efforts, but not in funding the OST programs themselves or becoming a perpetual donor. Thus, grantees were required to develop sustainable funding sources for OST programming and system-building activities.

Site-Level Goals. While the four goals drove the efforts, The Foundation understood that sites would have to apply them in accordance with their own specific circumstances and city needs; therefore, each site was to develop its own methods for meeting those goals. For example, a site might concentrate on improving access for a specific group of children—middle school teens, for example. It might already have a fully developed MI system; therefore, it would concentrate elsewhere or devote resources to one activity in the early years and focus on others in later years.

Purpose of This Monograph

To share the learning from this initiative with the larger OST field, The Wallace Foundation asked RAND to document the five cities’ progress toward building the systems infrastructure to provide more coordinated and effective services. The purpose of the RAND study, conducted between January 2008 and May 2009, was to examine how the participating cities were developing and aligning local assets to maximize collective effectiveness in delivering sustained, high-quality OST programming to school-age children. In our analysis, we focused on the many differences across the sites to provide insights into how grantees made important choices.

This examination had two tasks: (1) an analytic description of the development of the five OST sites supported by the grant, addressing what the sites attempted to do under the grant and the progress they made, and (2) a description of the MI systems established to track student participation in each of the Wallace-funded sites as well as in other cities. This monograph focuses on the first task and addresses the following questions:

1. What decisions did the sites make about approaches to improving OST systems during the early phases of the initiative? What drove these decisions?
2. What progress did sites make toward increasing access, improving quality, using data-based decisionmaking, and improving sustainability?
3. How did collaboration and coordination enable progress? What other enablers were important?

The analysis provides interesting examples of what the grantees did and the proximate result—the immediate effect on OST provision structure, access, quality-assurance processes, information for planning, and sustainability.

Methods

Our unit of analysis was the multiorganizational initiative in each city. We chose a replicated qualitative case-study design to answer our research questions. The study resulted in a descriptive analysis of the activities that the sites undertook and the conditions that led to progress toward each city’s specific goals (under the broader Wallace Foundation goals). The analysis involved examining the data for similarities and differences among the sites and extracting themes in terms of what enabled and hindered progress. We exploited the variation among the sites in context, conditions, and what was attempted to provide useful comparisons about the ways in which different choices influenced progress toward coordination and system building. These variations are covered in Chapter Two. The examples should help other cities better plan improvements in their OST infrastructure.

In this section, we present the findings from the literature reviews that guided our investigation, data sources for the research questions, and the analytic approach. We reviewed the literature on efforts to build greater coordination across public service agencies to help determine the types of mechanisms that the sites might use to promote system building, and we used this information to develop protocols and guide our analysis. We used the sites’ own proposals and plans to determine what their systems looked like before the grant and what they intended to accomplish, which we compared to what they had accomplished by the spring of 2009. The descriptions of the grant’s goals acted as the categories that we tracked and guided our search for themes in the efforts that promoted progress.

Themes from the Literature on Coordination and Collaboration Across Agencies

A review of the literature indicated that the sites would likely face challenges as they attempted to develop citywide approaches in which multiple organizations were at work.¹ These organizations might include government agencies, schools, CBOs, foundations, state and federal oversight agencies, and agencies with funding streams that target children. (See Bodilly and Becket, 2005, for a more detailed description of the actors involved in OST provision.)

In general, coordination of organizations in the public sector is undertaken to achieve a shared goal that is considered important to each organization but often not achievable individually due to a lack of political power or resources. In sectors characterized by resource constraints, such as public OST provision, coordinated approaches also hold the promise of increasing the efficiency of provision by reducing duplication and gaps in service. This increased efficiency can translate into expanded access. Furthermore, agreement among organizations as to what constitutes quality provision can result in more consistent quality across programs. However, these theoretical benefits are gained only through intense and sustained efforts at multiorganizational coordination or system building.

The literature on public-sector interagency coordination or collaboration to improve systems indicates that these types of efforts are slow to develop, fragile, typically struggle to sustain themselves over time, and develop differently in each site due to the heavy influence of city contextual factors. Past research has identified specific barriers and enablers to such multiorganizational coordination efforts. Factors influencing the success of initiatives include leadership capability, sufficient and capable staffing, buy-in from major stakeholders, public support, communication among stakeholders, funding, and the city context. In particular, past efforts have depended heavily on the emergence of legitimate initiative leaders who use unifying techniques to ensure buy-in and harmony among participating organizations and key managers.

The literature describes a series of activities in which sites might engage to varying degrees, which we used to develop our approach to data collection and to present our findings.

Conduct a needs and assets assessment. A starting point might be the identification of gaps in services and system weaknesses, along with community assets that can be leveraged to address gaps. Such data can be gathered through market research, discussions with stakeholders, and audits.

Build shared goals. Development of and buy-in for a shared set of goals is the foundation on which all activities depend. The group of agencies or organizations should work toward a commonly held series of goals or expected performance improvements, perhaps through meetings, regular communication, and sharing of information to build common understanding and purpose.

Consolidate or develop more coordinated structures and roles. To promote efficiency and clarity in the system, the effort might involve reorganizing agencies into more effective or efficient structures with clear roles and the oversight needed to move forward.

Coordinate among groups. Routine and effective communication and coordination among organizations is needed to facilitate joint work or group decisionmaking and may be accomplished through regular meetings among representatives or a coordinating organization.

Plan for and implement coordinated activities. The organizations might develop plans for joint or complementary activities to increase impact. Fundraising and resource redistribution might be key components of the effort.

Develop, analyze, and share information. Collaborative activities, like any improvement efforts, usually involve the development, analysis, and sharing of information. In this particular instance, that might include collecting and using data on access to and participation in programs, developing and using assessment tools to evaluate learning and diagnose and address failure, assessing the effectiveness of staff training and professional development, and collecting and using data to find funding flows and determine where additional resources are needed. MI systems provide information for these improvement efforts; analytic talent and dedicated time are also required to interpret and make use of the data collected. Defined reporting structures also encourage consistent sharing of information.

Communicate with the public or engage stakeholders. Coordination initiatives often involve the development and communication of information about the state of the field and what could be done to improve it, ensuring that this information flows to the public to garner increased partners and support in the effort. Public relations and advocacy campaigns help gain support among parents, policymakers, and community leaders by making visible the need and articulating the possible solutions. In addition, agencies must develop the means to communicate with the public about the services being provided to ensure usage. This is especially true for after-school programs with historically low rates of attendance.

Establish incentives, rules, and supports. To ensure that the efforts continue cities must put in place quality standards and evaluations of programs against standards and provide supports, such as professional development, to help providers meet those standards. They might also provide clear incentives for improvement or take punitive action if providers fail to meet standards or expectations.

Data Sources

Our primary goal was to track and describe what the sites did and how they did it. We could not observe progress as it was made; the study began well after the sites had started their work, and intensive observations across sites were beyond the resources of the study. To understand where sites started and what their intentions were (research question 1), we reviewed statements made in initial proposals and business plans concerning each of the four expectation categories: improving access, improving quality, developing MI systems, and developing and implementing plans for financial sustainability. We also collected background information from primary sources.

To determine the sites’ progress (research question 2), we collected data on activities during the grant period and records of accomplishment. A significant portion of the data came from the annual reports and business plans submitted by the sites to The Wallace Foundation, along with reports from market surveys and other data-collection efforts. The other primary source of data was interviews at the five sites and with staff at The Wallace Foundation. We conducted two rounds of interviews, the first in spring 2008 and the second in spring 2009. In the interviews, we asked about (1) the general conditions of OST at the sites at the beginning of the grant period and what the sites hoped to accomplish with the grant; (2) activities conducted during the planning phase and how they informed future efforts; (3) their progress toward the four categories of expectations; (4) how they used cooperation, coordination, or collaboration to accomplish their tasks and whether it was important to the effort; and (5) what enabled or impeded their efforts and why.

Table 1.1 shows the number of interviewees by type at each of the sites. We interviewed 125 individuals in total. Note that the table counts each interviewee once. In many cases, we interviewed these contacts twice over the course of the study. At each site, we aimed to interview key individuals who were involved with the initiative during the planning and implementation periods, including key city officials, school district officials, staff from intermediary organizations, OST providers, and foundation funders.

Table 1.1
Number of Interviews, by Site and Affiliation

At some sites, a large number of people were involved, while other sites had fewer people engaged in the process. In initial interviews with Wallace Foundation staff, we asked for the names of their main points of contact and key players in the initiative at each site. We then used phone interviews to contact these actors to determine whom they thought we should interview. We also used documents such as business plans to identify who had been involved in the site’s efforts over time. We contacted these individuals. We ensured that we interviewed the major players in the initiative and additional actors whom they recommended we talk with, including those in the provider community and administrators of school programs.

For each interview, RAND researchers took notes, which were supplemented by and checked against audio recordings of the interviews. Using the interview data, business plans, and progress reports submitted to The Wallace Foundation, we developed site-specific case studies, which can be found in Hours of Opportunity, Volume 3: Profiles of Five Cities Improving After-School Programs Through a Systems Approach (McCombs, Bodilly, et al., 2010). In developing the case studies, we took care to examine interviewees’ responses for consistency across individuals and also looked for consistency with documents submitted to The Wallace Foundation. Cases in which there was disagreement among respondents were noted and explored in follow-up interviews. Where there were clearly divergent views, we present both views or note the uncertainty surrounding the exact events.

Analysis

We analyzed the case-study data for cross-site similarities and differences concerning the progress made toward the four goals and whether and how coordination or collaboration was useful to that progress. Variation among the sites provided us with the means to draw contrasts to help determine the conditions under which certain approaches were chosen and the conditions under which they flourished. In addition, we reviewed the interviews for examples of how one set of activities might have influenced the progress of another set; for example, the development of an MI system to track participation led to other activities, such as an effort to argue effectively for funding.

To address research question 1, we simply condensed material from the sites’ proposals and our interviews to outline what each site wanted to accomplish, but we also used the coordination mechanisms listed earlier to organize the findings around key activities in the early phases, such as needs assessments and market research.

To assess the progress and how the sites accomplished it (research question 2), we took the information from the case studies and business plans and summarized it under the four goal categories.

We conducted a similar exercise on the coordination mechanisms used and how the respondents described them (research question 3). We relied on the site interviews for information about how they approached coordination and then placed that information into categories developed from the literature: conducting a needs assessment; building shared goals through meetings and ensuring regular communication and information sharing; consolidating organizations into more coordinated structures; coordinating among groups through regular meetings or the work of special committees or task forces; planning and implementing joint activities; developing, analyzing, and sharing information; communicating with the public; and providing incentives, rules, and support for improvements.

Finally, we reviewed the case studies for common enablers and constraints, allowing us to draw out themes.

The final findings were briefed to The Wallace Foundation, and each site was provided with its case study for comment. Later, the sites and The Foundation were provided with a copy of the draft report for review to ensure that our findings were factually correct as of spring 2009.

Study Caveats

This monograph offers important information about cities’ efforts to build and strengthen OST systems that provide youth with access to high-quality OST programming and suggests a set of lessons learned for other cities interested in improving their OST programs. It relies on reports from our interviewees—the actual actors involved in the initiative. While we attempt to corroborate their reports by looking for consistency with other interviewees and documents, the data are perceptual and, in some cases, retrospective, which may lead to an incomplete or unintentionally biased report of events.

Each site started with little data; therefore, we cannot quantitatively describe certain changes from before the initiative to after (e.g., participation rates of enrollees). However, we can describe how the cities developed such data where none existed before. Thus, the outcomes of note are the improvements made to the cities’ infrastructure that could logically connect to the initiative’s goals.

This monograph covers only the accomplishments from the beginning of the grant to spring 2009. However, in each city, work has continued and activities have evolved beyond what is presented here.

Organization of This Monograph

The remainder of this monograph is organized as follows. Chapter Two describes early planning efforts in the five sites. Chapter Three details what the sites did to address the four goals. Chapter Four reviews findings about the mechanisms for coordination that were used by the sites and the factors that enabled or constrained the sites’ efforts. The final chapter reflects lessons learned from these cities that could inform other cities’ efforts to improve their OST systems.

< < Previous | Next > >

References

1. We condensed findings from the following sources: Bodilly, Chun, et al., 2004; Bodilly and Beckett, 2005; Bodilly, Augustine, and Zakaras, 2008; Dluhy, 1990; Banathy and Jenlink, 2004; Hall and Harvey, 2002; Halpern, Sielberger, and Robb, 2001; Halpern, 2006; Keith, 1993; Mattressich and Monsey, 1992; Russell et al., 2006; and Tushnet, 1993.