Contents

• Introduction
• Executive Summary
• Chapter 1: The Central Office in District-wide Teaching and Learning Improvement: The Importance of Transformation
• Chapter 2: Dimension 1: Learning-focused Partnerships with School Principals to Deepen Principals’ Instructional Leadership Practice
• Chapter 3: Dimension 2: Assistance to the Central Office–Principal Partnerships
• Chapter 4: Dimension 3 :Reorganizing and Reculturing All Other Central Office Units to Support Teaching and Learning Improvement
• Chapter 5: Dimension 4: Stewardship of the Overall Central Office Transformation Process
• Chapter 6: Dimension 5: Use of Evidence Throughout the Central Office to Support Continual Improvement of Work Practices and Relationships with Schools
• Chapter 7: What the Study Says about Central Offices and the System-wide Improvement of Teaching and Learning
• Methodological Appendix
• Conceptual Framework Guiding Data Collection and Analysis
• Data Collection
• Observations
• Interviews
• Documents
• Data Analysis
• References

Click here to download the full report:
 Central Office Transformation for District-Wide Teaching and Learning Improvement

In this appendix, we briefly elaborate on our methods of data collection and analysis. To begin, we note the central role that a carefully developed conceptual framework played in both the design of data collection strategies and in the analysis of data we collected.

Conceptual Framework Guiding Data Collection and Analysis

We designed our data collection instruments and framed our analytic work using a conceptual framework derived from socio-cultural learning theory (e.g., Lave, 1998; Rogoff, 1994; Tharp & Gallimore, 1991; Wenger, 1998) and organizational learning theory (e.g., Levitt & March, 1998). (For a detailed discussion of our conceptual framework, see Honig, 2008.) We chose these strands of learning theory, in part, because recent studies of school district central offices, including some studies of our own (see Chapter 1), had productively framed central office administrators’ participation in educational improvement as a challenge of central office learning. In our own prior research, we demonstrated how socio-cultural learning theory and organizational learning theory, in particular, describe work practices and activities consistent with the design of central office transformation in our three districts.

In particular, socio-cultural learning theory identifies specific practices involved in assistance relationships—relationships in which people work together to strengthen how they go about their work. These work practices called our attention to particular aspects of how central office administrators in each district worked directly with school principals to strengthen principals’ instructional leadership practice. However, we assumed that not all central office administrators would be engaged in such direct assistance relationships. Rather, other central office administrators would be trying to learn from the experience of their colleagues in those relationships and from the experience of schools more broadly to inform their own efforts to support teaching and learning in schools. Concepts from organizational learning theory (specifically trial-and-error learning or learning from experience) called our attention to the extent to which other central office administrators were searching for that experience-based evidence and using it to inform their decisions about their own work practices (for an elaboration of the specific concepts that anchored our conceptual framework for data collection, please see Honig, 2008).

Overall, these theories helped us move beyond the general notion that central offices should support school improvement to look for specific work practices and activities that might be involved in the implementation of such an idea. Given the dizzying array of activities involved in the three transforming central offices, our conceptual framework prompted us to make strategic and ultimately extremely productive choices to invest significantly in the collection of data about the work of central office administrators in each system who were positioned to support principals’ instructional leadership. Our conceptual framework also helped us move beyond simple descriptions of what other central office administrators were doing to probe more deeply into how their work related, if at all, to the principal learning support relationships.

Data Collection

Our data collection methods involved observations, interviews, and document reviews. Data was collected by a team of researchers primarily during the 2007–08 school year (and in one case, began the preceding spring), through repeated visits to the sites, supplemented by ongoing data collection by on-site data collection staff.

Observations

Observations of how central office transformation unfolded in real time proved especially important to the study’s attempt to focus on central office administrators’ daily work. In each school district we took advantage of different observation opportunities. In Atlanta, we sampled a selection of central office administrators’ work days and in each instance shadowed them throughout the entire day. During shadowing observations we wrote notes in long hand and, periodically throughout the day, typed elaborated field notes that included mostly rich description punctuated with direct quotes. Because we were trying to capture the full breadth of central office administrators’ work, we wrote highly descriptive notes about all the activities we observed while shadowing. When possible, we taped conversations between central office administrators and school principals that happened to occur during our shadowing observations.

In New York City, the Empowerment Schools Organization (ESO) convened key central office staff (the network leaders who worked in direct learning partnerships with principals) twice each month for at least two hours to discuss the central office transformation process and to engage ESO staff in conversations about how to improve the quality of their work to support principals. We contracted with two field researchers who observed virtually all of these meetings during our study period and produced verbatim transcripts of conversations during those meetings. The notes were so detailed that most totaled over 50 single-spaced pages per meeting. Our New York field researchers also sampled meetings of other central office administrators and a council of school principals convened by ESO leaders to provide input on the central office’s work work.

Similarly, in Oakland we regularly observed a series of central office meetings relevant to the central office transformation process. An on-site field researcher constructed verbatim transcripts of virtually all twice-monthly meetings of the Education Leadership Organization (ELO), which convened the Network Executive Officers (NExO) who worked directly with school principals on their instructional leadership practice. One formal purpose of the ELO meetings was to engage NExOs in conversations with each other as well as with senior central office administrators to inform practice and the overall central office transformation process. We also observed monthly “Coordinators” Meetings at which the NExOs, senior central office administrators, and central office operations staff addressed mainly operational issues as part of the central office transformation process. In addition, between November of 2007 and June of 2008, we observed 25 meetings of principal networks convened by six of the eight NExOs. We observed between three to 10 meetings convened by each NExO that ran between two-six For district leaders who want to engage in central office transformation, hours in length. During these observations the field researcher as well as members of our main research team constructed verbatim transcripts of conversations during the meetings as well as descriptions of activities.

Interviews

We conducted in-depth, semi-structured interviews with central office administrators, school principals, and representatives from outside organizations involved in, or otherwise in a position to comment on, the central office transformation effort (e.g., school reform support organizations, foundations). We interviewed central office staff most centrally involved in the central office transformation process an average of three-four times per respondent. In all, we conducted 282 interviews with 162 respondents. In our interviews we probed for concrete, albeit self-report, data on central office administrators’ actual work experiences. For example, three times during our study period we asked each of the Instructional Leadership Directors to walk us through their calendars for the preceding weeks and describe how they had been using their time during that period. We asked for specific examples of reported practices and activities and for evidence of how typical their practices and activities were. In interviews with school principals, we focused on confirming or disconfirming central office administrators’ reports and our observations of central office administrators’ work with school principals.

Documents

Various documents helped us understand the design of the central office transformation effort at each site, communication and other dimensions of the relationships between central office administrators and school principals, and how central office administrators worked with school principals to improve their instructional leadership practice. For example, central office administrators generated curricular materials or material we refer to as “tools” (see Chapter 2). We focused our collection of documents on those that provided some evidence of how central office administrators worked with schools on teaching and learning improvement as well as documents that described the overall central office transformation process and its underlying rationale.

Data Analysis

We coded our data using NVIVO 8 software in several phases over 16 months of carefully scrutinizing our data for reliable patterns. During our initial coding phase we distinguished all data by type of data source to help us triangulate our findings and track developments over time. Our main analytic work in this phase involved coding data using an initial set of broad and relatively low-inference codes derived from our conceptual framework. For example, socio-cultural learning theory called our attention to whether or not central office administrators working directly with school principals modeled instructional leadership practice. While we had the theoretical definition of modeling in view during this coding phase, we did not carefully distinguish modeling from other forms of coaching. We created broad categories to separate out how other central office administrators participated in implementation. During this phase we also coded for any data that seemed to relate to the outcomes of the central office transformation process using simple categories to distinguish outcomes for principals, schools, and the district overall. We cast a broad net when coding for outcome data and included any evidence of the results of the central office transformation efforts (e.g., information regarding process outcomes such as the level of resources provided to principals for their instructional leadership; data concerning status outcomes such as changes in actual leadership practice).

In the second, “recoding” phase, we went back into our data, this time through the codes used in the first phase, and further scrutinized whether or not the data coded in the first phase actually fit the construct and we recoded those data using codes at a higher level of inference. For example, in this phase we looked carefully at all our data categorized as “modeling principals’ instructional leadership practice” and distinguished evidence that specifically fit the definition of modeling in our conceptual framework. In a third phase, which we called “reducing,” we took another pass through our data set to collapse redundant categories and eliminate categories whose points we could not substantiate with at least three different data sources (either a combination of interviews, observations, and documents, or self-reports of at least three different respondents). By this phase we had identified the five dimensions of central office transformation that we use to organize our findings.

During this phase we also linked central office work practices with outcomes when we could justify those associations with at least three different data sources. We linked different work practices with different outcomes in the following ways which we also summarize at the start of each chapter in the findings:

• Dimension 1: Identifying more and less promising practices in the ILDprincipal partnerships. In our findings about the central office-school principal learning partnerships, we distinguish practices that we argue actually had, or promised to have, a greater positive impact on principals’ instructional leadership from those that were likely to have less impact. To distinguish those practices, we first used our conceptual framework to sort data about how the ILDs worked with school principals. As noted above and elaborated in another publication (Honig, 2008), the research from which we derived that conceptual framework shows that those practices have demonstrated power for helping other professionals improve their work practice across a variety of workplaces and arenas. Accordingly, we viewed those practices as indicating potentially high-impact practices. We corroborated those practices with our first-hand observations of ILDs’ practice in Atlanta and Oakland and our extensive observations of how ILDs in New York City talked about their practice with colleagues in meetings. We found that all but one of the concepts from our conceptual framework helped us distinguish among ILDs’ work with principals, and we added a category not anticipated by our conceptual framework (i.e., “differentiating supports”) to help capture an additional dimension of our findings. While no one ILD demonstrated all of those practices at a high level with all their principals, in each system we were able to distinguish among ILDs who engaged in those practices most of the time, often, or seldom/not at all.
  
  We corroborated those distinctions with data from interviews. Specifically, in our interviews we collected reports from school principals about the quality of supports they believed they received from their ILDs as well as reports from other central office administrators about the relative strength of individual ILDs in supporting principals’ instructional leadership development. These reports to a person confirmed the distinctions we made between higher and lower potential ILDs based on our earlier analysis using our conceptual framework.
• Dimension 2: Identifying more and less promising practice in central office support for ILD-principal partnership work. In this dimension, concerning activities of other central office administrators engaged in support of the ILD-principal partnerships we distinguished more and less promising supports based on our observations of other central office administrators’ actions that (1) engaged the ILDs in challenging conversations about their own work with individual school principals and how to improve the quality of that work, or (2) helped ILDs maximize the time they spent on support for principals’ instructional leadership. We triangulated our observations with ILDs’ reports of the extent to which other central office administrators challenged them to improve the quality of their work with school principals or helped them maximize their time with school principals on principals’ instructional leadership. We verified our assumptions about how ILDs were using their time with principal interviews about the frequency of their interactions with their ILDs and reviews of a random sample of ILDs’ calendars reviewed as part of interviews three times over the course of the study period.
• Dimension 3: Identifying more and less promising activities by the rest of the central office to focus their work on the improvement of teaching and learning. With this set of findings we make claims about particular activities that seemed promising for focusing the work of the rest of the central office on teaching and learning improvement. We derived the claims in this subsection about what other central office administrators were doing based on multiple interviews with central office administrators about the nature of their daily work, drawing conclusions only if we could corroborate self-reports with reports of three different respondents or at least three different data sources (e.g., an interview, documents, and observations). We considered those activities promising for focusing other central office units on teaching-and-learning support if: (1) central office administrators could provide an explicit rationale or explanation for why specific reorganization and reculturing activities mattered to teaching and learning improvement in schools, or (2) if they demonstrated that the reorganizing and reculturing activities had resulted in additional teaching and learning resources in schools (such as freeing up principals’ time for instructional leadership).
• Dimension 4: Identifying stewardship practices that were likely to foster and sustain the central office transformation process. In this dimension we make various claims about what stewardship of the central office transformation process involved. In distinguishing what specific stewardship practices fostered the central office transformation effort, we again triangulated observations and self-reports and inductively narrowed down a set of activities that we and our respondents consistently identified as supportive of the development and implementation of the overall transformation process.
• Dimension 5: Identifying prevalent and important practices in the use of evidence throughout the central office. Our claims in Dimension 5 relate to the prevalence of particular forms of evidence use. We based our claims about what these evidence-use activities involved mainly from observations of central office meetings as well as documents corroborated by interviews. We claim that these activities are important to central office transformation based on: (1) the research on organizational learning in our conceptual framework that shows how the collection and use of information from experience can help organizations realize their goals, and (2) observations and interviews in each site that demonstrated how integral such evidence-use activities were to the implementation of the other dimensions of central office transformation.

< < Previous | Next > >