B. Technical Capabilities of the State Education Agency |
| Underdeveloped | Workable | Ideal |
B1. Data and data system requirements |
Program data system: System that collects program
data (e.g., number of applicants, clinical hours required, 100-word description) from preparation
programs | Data are available in isolated locations without an overarching system for integrating the different sources or linking the data points. | A program data system is in place, but it may not include all data points needed for the SEA's annual report; some data may be missing, inaccurate, or lack comparability. Systems support might be needed to design new tools or interfaces to collect needed information from multiple sources and/or agencies. Substantial budgeting would be required for staff time to request missing data, monitor data completion, and build necessary data systems. Time is allocated to ensure data integrity. | A program data system is in place and includes all fields/variables needed for the SEA's annual report. The system enables consistent reporting and data aggregation. Data are complete and accurate. Programs use common definitions of indicators, making the data comparable across programs. The system is not overly burdensome for programs, districts, or school partners. |
Placement data systems: Systems that track individual
educators and their annual placement role (teacher, principal, assistant principal, other school leader, district leader) | Data are available in isolated locations without an overarching system for integrating the different sources or linking the data points. | Placement data systems exist and are coordinated but have lots of inaccuracies and missing data. Budgeting would be required for staff time to request missing data and monitor data. Time is allocated to clean data. | Placement data systems are complete and accurate. |
Unique identifiers for program participants: Identifiers that link data from preparation programs, licensure status, placement data systems, and effectiveness ratings from educator evaluation system | It is not possible to link individuals across data systems (for programs, licensure, placements, school outcomes). | Unique identifiers do not exist, but it is possible to link two or more data systems, and the SEA has the capacity to do this. Budgeting would be required for junior analyst time to link data systems. | Unique state-level identifiers are in place to link individuals to all of the data required by the evaluation system. |
Comparable survey data: Common survey administered to program graduates that gathers their perceptions of program
process indicators | Graduates of most programs are not surveyed, or the response rates are too low to make results meaningful. | Surveys of program graduates exist, and response rates are reasonable, but the surveys differ, preventing comparison of data across programs. | A common survey is administered to all graduates in the state with reasonable response rates, enabling comparison of data across programs. |
Measures of teacher and leader effectiveness: Ratings of individual teachers and principals on the state performance evaluation system | Measures of teacher and leader effectiveness: Ratings of individual teachers and principals on the state performance evaluation system. | Measures exist and have some variability but lack validity and reliability. The SEA does not share results publicly and does not provide caveats that caution users on interpretation. | Measures exist, have variability, and have been found to be both reliable and valid. The SEA has the capacity to use measures in contextually appropriate ways. The SEA ensures that any public release of data meets federal and state privacy guidelines. |
Measures of student learning gains: Student achievement scores across grade levels in core subject areas | Measures are not based on individual student growth from year to year. | Consistent and methodologically sound measures of individual student growth, including proper controls for student- and school-level variables, exist, but they are not comprehensive across grade levels and subject areas. Measures exist, but ns are small (less than 10 individuals) for most programs. | Consistent and methodologically sound measures of individual student growth, including proper controls for student- and school-level variables, exist. These measures allow for assessment of school leaders' influence
on student learning after three years at a school site. Adequate consideration is given to bias against high-needs schools. |
B2. Data compilation and analysis capacity |
Monitoring data reporting completion and accuracy: Requires staffing to ensure the submission and accuracy of data from preparation programs and other data sources | No staff or resources exist. | Staff assignments and/or resources could be prioritized for data monitoring. | Staff and/or resources are already assigned to data monitoring. |
Creating and publishing annual reports: Requires technical skill for website/report design and senior analytical skill to make methodological decisions | No staff or resources exist. | Staff assignments and/or resources could be prioritized for data reporting. | Staff and/or resources are already assigned to data reporting. |
Creating and implementing methodology for summative rating: Requires specialized assessment and statistical skill | No staff or resources exist. | Staff assignments and/or resources could be prioritized for data analysis/methodology. | Staff and/or resources are already assigned to data analysis/methodology. |
B3. Review process capabilities |
Staffing: Requires specialized leadership experience and skills | There is no SEA staff committed to leadership
preparation, or those responsible have multiple other roles. | There are staff members at the SEA focused on school leadership, including preparation, but they have limited backgrounds in school leadership or adult leadership. | There are staff members at the SEA focused on school leadership, including preparation, and they are deeply credible with leaders and preparation providers in the state. |
Management and training of reviewers: Requires specialized review process capabilities | No staff or resources exist. | The state has a reasonably adequate pool of high-quality, credible reviewers but does not have a track record of systematically vetting them for leadership expertise or training them for inter-rater reliability. The state does not have a strong track record of outsourcing functions and maintaining quality. | The state has a robust pool of high- quality, credible reviewers who have been (or could be) trained for inter-rater reliability and normed to provide useful feedback to programs. or The state has a strong track record of outsourcing
functions and maintaining quality. This allows for bringing in an established process (e.g., review by the Educational Leadership Constituent Council). |
Implementation of reviews: Requires financial
and human resources | No staff or resources exist. | A review process exists, but it is not sufficient
for quality, in-depth review of all programs flagged. | Sufficient resources exist to carry out in-depth reviews for all programs flagged as needing it, and for conducting periodic reviews of all programs. |