 |  |  |
|
|||
|
|||
|
|||
| |
||
| |||
 |
         |
|
|
Paper
Highlights of the Year Three LSC Cross-Site Report
December 1998
Table of ContentsAcknowledgments Introduction Participation in LSC Activities Quality of LSC Professional Development Programs Current Status of Teachers' Beliefs and Practices Impact of LSC Activities Supportiveness of Context Sustainability of Reform Summary and Recommendations AcknowledgmentsThe core evaluation of the Local Systemic Change Initiative requires the energy, efforts, and insights of a very large number of people.Conrad Katzenmeyer in the National Science Foundation's (NSF) Division of Research, Evaluation, and Communication; Joyce Evans, Susan Snyder, and Diane Spresser in NSF's Division of Elementary, Secondary, and Informal Education (ESIE); and Joy Frechtling of Westat, Inc. were instrumental in the design and implementation of the core evaluation system. Numerous other NSF/ ESIE program officers provided valuable assistance throughout the process, as well. Within HRI, in addition to the authors of this report, Eric Banilower, Alison Bowes, Gail Gellatly, Elizabeth Hammond, Scott Hanrath, Susan Hudson, Ben Kurgat, Joan Pasley, Sheila Richmond, Sean Smith, Eugene Soar, and Claudia Templeton were responsible for various tasks essential to the production of this report, from instrument design, to data processing and analysis, to report production. David Flora and Abigail Panter of the L. L. Thurstone Psychometric Lab at the University of North Carolina at Chapel Hill assisted in the data analysis. Most notably, this report would not have been possible without the efforts of the LSC project evaluators, whose work formed the basis for the analyses presented here. And special thanks are due to the thousands of teachers throughout the 263 participating districts who took time from their busy schedules to provide information about their mathematics and science teaching. Introduction In the spring and summer of 1995, the National Science Foundation (NSF) funded the first cohort of eight projects in a new initiative, the Local Systemic Change through Teacher Enhancement (LSC) program. Eighteen additional projects were funded in 1996 and 20 more in 1997, for a total of 46 projects in Cohorts 1, 2, and 3. The goal of the LSC program is to improve the teaching of science, mathematics, and technology by focusing on the professional development of teachers within whole schools or school districts. Each targeted teacher in a K- 8 project is to participate in a minimum of 100 hours of professional development; for projects targeting teaching in grades 6- 12, the minimum is 130 hours over the course of the project. In addition to its focus on involving all teachers in a jurisdiction, the LSC initiative is distinguished from previous teacher enhancement efforts by its emphasis on preparing teachers to implement designated exemplary mathematics and science instructional materials in their classrooms. LSC projects are expected to align policy and practice within the targeted district( s) and to include:
The LSC solicitation indicated NSF's plan to "provide a framework for data collection (including a set of instruments and procedures) that will allow the Foundation to evaluate individual projects, aggregate data and information across projects, and produce a cross- project analysis" (NSF 94-73). NSF contracted with Horizon Research, Inc. (HRI) of Chapel Hill, NC to design the data collection framework, provide technical assistance in its implementation, and prepare a cross- site analysis of the evaluation results. An Overview of Cohorts 1, 2, and 3 Project data sheets completed by the principal investigators (PIs) provide some basic information about the 46 LSC projects included in Cohorts 1, 2, and 3.
Description of Core Evaluation Data Collection and Analysis HRI has worked with the National Science Foundation and PIs and evaluators of the LSC projects on the design and implementation of a core evaluation system to allow aggregating information across projects in response to six core evaluation questions.
Data collection activities for the projects' 1996- 97 Core Evaluation Reports were conducted from September 1, 1996 through August 31, 1997. Cohort 3 projects were collecting baseline data for their first year of funding; this was the second year of data collection for Cohort 2 projects and the third year for Cohort 1 projects. Data collection activities included the following: 1. Observations of professional development activities The core evaluation calls for projects to conduct 5- 8 observations of professional development sessions each year and record their observations on standardized protocols. Evaluators were to consult with PIs on what professional development experiences were planned throughout the data collection year, and select a sample that was representative of the diversity of the project's activities. Program- wide, a total of 276 observations of professional development sessions were conducted. 2. Classroom observations HRI provided the lead evaluator of each project with a list of 10 randomly selected teachers for each targeted subject. These teachers were to be observed in the spring of 1997. There was a total of 517 classrooms observed, including 299 classes taught by teachers who had participated in at least 20 hours of LSC professional development, and 218 classes as baseline for Cohort 3 projects. 3. Teacher questionnaires Each project was asked to administer teacher questionnaires developed for the core evaluation to a sample of 300 teachers per targeted subject; the median response rate was 84 percent. A total of 10,054 teacher questionnaires were returned to HRI, including 6,126 from K- 8 science teachers; 2,347 from K- 8 mathematics teachers; and 1,581 from 7- 12 mathematics teachers. 4. Principal questionnaires Projects were also asked to administer questionnaires to the entire population of principals of targeted schools. Return rates on the principal questionnaire were generally higher than for the teacher questionnaire, with a median response rate of 92 percent; a total of 1,905 principal questionnaires were returned. 5. Teacher interviews Evaluators of each Cohort 1 and Cohort 2 project were asked to interview a sample of 10 teachers who had participated in at least 20 hours of professional development activities in that project. A total of 249 interviews were conducted among the 26 projects. About two- thirds of the interviews were conducted by phone, and the remaining one- third in person. Project evaluators were asked to report their findings using guidelines developed for the core evaluation system, including responding to the six core evaluation questions. Evaluators were also asked to provide overall ratings of the quality of professional development activities, the supportiveness of the context, and the sustainability of high- quality professional development systems. Participation in LSC Activities LSC projects are responsible for serving all of the teachers who teach the targeted subject and grade range, rather than only those who volunteer to participate. Data on participation in LSC professional development activities come from teacher and principal questionnaires. Evaluators report that most LSC projects are on schedule for providing in- depth professional development to all teachers in the targeted subjects and grades. To date, the population of elementary teachers participating in LSC professional development activities closely mirrors the overall targeted population in these districts in terms of teacher gender, race/ ethnicity, and course background preparation. In contrast, secondary mathematics projects appear to be starting with relatively less well- prepared teachers in the middle rather than the high school grades. Based on their responses to core evaluation questionnaires, a substantial proportion of principals is neither involved in nor knowledgeable of LSC activities. However, a number of evaluators pointed out that principals they know to have been involved indicated otherwise on the core evaluation questionnaires, perhaps not associating those activities with the term "LSC program." A more serious concern is the fact that a few projects appear to have redefined targeted teacher population to mean those who are willing to participate, which is clearly inconsistent with the intent of the LSC initiative. Quality of LSC Professional Development Programs Project evaluators were asked to observe 5- 8 professional development activities for the core evaluation. Evaluators and PIs were to decide, jointly, which activities would be observed, selecting sessions to represent the diversity of the project's professional development program and to reflect the extensiveness and importance of the various activities offered. Description of Observed Sessions Evaluators documented a number of descriptive features of each professional development session, providing information across all projects about targeted participants, presenters/ facilitators, purposes and content focus, and the major types of activities that characterized the sessions. The typical professional development session observed for the LSC core evaluation had between 21 and 50 participants. Some sessions exclusively targeted lead teachers (22 percent); other sessions targeted non- lead teachers (47 percent); and some sessions targeted both lead and nonlead teachers (27 percent). A total of 12 percent of the sessions included principals or other administrators. LSC professional development involves presenters/ facilitators from a variety of settings. Lead teachers served in this capacity in 40 percent of the observed sessions, while only 27 percent of the sessions included university faculty as presenters or facilitators. Two- thirds of the presenters/ facilitators were female, 87 percent were white, 8 percent African- American, 3 percent Hispanic, and 2 percent Asian or Pacific Islander. Evaluators were asked to indicate the major intended purposes of each observed session based on information provided by the session facilitators. As can be seen in Table 1, the most frequently cited purposes were working on classroom pedagogy issues and enhancing teachers' understanding of science/ mathematics concepts. Table 1
* Percents add to more than total for category because sessions could include more than one purpose. When sessions focused on one or more disciplinary content areas, evaluators were asked to categorize that content. In K- 8 science projects, evaluators reported that nearly half of the sessions that had a disciplinary content focus dealt with physical science concepts (46 percent), nearly that many with life science content (41 percent), and somewhat fewer with concepts from the earth and environmental sciences (32 percent). In projects targeting K- 8 mathematics, the most heavily emphasized topics were patterns and relationships (32 percent of the sessions that dealt with disciplinary content), numeration and number theory (29 percent), and data collection and analysis (23 percent). The most commonly emphasized topics in sessions for 7- 12 mathematics teachers were patterns and relationships, algebra, and geometry/ spatial sense, each the focus of about 30 percent of the sessions that dealt with disciplinary content. Most of the observed LSC professional development sessions included several different instructional strategies. Most sessions included discussions or seminars (83 percent). Roughly two- thirds of sessions included formal presentations and a similar proportion included investigative/ problem solving activities. Observer Ratings of LSC Professional Development Sessions In order to assess the quality of professional development sessions, observers were asked to rate a number of components for each session they observed. 1. Design For design of the professional development sessions, the indicators that most often received high ratings were: the extent to which the session design reflected careful planning and organization; how well the session encouraged a collaborative approach to learning and incorporated tasks and interactions consistent with a spirit of investigation; and the appropriateness of strategies used in the session for accomplishing the purposes of the LSC professional development. 2. Implementation In the area of implementation of professional development activities, indicators most frequently rated high were: whether the facilitator's background and expertise or management style enhanced the quality of the session and the extent to which the session incorporated instructional strategies appropriate for its purposes and the needs of adult learners. 3. Content Observers were asked to rate either the quality of the disciplinary content of the observed session, its pedagogical content, or both, depending on the focus of the session. Disciplinary content was rated in 176 of the 276 sessions; these sessions were most likely to receive high ratings for: the appropriateness of the disciplinary content for the purposes of the session and the background of the participants; the extent to which the facilitators displayed an understanding of mathematics/ science content; and the soundness and appropriateness of mathematics/ science content. Observers rated 198 of the 276 observed professional development sessions on the quality of their pedagogical content. As was the case for quality of disciplinary content, these sessions tended to be highly rated for: the appropriateness of the pedagogical content for the purposes of the session and the background of the participants and the extent to which the facilitator displayed an understanding of pedagogical concepts. 4. Culture The literature on effective staff development emphasizes the importance of establishing a professional development culture where teachers can explore content and pedagogy in a collegial, risk- free environment, and most observed LSC professional development sessions were successful in this regard. Indicators that were most likely to receive high ratings included: whether active participation of all was encouraged and valued; the extent to which there was a climate of respect for participants' experiences, ideas, and contributions; the extent to which interactions reflected collaborative working relationships among participants and between facilitators and participants; and whether participants were intellectually engaged with important ideas relevant to the focus of the session . 5. Leadership Most of the LSC projects have chosen to develop teacher leaders as part of their strategy for reaching large numbers of teachers, and 43 of the 276 observed sessions focused specifically on leadership content, such as planning and implementing effective professional development sessions. These sessions were most likely to receive high ratings for: the extent to which the facilitators displayed an understanding of leadership concepts; whether the leadership content was sound and appropriately presented; and the extent to which the leadership content was appropriate for the purposes of the session and the backgrounds of the participants. Overall ratings for individual professional development sessions were quite favorable. Only 1 percent of observed LSC sessions were rated as ineffective professional development (Level 1), and 7 percent were rated at Level 2, having quite limited likelihood of helping participants implement exemplary mathematics/ science instruction or be leaders in reform. Overall, 71 percent of the observed professional development sessions received ratings of 4 or 5, indicating that those sessions were skillfully facilitated, engaging participants in purposeful work that would likely lead to enhanced capacity to implement exemplary instruction. (See Figure 1.)
Figure 1 Teacher Ratings of LSC Professional Development Programs While observations were the main source of information on the quality of LSC professional development, information was also gathered from the participating teachers via questionnaires and interviews. On teacher questionnaires, LSC participants described the extent to which a variety of features characterized the LSC professional development, rating each item on a scale ranging from 1, "not at all" to 5, "to a great extent." Table 2 shows the percent of teachers rating the LSC professional development high (4 or 5 on the five- point scale), medium (a rating of 3), and low (a rating of 1 or 2) on each of several items. Note that the LSC received highest marks for providing both opportunities for professional development and support for teachers as they implement what they have learned; still only about half of the participating teachers gave the LSC ratings of 4 or 5 on these items. Table 2 Teacher Ratings of LSC Professional Development*
* Only teachers who had participated in LSC professional development were included in these analyses. It is interesting to note that the more time teachers had spent in LSC professional development, the more highly they rated it. (See Figure 2.)
Figure 2 Evaluators' Overall Ratings of the LSC Professional Development Programs Using the observation results, as well as teacher questionnaire and interview data, evaluators then rated the overall quality of the professional development program, noting particular strengths and areas in need of additional attention. Evaluators indicated the following as key strengths in the programs:
Evaluators also mentioned areas that needed further attention:
It is interesting to note that while observers typically gave high ratings to the quality and likely impact of individual leadership development sessions, they were less impressed with the overall quality of the project's handling of leadership content. Only 40 percent of the projects were rated highly (4 or 5 on a five- point scale) with regard to leadership content, compared to more than 70 percent of projects receiving high ratings in each of the other component areas. Evaluators identified a number of key components that were important for effective professional development for teacher leaders. They included:
As a culminating task in their analysis, evaluators were asked to place each project on a five- level continuum describing the quality of the professional development program. As can be seen in Table 3, none of the 46 projects was rated at Level 1 (Predominance of Ineffective Professional Development), and only 1 project was considered to be at the Level 2, "Exploring" stage. Nine projects were rated at Level 3 (Transitioning to Quality Professional Development). Twenty- six projects were rated at Level 4 (Emerging Infrastructure of Well- Designed Professional Development), indicating that their professional development plan and activities incorporated many features reflective of current standards- based approaches, that the professional development activities were well- implemented, and that in the evaluator's judgment they will likely enhance participants' capacity to provide high quality mathematics/ science instruction to their students. The final 10 projects (22 percent) were rated at Level 5, indicating consistently excellent quality. Table 3
* Projects that address two subject areas are included in each subject, but counted only once in the total of all projects.
|
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
