Paper

Core evaluation data to assess the quality of LSC professional development activities come from observations of the professional development activities, and to a lesser extent, from teacher questionnaires. Cohort 1 projects augmented these data with teacher interviews. Based on these data, evaluators were asked to rate the quality of the project's professional development in eight component areas and to provide an overall rating of the project's professional development system.

Cohort 1 projects reported on their second year of implementing LSC professional development activities. These projects all target elementary science teachers, and are analyzed in a single group, referred to as C1S. For the newly-funded Cohort 2 projects, these 1996 data represent the status of professional development during the first year of LSC funding¹. Cohort 2 includes both projects that target elementary science (C2S) and projects that target elementary mathematics (C2M).

Teacher Ratings of District-Provided Professional Development

The LSC teacher questionnaire included several questions about the quality of the science/mathematics professional development available in their district. There was no attempt to focus on LSC professional development programs, but rather teachers were asked to think about district-provided professional development generally.

• Involved in planning their professional development;
• Encouraged to develop an individualized professional development plan;
• Given time to work with other teachers and to reflect on what they have learned; and
• Received follow-up support.

Ratings on this composite are relatively low and show a lot of variation among projects; project means ranged from 43 to 65 percent of total possible points. (See Figure 6.)

Figure 6

Roughly one-fourth to one-third of teachers in each cohort reported that they were fairly heavily involved in planning their own science- or mathematics-related professional development and encouraged to develop individual professional development plans to address their needs and interests. Only about 20 percent of the teachers indicated that they are given ample opportunities to work with other teachers or reflect on how to apply what they have learned to the classroom as part of their professional development. Roughly 1 in 3 Cohort 1 teachers, but only 1 in 5 Cohort 2 teachers reported that they were given the follow-up support they needed to implement what they had learned in professional development experiences. (See Table 2.)

Given current standards for best practice in professional development, there is a great deal of room for improvement in the quality of district-provided professional development in science and mathematics. In the following sections, data from evaluators' observations of LSC professional development activities are used to describe how LSC projects are addressing this challenge.

Description of LSC Professional Development Activities

Evaluators and PIs were asked to decide jointly which project activities would be observed for the core evaluation, selecting sessions to represent the diversity of the project's activities and reflect the extensiveness and importance of the various activities offered. LSC evaluators observed a total of 180 professional development sessions, an average of seven observations per project. The following description of LSC professional development activities is based on these sessions.²

Participants

The typical professional development session observed for the LSC core evaluation had between 21 and 50 participants, mostly teachers from a range of the elementary grades. As can be seen in Figure 7, Cohort 1 projects continued to focus on preparing lead teachers (almost three-quarters of Cohort 1 science sessions included lead teachers); far fewer Cohort 2 sessions included lead teacher participants (36 percent). Overall, 7 out of 10 observed professional development sessions included regular (non-lead) teachers and fewer than 2 in 10 included administrators as participants.

Figure 7

Presenters/Facilitators

Typically, LSC professional development sessions were conducted by a team, sometimes including lead teachers, but more frequently composed of other personnel both from the district and from outside the district. Almost 90 percent of the observed sessions had at least one female presenter or facilitator, and 17 percent included non-white women. A little over half of the sessions had male presenters, in a few cases including African American or Hispanic males.

Session Characteristics and Foci

Evaluators were asked to indicate the purposes of each professional development session they observed, choosing from twelve possibilities; for most sessions, evaluators checked from 2 to 4 purposes. Nearly half of the observed sessions focused on helping participants create a vision of learning by inquiry, try out activities or curricula for classroom use, or learn science/mathematics concepts. (See Table 3.) Almost 40 percent of sessions centered on working with teachers on pedagogical/classroom management strategies; about a fifth of observed sessions concentrated on building a professional network among educators or developing participants' leadership capacity.

Nearly half of the observed science sessions included physical science content, 37 percent focused on life science, and 33 percent had an earth-science content focus. (See Figure 8.) Among projects targeting mathematics, the content areas of focus were most often computation and estimation, numeration and number theory, patterns and relationships, and mathematical problem solving. While a few of the science sessions included emphasis on one or more mathematics topics (especially measurement), none of the mathematics sessions included science content as a major focus. (See Figure 9.)

Figure 8

Figure 9

As can be seen in Figure 10, when asked to describe the "pedagogical content" of the observed session, evaluators in each cohort noted that over half of the observed sessions emphasized instructional strategies. Somewhat fewer sessions focused on providing visions of effective science/mathematics instruction and even fewer on how students learn. Professional development sessions in Cohort 1 were much more likely than those in Cohort 2 to include consideration of assessment strategies and issues related to access, equity, and diversity.

Figure 10

Most LSC professional development sessions included a formal presentation by a facilitator or other presenter. Teachers were involved in field activities, research, or other hands-on activities during 70 percent of the observed sessions; discussions or seminars occurred at a similar number of sessions. Very few sessions (12 percent) involved participants actively exploring uses of technology. (See Table 4.)

Evaluator Ratings of the Quality of the LSC Professional Development

Evaluators' overall ratings of the LSC professional development indicate that LSC projects are generally providing high-quality staff development. Each project evaluator was asked to rate the project's professional development system holistically, considering all of the data available, and place their LSC project on a five-level continuum. (See Table 5.)³

Ratings for the two cohorts were quite similar, with mean ratings of 3.8 on the 5 point scale. Eleven out of 25 projects were rated at Level 3 ("Transitioning to Quality Professional Development") by their project evaluators.⁴ All of the remaining projects were rated even higher on the continuum, indicating that their professional development plan and activities incorporated many features reflective of current standards-based approaches, that the professional development activities were well-implemented, and that in the evaluator's judgment they will likely enhance participants' capacity to provide high quality science/mathematics instruction to their students.

In addition to overall continuum ratings, evaluators provided ratings for each of seven component areas for professional development, including: the design of activities, their implementation, the disciplinary and pedagogical content, the professional development culture, the likelihood that the professional development would build participants' capacity to provide exemplary science/mathematics instruction, and the likelihood it would build their capacity to be leaders in science/mathematics instructional reform. The largest difference between the cohorts was in ratings for capacity-building for instruction, with Cohort 1 projects more likely to be highly rated in this area. (See Figure 11.) No evaluators rated a component area for their project at Level 1, and very few components were rated at Level 2.

Figure 11

Strengths of the LSC Projects

Evaluators were asked to describe their rationale for their ratings, particularly highlighting strengths of the professional development activities and areas in need of improvement. Aspects of the LSC professional development that were most often described by evaluators as strengths were:

• strategies for professional development;
• the skills, competence, and expertise of facilitators;
• the systemic focus on the broader science/mathematics education system;
• a culture conducive to and supportive of learning;
• relevant, high-quality content;
• models for effective development of teacher leaders; and
• reflection of project staff for program improvement.

Two features that distinguish this group of professional development projects from many past teacher enhancement projects are the systemic focus and the consistency of attention directed toward the use of exemplary curriculum materials. Evaluators commented frequently on these aspects of the LSC projects.

Evaluators of the LSC projects described a number of ways in which projects targeted the broader science/mathematics education system. For example, a requisite commitment for LSC funding is to serve all eligible teachers for a substantial number of hours. Many programs are choosing to begin by "exposing" large proportions of teachers to one or two kits and achieve at least a "mechanical" knowledge of the materials early on. One evaluator pointed out the benefit of this strategy as a step toward building support among stakeholders:

"Through the [teacher workshops], the Partnership is reaching larger numbers of teachers and providing exposure to inquiry-centered instructional approaches that can readily be used in elementary classrooms. These workshops, by increasing the number of teachers involved in the Partnership, are also expanding the cadre of stakeholders involved in the reform of science and mathematics."

Most projects are implementing curricula that the district has chosen or agreed to adopt, reinforcing the alignment of district curriculum and goals. One evaluator made this observation:

"[The district] has recently adopted FOSS (Full Options Science System) as its district science curriculum. Many of the pedagogy sessions and some of the content sessions encourage exploration of specific FOSS kits. By utilizing the district curriculum as a starting point for good hands-on science, [the project] is seen by participants as supporting the district-identified process to improve science teaching."

The LSC projects are also distinguished, as a group, by being highly reflective of current standards for best practice in professional development. Features described most often by evaluators of the LSC projects are described below:

• Relevancy of professional development activities to teachers' work in classrooms

  Evaluators found, for example, that activities in which teachers were engaged were closely linked to their needs as classroom teachers. Said one,

  "This professional development model establishes a different dynamic between practitioners and their professional growth. There is a relationship between what occurs inside and outside of their classroom. The school adopts math and science curricula considered exemplary and in the best interest of students, and complements the adoption with extended staff development to assist teachers in the implementation process."

  A strategy mentioned as particularly successful by another evaluator was allowing teachers to focus on grade-specific activities. This evaluator reported these teacher comments:

  "What I got a lot from in [the workshop] was when I sat with other fourth grade teachers and talked about how we implemented the kit"

  "The people who did kindergarten [sessions] were actual kindergarten classroom teachers so when they gave information it had been used and tested and worked. Those were very useful, very helpful."

• Modeling effective strategies

  Evaluators found that strategies advocated by the projects were modeled in professional development activities. Activities were characterized, for example, by active involvement of teachers and participatory hands-on activities; in many cases teachers worked on activities from modules that they would later use in their classes.

  Experiencing instructional strategies firsthand was particularly important for teachers being introduced to inquiry-based strategies, as described by this evaluator:

  "In order to teach in an inquiry-based environment, teachers must experience inquiry learning for themselves. The introductory summer institute provides the foundation upon which teachers must stand if they are to transform their teaching approaches to be truly inquiry-based. Two weeks enables participants to thoroughly explore a particular scientific phenomenon, and to come to conclusions of their own about why "x" works, drawing on their own experiential learning. In addition, teachers learn how a real classroom might function as an inquiry-based learning environment. This is often difficult to even imagine, given the relatively fixed structured nature of most elementary schools. In discussions and debates, the institute directors encourage participants to consider how to translate their experience back to their classrooms."

• Provides opportunities for teachers to reflect and talk with peers

  Time for reflection was incorporated into many of the projectsi professional development activities sometimes within individual sessions; sometimes as the major focus of entire sessions as in school-based peer group meetings to discuss the implementation of a kit. This comment is reflective of evaluators' (and participants') praise for projects' providing opportunities for reflection:

  "Overall, lead teachers, in interviews and in the post-institute evaluation surveys highly praised the academic year sessions-particularly noting the opportunities to share classroom practices, to practice inquiry, and to discuss national standards with peers. Through the [institute] lead teachers have opportunities for reflection, dialogue, and peer support."

• Strong components of support and follow-up

  Support throughout the academic year is a consistent feature of the LSC projects, and one that evaluators recognized as a strength. One evaluator, for example, highlighted the importance of providing opportunities for teachers to meet throughout the year for discussions with peers after actually trying out modules in classrooms.

  Projects have explored different strategies for providing on-going support to teachers. A number of projects that focus on kit-based curriculum have organized regular meetings in which teachers may gather to discuss the implementation of the kits. Some projects have worked to build the expertise of in-school teacher leaders who are available for individual support to their peers. Another follow-up strategy has been for teachers to develop personal action plans with support provided throughout the year by mentor teachers or project staff.

• Active involvement of teachers

  The LSC projects are clearly attentive to actively involving the teachers as key players in their own professional development. Evaluators described a number of mechanisms designed to engage teachers as active participants and leaders of professional development in their districts.

  A number of projects incorporate teacher-leader models designed to build capacity of teachers within the schools and districts, and to reach larger numbers of teachers. Other strategies for encouraging teachers' involvement include options for teachers to select from multiple professional development activities; facilitating active sharing, reflecting and collaboration in small groups; fostering school-based implementation built on "school-site learning communities"; and assembling action research teams that offer teachers the opportunity to network and share with each other.

Challenges Faced by LSC Projects

Interestingly, some of the same features highlighted as strengths by some evaluators were identified by other evaluators as areas of challenge for their projects. Aspects of the professional development most often described as needing increased attention for improvement were:

• Time for participant reflection;
• Attention to content;
• Attention to classroom application;
• Consistency in the skills and expertise of facilitators;
• Attention toward equity; and
• Teachers' use of available resources.

The overlap in features most likely to be mentioned as strengths when they are present and as areas in need of improvement when they are not include: time for reflection, attention to classroom application, high-quality content, and skill and expertise among facilitators. The following comments of evaluators are representative of the concerns expressed by some.

Regarding the need for increased time for reflection:

"Although there was active learning going on in this and other activities during the session, there were missed opportunities for reflection, individually and as a group on what was being taught and learned. That is, although teachers were asked good questions, for example comparing actual [data] with predictions and providing what they thought accounts for differences, the discussions tended to be very limited in time and scope."

"Complex ideas were demonstrated in an interesting way, but some of the teachers did not internalize the concepts. The minds-on portion of the workshop was missing. As a result, the teachers may have some difficulty developing assessment tools and questioning skills when introducing the concepts in their classroom. The workshop seemed more of a show than an integration of content, process, reflection and application. The teachers needed to reflect on the demonstrated concepts and perhaps discuss age-appropriate applications of the concept."

Regarding content-particularly finding an appropriate balance in the content and process of inquiry:

"To elevate the quality of the kit workshops still higher the focus needs to expand to include an emphasis not only on the procedures and organization of the kits, but on the science content and the conceptual development embedded within them. Sole focus on the mechanics of the kits does not allow teachers time to reflect on their teaching practice or the rationale for hands-on, inquiry-based science education. This may lead participants to view kit-based science teaching as an end rather than a means to exemplary inquiry-based science instruction."

Regarding attention to classroom application:

"The teachers needed more time to consider classroom applications of resources, strategies, and techniques introduced in the sessions. For example, in one of the sessions observed, the participants viewed a video called "A Private Universe" which illustrates how students often develop misconceptions about science concepts and suggests methods to identify and resolve these misconceptions. The teachers in this session, however, were unclear as to how to use this information in their own classrooms. The method demonstrated in the video, individual student interviews, was not an option for most teachers, and they needed time to discuss other strategies."

Regarding the skill and expertise of facilitators:

In some cases, lapses in quality of the professional development were due to facilitators whose expertise in content areas was greater than their knowledge of appropriate professional development strategies. In other cases the challenge was to achieve consistency among teacher leaders working with peers. For example, one evaluator made this observation:

"The quality of those sessions was somewhat variable. The presentations made in the fall for colleagues were exceedingly well-executed and reflective of best practice, keeping in mind that many of the activities had been modeled for them by [project staff] in their first summer of professional development. When they were on less familiar ground [the] sessions were not always as well-executed."

Regarding increased attention toward equity:

In addition to the general need for equity issues to "gain visibility", evaluators commented on the challenge of establishing equity as a priority for participants:

"A major focus of the professional development sessions during the summer and year is how to successfully implement the science curricula in a manner that is accessible to all students. In addition, this summer's institute had a series of workshops on cooperative learning as a direct attempt to address the issue of access for new and beginning teachers. Although [the project] is accomplishing some things in the area of equity, the associate superintendent for curriculum and instruction believes that much more needs to be done in this area as test scores for the African-American and Latino populations have not showed expected gains."

"One issue that must be dealt with in the future is the perception of teachers that they know all about equity already and they do not understand why time is spent on this topic when there are so many other content, assessment, and procedural topics that they would like to hear about in professional development sessions."

Regarding use of available resources:

Teachers' failure to use existing resources was described as a challenge for a number of the projects. In most cases the issue was that teachers were not taking advantage of opportunities available to them, as illustrated by these evaluators' comments:

"While monthly meetings, content courses, and [teacher] assistance is available to classroom teachers, many do not make use of these resources..."

"Generally, [the project] is providing high-quality professional development activities by teachers are not feeling supported in the classroom implementation of what they have learned. One area where [the project] attempted to address this need was the creation of kit clubs. Unfortunately, not many teachers attended these even though they would have been paid $20 for every two-hour after-school session they attended."

Teacher Ratings of the Quality of LSC Professional Development

As part of the core evaluation, evaluators of Cohort 1 projects were asked to interview a random sample of 10 teachers who had participated in the project's professional development activities. Among the teachers interviewed, over two-thirds of the group had received more than 30 hours of LSC professional development; the others had participated in 10 to 30 hours of professional development. As seen in Figure 12 the majority of teachers had quite positive impressions of the professional development in which they had participated.

Comments of participants who were most positive about the professional development indicate that high-quality curriculum materials were a particularly important component of the professional development. Teachers described curriculum materials as "complete and comprehensive," providing "easy access to materials" and access to "meaningful investigations." Teachers were also very positive about their experiences with inquiry-based science, the support provided through on-going activities, and strategies that involved lead teachers or core teams at their schools.

Figure 12

Even teachers who had overall very positive comments about the LSC professional development sometimes had suggestions for improvement. A number of comments focused on the need for more direct experience with the type of teaching they were striving to implement through the LSC project. A couple of participants commented, for example, that they would like to observe teachers using inquiry-based or constructivist strategies; another participant expressed the desire to have a visiting teacher come to his classroom to demonstrate these instructional strategies.

Only a small proportion of teachers who were interviewed were described by evaluators as either "somewhat" or "mostly" negative, and all of these teachers came from 3 of the 8 Cohort 1 projects. Those teachers who were negative about the professional development they had experienced complained of being unclear about expectations of the district and needing professional development that better met their particular needs with regard to content or pedagogical strategies.

Ratings of Individual Professional Development Sessions

Observations of selected professional development sessions at each LSC site are key to the core evaluation analysis of the quality of the professional development offered by LSC projects; although evaluators were asked to take other factors into account when making continuum and component ratings of their project, observations were the main standardized data source.

The following sections describe elements of professional development that observers were asked to rate for each professional development session observed, including:

• Design of the session;
• Its implementation;
• The disciplinary and pedagogical content;
• The character of the culture of the session;
• The likely impact on participants' capacity for exemplary science and mathematics instruction; and
• The likely impact on participants' leadership capacity.

Each section includes details about the pertinent indicators that contributed to evaluators' ratings of the individual professional development sessions observed for the core evaluation, as well as the percent of sessions receiving overall ratings of 4 or 5 on a 5-point scale.

Design of Professional Development

Evaluators were asked to assess the design of professional development sessions by rating a series of individual indicators based on current understandings of best practice. Several of these indicators received high ratings (4 or 5 on a five-point scale) in the majority of observed sessions. Those indicators that were most often highly rated included the following:

• The appropriateness of strategies used in the session for accomplishing the purposes of the LSC professional development (78 percent);
• How well the session encouraged a collaborative approach to learning and incorporated tasks and interactions consistent with a spirit of investigation (75 percent); and
• How effectively the session built on participants' knowledge of content, teaching, learning, and/or the reform/change process (71 percent).

Somewhat fewer sessions were rated highly on:

• How well the strategies reflected attention to the varied needs and learning styles of the participants (65 percent); and
• Whether the design of the session provided adequate time and structure for reflection (53 percent).

Figure 13

Evaluators were also asked to provide an overall rating for the design of the session. As can be seen in Figure 13, 74 percent of the professional development sessions received overall design ratings of 4 or 5.

Implementation of Professional Development

Evaluators rated several indicators regarding the implementation of the professional development session. Ratings on the implementation tended to be lower than those on design. The indicators most frequently rated 4 and 5 were:

• Whether the facilitator's background and expertise enhanced the quality of the session (82 percent); and
• The appropriateness of the pace of the session for adult learners (69 percent).

Fewer LSC professional development sessions were rated highly on other indicators, including:

• How well the session modeled questioning strategies that are likely to enhance the development of conceptual understanding (60 percent); and
• How well the session modeled effective assessment strategies (42 percent).

Figure 14

As can be seen in Figure 14, about two-thirds of the sessions received overall ratings of 4 or 5 on the quality of implementation.

Content of Professional Development

Evaluators rated both disciplinary and pedagogical content of the professional development sessions. The following individual indicators were highly rated in most of the observed sessions:

• The appropriateness of the disciplinary and pedagogical content for the participants' backgrounds (80 percent);
• The extent to which appropriate connections were drawn to real-world contexts (70 percent); and
• The soundness of the disciplinary/pedagogical content and how appropriately it was explored or presented at the session (68 percent).

As can be seen in Figures 15 and 16, 65 percent of projects were rated 4 or 5 on disciplinary content and 67 percent were rated a 4 or 5 on pedagogical content. Note also that while ratings on disciplinary content were similar for the two cohorts, Cohort 1 projects were more likely than those in Cohort 2 to receive the highest possible ratings for pedagogical content.

Figure 15

Figure 16

Professional Development Culture

Most individual indicators for professional development culture were rated highly by evaluators. Indicators that received ratings of 4 or 5 in most of the sessions included:

• How encouraged participants were to generate ideas, conjectures, and propositions and to what extent investigation and risk-taking were valued (74 and 70 percent, respectively);
• Whether participants were intellectually engaged with important ideas relevant to the focus of the session (72 percent); and
• How highly valued intellectual rigor, constructive criticism, and the challenging of ideas were during the session (65 percent).

Figure 17

Figure 17 shows that 73 percent of the sessions received ratings of 4 or 5, with fairly similar ratings for the two cohorts.

Impact on Participants' Capacity for Exemplary Science/Mathematics Instruction

LSC evaluators were asked to rate how likely they felt the session was to have an impact on teachers' capacity for exemplary science/mathematics instruction. Ratings for this area were quite low, with 60 percent or fewer projects receiving ratings of 4 or 5 on any of the indicators:

• How likely the session was to enhance participants' ability to identify and understand important ideas of science/mathematics (60 percent);
• Whether the session enhanced professional networking among participants (60 percent);
• The likelihood that the session would enhance participants' ability to plan and implement exemplary classroom instruction (55 percent); and
• Whether the session was likely to help participants better understand how students learn (48 percent).

Figure 18

As seen in Figure 18, only 50 percent of the sessions received overall ratings of 4 or 5 in this category. Apparently, evaluators often saw individual elements of high quality, but felt that the experience as a whole was not likely to translate into a marked positive improvement in teaching.

Impact on Participants' Leadership Capacity

The 38 observed sessions in which leadership development was a focus were rated on their likelihood to have an impact on participants' capacity to be leaders in science/mathematics instructional reform. In these sessions, the indicators most frequently receiving ratings of 4 or 5 were:

• The likelihood of the session to enhance participants' understanding of the reform/ change process (70 percent); and
• Whether the session would help participants be more confident in their ability as leaders of science/mathematics reform efforts (68 percent).

Fewer of the observed sessions that focused on leadership-building received high ratings on other indicators such as:

• Whether it would enable participants to plan/implement exemplary professional development (54 percent);
• How well the session enhanced professional networking among participants with regard to leadership (50 percent);
• Whether the session would help teachers identify and understand important strategies for reform (44 percent); and
• Whether the session enhanced participants' understanding of adult learners (37 percent).

Figure 19

As seen in Figure 19, 64 percent of the sessions received overall ratings of 4 or 5 in this area; however, there were striking differences in the ratings of Cohort 1 and Cohort 2 projects. Only 5 of the 13 Cohort 1 leadership sessions (38 percent) were rated a 4 or 5 on likelihood of building leadership capacity compared to 18 of the 23 (79 percent) Cohort 2 sessions.

Capsule Ratings of Observed Professional Development Sessions

Evaluators were asked to provide an overall "capsule rating" of each observed session, taking into account the relative impact of each of the component areas. As would be expected, given the high overall continuum ratings assigned by evaluators for the professional development systems, ratings for individual professional development sessions were similarly quite favorable. Very few LSC sessions were rated as ineffective professional development (Level 1), and fewer than 10 percent were rated at Level 2, having quite limited likelihood of helping participants implement exemplary science/mathematics instruction or be leaders in reform. (See Figure 20.) More than 60 percent of the observed professional development sessions received ratings of 4 or 5, with Cohort 1 sessions somewhat more likely than Cohort 2 sessions to be considered "exemplary professional development."

Figure 20

The vignettes below illustrate a variety of types of sessions observed and some of the strengths and areas of need noted by evaluators.

Summary

In summary, the LSC projects are providing high-quality professional development to their targeted teachers. All projects were rated at level 3 or above on a five-level continuum, indicating that the projects are either transitioning toward, or have already achieved a level of professional development that incorporates many features reflective of current standards-based approaches and that these activities will likely enhance participants' capacity to provide high-quality science and mathematics instruction to their students.

Each of the component areas of professional development-the design, the implementation, the disciplinary and pedagogical content, and the culture of the sessions-received very high ratings, as well. Mean ratings for each of those areas, as well as the likelihood that the project would enhance teachers' instructional capacity were consistently between 3.5 and 4.0 on the five-point scale. Projects that focused on developing the leadership capacity of teachers in the district were rated somewhat lower in that area, with a mean rating of approximately 3.3.

The LSC projects are distinguished from teacher enhancement projects of the past by their systemic focus, including professional development for all teachers of science/mathematics at the targeted grade levels, the implementation of exemplary instructional materials, and the alignment of district policies and practices with the reform vision.

Evaluators found that the professional development sessions generally incorporated state-of-the-art strategies, including activities that were relevant to the teachers' work in classrooms, modeling of effective pedagogical strategies, opportunities for teachers to reflect and talk with peers, strong components of follow-up and support, and active involvement of teachers as key players in their own professional development. The sessions were generally led by competent and skilled facilitators and characterized by very positive cultures that encouraged the engagement of teachers as professionals in learning communities.

0. One project in Cohort 2 did not collect professional development observation data in 1996 because of a late start-up date; this project's baseline questionnaire data are included in the cross-site analyses.
1. HRI re-rated all 1995-96 professional development observations based on the updated continuum developed for the 1997 observation protocol. Evaluators were asked to verify these ratings and given the opportunity to make changes as needed. Ratings reported here have either been provided by or verified by the evaluators.
2. A number of projects reported observing professional development sessions in addition to those rated for the core evaluation; these additional observations were considered by evaluators when making overall ratings of the project's professional development system.
3. A project placed at Level 3.5 was included with the Level 3 group in this report.
4. The elements of individual professional development sessions discussed in the following sections parallel the components of the professional development system reported earlier (Figures I-6 and I-7). When rating the system components, evaluators had been asked to rate each component holistically, using all the data available to them. In contrast, ratings in the following sections are based solely on individual professional development sessions observed for the core evaluation.