Paper

As part of the core evaluation, evaluators of Cohort 1 projects were asked to interview a random sample of 10 teachers who had participated in the project's professional development activities. Among the teachers interviewed, over two-thirds of the group had received more than 30 hours of LSC professional development; the others had participated in 10 to 30 hours of professional development. As seen in Figure 12 the majority of teachers had quite positive impressions of the professional development in which they had participated.

Comments of participants who were most positive about the professional development indicate that high-quality curriculum materials were a particularly important component of the professional development. Teachers described curriculum materials as "complete and comprehensive," providing "easy access to materials" and access to "meaningful investigations." Teachers were also very positive about their experiences with inquiry-based science, the support provided through on-going activities, and strategies that involved lead teachers or core teams at their schools.

Figure 12

Even teachers who had overall very positive comments about the LSC professional development sometimes had suggestions for improvement. A number of comments focused on the need for more direct experience with the type of teaching they were striving to implement through the LSC project. A couple of participants commented, for example, that they would like to observe teachers using inquiry-based or constructivist strategies; another participant expressed the desire to have a visiting teacher come to his classroom to demonstrate these instructional strategies.

Only a small proportion of teachers who were interviewed were described by evaluators as either "somewhat" or "mostly" negative, and all of these teachers came from 3 of the 8 Cohort 1 projects. Those teachers who were negative about the professional development they had experienced complained of being unclear about expectations of the district and needing professional development that better met their particular needs with regard to content or pedagogical strategies.

Ratings of Individual Professional Development Sessions

Observations of selected professional development sessions at each LSC site are key to the core evaluation analysis of the quality of the professional development offered by LSC projects; although evaluators were asked to take other factors into account when making continuum and component ratings of their project, observations were the main standardized data source.

The following sections describe elements of professional development that observers were asked to rate for each professional development session observed, including:

• Design of the session;
• Its implementation;
• The disciplinary and pedagogical content;
• The character of the culture of the session;
• The likely impact on participants' capacity for exemplary science and mathematics instruction; and
• The likely impact on participants' leadership capacity.

Each section includes details about the pertinent indicators that contributed to evaluators' ratings of the individual professional development sessions observed for the core evaluation, as well as the percent of sessions receiving overall ratings of 4 or 5 on a 5-point scale.

Design of Professional Development

Evaluators were asked to assess the design of professional development sessions by rating a series of individual indicators based on current understandings of best practice. Several of these indicators received high ratings (4 or 5 on a five-point scale) in the majority of observed sessions. Those indicators that were most often highly rated included the following:

• The appropriateness of strategies used in the session for accomplishing the purposes of the LSC professional development (78 percent);
• How well the session encouraged a collaborative approach to learning and incorporated tasks and interactions consistent with a spirit of investigation (75 percent); and
• How effectively the session built on participants' knowledge of content, teaching, learning, and/or the reform/change process (71 percent).

Somewhat fewer sessions were rated highly on:

• How well the strategies reflected attention to the varied needs and learning styles of the participants (65 percent); and
• Whether the design of the session provided adequate time and structure for reflection (53 percent).

Figure 13

Evaluators were also asked to provide an overall rating for the design of the session. As can be seen in Figure 13, 74 percent of the professional development sessions received overall design ratings of 4 or 5.

Implementation of Professional Development

Evaluators rated several indicators regarding the implementation of the professional development session. Ratings on the implementation tended to be lower than those on design. The indicators most frequently rated 4 and 5 were:

• Whether the facilitator's background and expertise enhanced the quality of the session (82 percent); and
• The appropriateness of the pace of the session for adult learners (69 percent).

Fewer LSC professional development sessions were rated highly on other indicators, including:

• How well the session modeled questioning strategies that are likely to enhance the development of conceptual understanding (60 percent); and
• How well the session modeled effective assessment strategies (42 percent).

Figure 14

As can be seen in Figure 14, about two-thirds of the sessions received overall ratings of 4 or 5 on the quality of implementation.

Content of Professional Development

Evaluators rated both disciplinary and pedagogical content of the professional development sessions. The following individual indicators were highly rated in most of the observed sessions:

• The appropriateness of the disciplinary and pedagogical content for the participants' backgrounds (80 percent);
• The extent to which appropriate connections were drawn to real-world contexts (70 percent); and
• The soundness of the disciplinary/pedagogical content and how appropriately it was explored or presented at the session (68 percent).

As can be seen in Figures 15 and 16, 65 percent of projects were rated 4 or 5 on disciplinary content and 67 percent were rated a 4 or 5 on pedagogical content. Note also that while ratings on disciplinary content were similar for the two cohorts, Cohort 1 projects were more likely than those in Cohort 2 to receive the highest possible ratings for pedagogical content.

Figure 15

Figure 16

Professional Development Culture

Most individual indicators for professional development culture were rated highly by evaluators. Indicators that received ratings of 4 or 5 in most of the sessions included:

• How encouraged participants were to generate ideas, conjectures, and propositions and to what extent investigation and risk-taking were valued (74 and 70 percent, respectively);
• Whether participants were intellectually engaged with important ideas relevant to the focus of the session (72 percent); and
• How highly valued intellectual rigor, constructive criticism, and the challenging of ideas were during the session (65 percent).

Figure 17

Figure 17 shows that 73 percent of the sessions received ratings of 4 or 5, with fairly similar ratings for the two cohorts.

Impact on Participants' Capacity for Exemplary Science/Mathematics Instruction

LSC evaluators were asked to rate how likely they felt the session was to have an impact on teachers' capacity for exemplary science/mathematics instruction. Ratings for this area were quite low, with 60 percent or fewer projects receiving ratings of 4 or 5 on any of the indicators:

• How likely the session was to enhance participants' ability to identify and understand important ideas of science/mathematics (60 percent);
• Whether the session enhanced professional networking among participants (60 percent);
• The likelihood that the session would enhance participants' ability to plan and implement exemplary classroom instruction (55 percent); and
• Whether the session was likely to help participants better understand how students learn (48 percent).

Figure 18

As seen in Figure 18, only 50 percent of the sessions received overall ratings of 4 or 5 in this category. Apparently, evaluators often saw individual elements of high quality, but felt that the experience as a whole was not likely to translate into a marked positive improvement in teaching.

Impact on Participants' Leadership Capacity

The 38 observed sessions in which leadership development was a focus were rated on their likelihood to have an impact on participants' capacity to be leaders in science/mathematics instructional reform. In these sessions, the indicators most frequently receiving ratings of 4 or 5 were:

• The likelihood of the session to enhance participants' understanding of the reform/ change process (70 percent); and
• Whether the session would help participants be more confident in their ability as leaders of science/mathematics reform efforts (68 percent).

Fewer of the observed sessions that focused on leadership-building received high ratings on other indicators such as:

• Whether it would enable participants to plan/implement exemplary professional development (54 percent);
• How well the session enhanced professional networking among participants with regard to leadership (50 percent);
• Whether the session would help teachers identify and understand important strategies for reform (44 percent); and
• Whether the session enhanced participants' understanding of adult learners (37 percent).

Figure 19

As seen in Figure 19, 64 percent of the sessions received overall ratings of 4 or 5 in this area; however, there were striking differences in the ratings of Cohort 1 and Cohort 2 projects. Only 5 of the 13 Cohort 1 leadership sessions (38 percent) were rated a 4 or 5 on likelihood of building leadership capacity compared to 18 of the 23 (79 percent) Cohort 2 sessions.

Capsule Ratings of Observed Professional Development Sessions

Evaluators were asked to provide an overall "capsule rating" of each observed session, taking into account the relative impact of each of the component areas. As would be expected, given the high overall continuum ratings assigned by evaluators for the professional development systems, ratings for individual professional development sessions were similarly quite favorable. Very few LSC sessions were rated as ineffective professional development (Level 1), and fewer than 10 percent were rated at Level 2, having quite limited likelihood of helping participants implement exemplary science/mathematics instruction or be leaders in reform. (See Figure 20.) More than 60 percent of the observed professional development sessions received ratings of 4 or 5, with Cohort 1 sessions somewhat more likely than Cohort 2 sessions to be considered "exemplary professional development."

Figure 20

The vignettes below illustrate a variety of types of sessions observed and some of the strengths and areas of need noted by evaluators.

Summary

In summary, the LSC projects are providing high-quality professional development to their targeted teachers. All projects were rated at level 3 or above on a five-level continuum, indicating that the projects are either transitioning toward, or have already achieved a level of professional development that incorporates many features reflective of current standards-based approaches and that these activities will likely enhance participants' capacity to provide high-quality science and mathematics instruction to their students.

Each of the component areas of professional development-the design, the implementation, the disciplinary and pedagogical content, and the culture of the sessions-received very high ratings, as well. Mean ratings for each of those areas, as well as the likelihood that the project would enhance teachers' instructional capacity were consistently between 3.5 and 4.0 on the five-point scale. Projects that focused on developing the leadership capacity of teachers in the district were rated somewhat lower in that area, with a mean rating of approximately 3.3.

The LSC projects are distinguished from teacher enhancement projects of the past by their systemic focus, including professional development for all teachers of science/mathematics at the targeted grade levels, the implementation of exemplary instructional materials, and the alignment of district policies and practices with the reform vision.

Evaluators found that the professional development sessions generally incorporated state-of-the-art strategies, including activities that were relevant to the teachers' work in classrooms, modeling of effective pedagogical strategies, opportunities for teachers to reflect and talk with peers, strong components of follow-up and support, and active involvement of teachers as key players in their own professional development. The sessions were generally led by competent and skilled facilitators and characterized by very positive cultures that encouraged the engagement of teachers as professionals in learning communities.