Paper

LSC Year Three Cross-Site Report

Project evaluators were asked to observe 5- 8 professional development activities for the core evaluation. Evaluators and PIs were to decide jointly which activities would be observed, selecting sessions to represent the diversity of the project's professional development program and to reflect the extensiveness and importance of the various activities offered. A total of 276 professional development sessions were observed, an average of six observations per project. This chapter presents a summary of data collected from observations of individual sessions across all LSC projects. The chapter includes descriptive information about the observations, followed by a presentation of evaluators' assessments of the quality of the professional development sessions.

Description of LSC Professional Development Sessions

Evaluators documented a number of descriptive features of each professional development session, providing information across all projects about targeted participants, presenters/ facilitators, purposes and content focus, and the major types of activities that characterized the sessions.

Participants

The typical professional development session observed for the LSC core evaluation had between 21 and 50 participants; only 4 percent of the sessions had more than 100 participants. Some sessions exclusively targeted lead teachers (22 percent); most sessions targeted non- lead teachers (47 percent); and some sessions targeted both lead and non- lead teachers (27 percent). A total of 12 percent of the sessions included principals or other administrators.

Presenters/ Facilitators

As noted earlier, LSC professional development involves presenters/ facilitators from a variety of settings. Lead teachers served in this capacity in 40 percent of the observed sessions, while only 27 percent of the sessions included university faculty as presenters or facilitators. (See Figure 6.) Summing up over all of the observed sessions, two- thirds of the presenters/ facilitators were female and one- third, male. As can be seen in Figure 7, 87 percent of the presenter/ facilitators were white, and 13 percent members of other race/ ethnic groups.

Figure 6

Figure 7

Purposes of the Professional Development Sessions

Evaluators were asked to indicate the major intended purposes of each observed session based on information provided by the session facilitators. The most frequently cited purposes were working on classroom pedagogy issues (75 percent), including learning about specific instructional materials, and enhancing teachers' understanding of mathematics/ science concepts (40 percent). (See Table 5.) Overall, fewer than 10 percent of the observed professional development sessions had a major focus on student assessment, equity issues, or developing the capacity of participants to use technology.

Table 5
Major Intended Purposes of LSC Professional Development Sessions

	Percent of Sessions
	All Sessions	K- 8 Science	K- 8 Mathe- matics	7- 12 Mathe- matics
Working on classroom pedagogy issues* Learning about specific instructional materials Learning pedagogical/ classroom management strategies Creating a vision of learning through investigation Designing/ scoring student assessments Considering issues of access, equity, and diversity	75 38 36 35 9 6	77 37 37 39 11 7	69 31 39 27 6 4	75 51 30 30 4 4
Increasing teacher mathematics/ science content knowledge	40	37	39	49
Promoting reflective practice	17	18	10	19
Developing teacher leaders	15	14	15	15
Building professional networks among educators	14	14	12	15
Orientation to the project Developing capacity of participants to use technology	14 5	13 5	23 2	11 9

* Percents add to more than total for category because sessions could include more than one purpose.

Content Focus of Professional Development Sessions

When sessions focused on one or more disciplinary content areas, evaluators were asked to categorize that content. In K- 8 science projects, evaluators reported that nearly half of the sessions that had a disciplinary content focus dealt with physical science concepts (46 percent), nearly that many with life science content (41 percent), and somewhat fewer with concepts from the earth and environmental sciences (32 percent). Evaluators also reported that a number of the observed K- 8 science sessions dealt with data collection and analysis, measurement, engineering and design principles, estimation, and patterns and relationships. (See Figure 8.) None of the observed K- 8 science sessions emphasized the history of science.

Figure 8

Percentages add to more than 100% because sessions could include more than one content area.
* Only sessions that focused on disciplinary content were included in these analyses.

In projects targeting K- 8 mathematics, the most heavily emphasized topics were patterns and relationships (32 percent of the sessions that dealt with disciplinary content), numeration and number theory (29 percent), and data collection and analysis (23 percent). Computation, geometry and spatial sense, probability, measurement, and statistics were each the focus of 10- 16 percent of K- 8 mathematics content-focused professional development sessions; sessions for K- 8 mathematics teachers rarely focused on pre- algebra or algebra. (See Figure 9.)

Figure 9

Percentages add to more than 100% because sessions could include more than one content area.
* Only sessions that focused on disciplinary content were included in these analyses.

As can be seen in Figure 10, the most commonly emphasized topics in sessions for 7- 12 mathematics teachers were patterns and relationships, algebra, and geometry/ spatial sense, each the focus of about 30 percent of the sessions that dealt with disciplinary content. Roughly 20 percent of the content- focused sessions addressed data collection/ analysis and probability, while about 10 percent focused on estimation, measurement, and statistics.

Figure 10

Percentages add to more than 100% because sessions could include more than one content area.
* Only sessions that focused on disciplinary content were included in these analyses.

Session Activities

Most of the observed LSC professional development sessions included several different instructional strategies. As can be seen in Table 6, most sessions included discussions or seminars (83 percent), typically either in whole groups led by professional development providers or in small groups or pairs of participants. Roughly two- thirds of sessions included formal presentations, usually by project staff as opposed to participants; and a similar number included investigative/ problem solving activities. Fewer of the observed sessions involved participants in reading (21 percent) or writing (25 percent) about disciplinary content, pedagogy or reform issues.

Table 6
Major Activities of LSC Professional Development Sessions

	Percent of Sessions
	All Sessions	K- 8 Science	K- 8 Mathematics	7- 12 Mathematics
Engaged in discussions/ seminars Whole group led by facilitator Whole group led by participants Small groups/ pairs	83 64 9 49	83 66 9 45	77 57 8 55	92 65 13 57
Engaged in problem- solving/ investigation	68	70	58	73
Listened to a formal presentation By presenter/facilitator By participants	68 65 11	69 67 10	77 71 14	54 50 10
Wrote about disciplinary content, pedagogy, or reform issues	25	23	27	29
Read about disciplinary content, pedagogy, or reform issues	21	20	31	15

Observer Ratings of Session Components

In order to assess the quality of professional development sessions, evaluators were asked to rate a number of components for each session they observed, including:

• The design of the session;

• The implementation of the professional development activities;
• The disciplinary, pedagogical, and/ or leadership content; and
• The culture of the session.

For each component area, observers first rated a series of individual indicators of best practice in professional development and/ or standards- based mathematics/ science instruction. These indicators were rated on a scale ranging from 1, "not at all" to 5, "to a great extent" to document the extent to which that feature characterized the observed professional development session.

Considering those "on- ramp" indicators, observers then assessed the overall quality of each component area. The lowest rating for component areas (Level l) indicated that the session was not at all reflective of best practice. The highest rating (Level 5) indicated that component of the session was extremely reflective of best practices for standards-based mathematics and science education.¹ Evaluators' ratings of the component areas are presented in the following sections.

Design of Professional Development Sessions

As noted above, observers assessed the design of professional development sessions by rating a series of individual indicators based on current understandings of best practice. Several of these indicators received high ratings (4 or 5 on a five- point scale) in a large majority of the observed sessions. Those indicators that were most often highly rated included:

• The extent to which the session design reflected careful planning and organization (84 percent);

• How well the session encouraged a collaborative approach to learning and incorporated tasks and interactions consistent with a spirit of investigation (82 percent); and

• The appropriateness of strategies used in the session for accomplishing the purposes of the LSC professional development (80 percent).

Somewhat fewer sessions were rated highly on:

• The extent to which participants were given an opportunity to share experiences and insights (64 percent);

• How well the strategies reflected attention to the varied needs and learning styles of the participants (60 percent); and

• Providing adequate time and structure for reflection (59 percent).

Observers found that the majority of professional development sessions were highly reflective of best practice. As can be seen in Figure 11, 76 percent of the professional development sessions received overall design ratings of 4 or 5.

Figure 11

Implementation of Professional Development Sessions

Observers also assessed the implementation of professional development sessions. Indicators most frequently rated 4 and 5 were:

• Whether the facilitator's background and expertise (84 percent) or management style (76 percent) enhanced the quality of the session and

• The extent to which the session incorporated instructional strategies appropriate for its purposes and the needs of adult learners (76 percent).

Fewer LSC professional development sessions were rated highly on other indicators, including:

• How well the session modeled questioning strategies that are likely to enhance the development of conceptual understanding (58 percent) and

• How well the session modeled effective assessment strategies (54 percent).

As can be seen in Figure 12, 71 percent of the sessions received overall ratings of 4 or 5 on the quality of implementation.

Figure 12

Disciplinary and Pedagogical Content

Evaluators were asked to rate either the quality of the disciplinary content of the observed session, its pedagogical content, or both, depending on the focus of the session. Disciplinary content was rated in 176 of the 276 sessions, with 73 percent of the sessions receiving overall ratings of 4 or 5 in this area. (See Figure 13.) Disciplinary content sessions were most likely to receive high ratings for:

• The appropriateness of the disciplinary content for the purposes of the session and the background of the participants (86 percent);

• The extent to which the facilitators displayed an understanding of mathematics/ science content (86 percent); and

• The soundness and appropriateness of mathematics/ science content (80 percent).

Far fewer sessions received high ratings on:

• The extent to which appropriate connections were made to other areas of mathematics/ science, to other disciplines and/ or to real- world contexts (56 percent); and

• Whether the degree of closure of conceptual understanding was appropriate for the purposes of the session (52 percent).

Figure 13

Observers rated 198 of the 276 observed professional development sessions on the quality of their pedagogical content. As can be seen in Figure 14, 71 percent of professional development sessions received ratings of 4 or 5 for overall pedagogical content.

Figure 14

As was the case for disciplinary content, sessions tended to be highly rated for:

• The appropriateness of the pedagogical content for the purposes of the session and the background of the participants (80 percent) and

• The extent to which the facilitator displayed an understanding of pedagogical concepts (78 percent).

However, where 80 percent of the sessions that focused on disciplinary content received high ratings for content that was "sound and appropriately explored," only 66 percent of the sessions rated for pedagogical content received high ratings.

The two areas were more similar in terms of ratings of closure, with 57 percent of sessions focusing on pedagogical content (compared to 52 percent of those focusing on disciplinary content) receiving high ratings in this area.

Professional Development Culture

The literature on effective staff development emphasizes the importance of establishing a professional development culture where teachers can explore content and pedagogy in a collegial, risk- free environment. Seventy- eight percent of the sessions received synthesis ratings of 4 or 5 in this area. (See Figure 15.) Indicators that were most likely to receive high ratings included:

• Whether active participation of all was encouraged and valued (88 percent);

• The extent to which there was a climate of respect for participants' experiences, ideas, and contributions (87 percent);

• The extent to which interactions reflected collaborative working relationships among participants (83 percent), and between facilitators and participants (80 percent); and

• Whether participants were intellectually engaged with important ideas relevant to the focus of the session (80 percent).

Two areas that were somewhat less likely to receive high ratings were:

• The extent to which investigation and risk- taking were encouraged (70 percent) and

• Whether intellectual rigor, constructive criticism, and the challenging of ideas were valued (64 percent).

Figure 15

Overall Assessment of Observed Professional Development Sessions

Observers were asked to consider all information available to them-- their own ratings of session components, related interviews, and their knowledge of the projects' professional development program-- as they assessed the overall quality of each observed session. Observers first considered the likely impact of the session on participants' capacity for exemplary mathematics/ science instruction and the likely impact on leadership capacity when leadership development was a focus of the session instruction.² Finally, "capsule ratings" were assigned to characterize the overall quality of the professional development session. Ratings on a five- level scale ranged from "ineffective professional development" (Level 1) to "exemplary professional development" (Level 5).

Results of observers' ratings of impact and overall quality, as well as features that distinguished highly effective sessions from less effective sessions, are described in the sections below.

Sessions' Impact on Participants' Capacity for Exemplary Mathematics/ science Instruction

Observers rated the likely impact of each session on teachers' capacity for exemplary mathematics/ science instruction. According to these observers, LSC professional development sessions were most likely to have a positive effect on participants' ability to:

• Implement exemplary classroom instructional materials (77 percent);

• Identify and understand important ideas of mathematics/ science (76 percent); and

• Plan and implement exemplary classroom instruction (75 percent).

Fewer sessions were judged likely to have a positive effect on participants':

• Understanding of how students learn (60 percent) and

• Proactiveness in addressing their professional development needs (58 percent).

Capsule Ratings of Observed Professional Development Sessions

As would be expected, given the high ratings assigned by evaluators for the various components, overall ratings for individual professional development sessions were quite favorable. Only 1 percent of observed LSC sessions were rated as ineffective professional development (Level 1), and 7 percent were rated at Level 2, having quite limited likelihood of helping participants implement exemplary mathematics/ science instruction or be leaders in reform. Overall, 71 percent of the observed professional development sessions received ratings of 4 or 5, indicating that those sessions were skillfully facilitated, engaging participants in purposeful work that would likely lead to enhanced capacity to implement exemplary instruction. (See Figure 16.)

Figure 16

Comparison of Highly Effective and Less Effective Professional Development Sessions

To help identify the features that seem most critical in determining the quality of professional development sessions, HRI compared the most highly rated professional development sessions to those with lower ratings. In each of the component areas-- the design of the session, its implementation, the disciplinary and pedagogical content, and the culture of the session-- there were specific indicators that distinguished sessions judged to be highly effective from those that were considered less effective.

Table 7 shows the indicators in each of the component areas with the largest differences between highly effective (capsule ratings 4 and 5) and less effective (capsule ratings 1- 3) professional development sessions. For example, while the majority of mathematics and science professional development sessions (80 percent) included in their design, strategies that were appropriate for the purposes of the session, this feature characterized 97 percent of the sessions judged to be highly effective, but only 38 percent of the less- effective sessions. Similarly, over half of the observed professional development sessions (58 percent) included effective questioning strategies; however, this feature was highly rated in 79 percent of the effective sessions, as compared to only 8 percent of the less- effective sessions, a difference of 71 percent.

Other indicators showed smaller differences when comparing highly effective sessions to those rated less effective. These were typically areas of relative strength for the sessions with low capsule ratings. For example, 60 percent or more of the professional development sessions rated ineffective overall, nevertheless received high ratings for the appropriateness of the disciplinary content, having a climate of respect, valuing active participation of all, and having collaborative working relationships among participants. The differences in percentages of sessions receiving high ratings on those indicators were in the 26- 37 percent range, compared to the 55- 71 percent range for the indicators in Table 7.

Summary

• A major data source in the core evaluation for assessment of the professional development was observations of representative professional development sessions in each LSC project. Across all 46 LSC projects, a total of 276 professional development sessions was observed. A cross- site analysis of all professional development sessions provided an overview of the key purposes and activities that characterized most sessions, as well as insight into major strengths and areas of need in the design and implementation of professional development activities.

• The data on LSC professional development sessions showed a number of interesting patterns. In addition to the fact that the observed sessions were much more likely to focus on pedagogical content than disciplinary content, they were much more likely to be led by teacher leaders and other district personnel than by university faculty. These data may indicate an imbalance between content and pedagogy in LSC professional development, echoing a concern voiced by a number of project evaluators.

• The data also showed that only 13 percent of presenters/ facilitators were members of minority groups, which is reflective of neither the targeted teacher population (25 percent minority) nor the targeted student population (51 percent minority). In all likelihood, these figures reflect the underrepresentation of minorities in the professional development provider pool as a whole, rather than any bias or insensitivity on the part of project staff. Accordingly, the data point to the need for the LSC projects, and NSF, to pay particular attention to increasing the pool of minorities prepared to serve in leadership roles in mathematics and science professional development.

• Observers found that the majority of LSC sessions (71 percent) provided high- quality professional development experiences that were likely to enhance the capacity of teachers to implement exemplary instruction in their classrooms. Of special note was the collegial and engaging culture established among participants and facilitators. Observers found that most sessions were also well- facilitated, and targeted mathematics/ science content that was both sound and appropriate for the purposes of the session and background of participants.

• Areas that observers identified as typically problematic were attention to closure at appropriate points in the professional development sequence and providing adequate time and structure for reflection.

1 Copies of the Professional Development Observation Protocol may be found on the LSC Documents Page of the HRI web site: http://www.horizon-research.com/LSC.

2 Assessment of the leadership component of the LSC professional development is discussed in Chapter 5.