Core Evaluation Question I
What is the overall quality of the LSC professional development activities?
Core evaluation data to assess the quality of LSC professional development activities come from observations of the professional development activities, and to a lesser extent, from teacher questionnaires. Cohort 1 projects augmented these data with teacher interviews. Based on these data, evaluators were asked to rate the quality of the project's professional development in eight component areas and to provide an overall rating of the project's professional development system.
Cohort 1 projects reported on their second year of implementing LSC professional development activities. These projects all target elementary science teachers, and are analyzed in a single group, referred to as C1S. For the newly-funded Cohort 2 projects, these 1996 data represent the status of professional development during the first year of LSC funding1. Cohort 2 includes both projects that target elementary science (C2S) and projects that target elementary mathematics (C2M).
Teacher Ratings of District-Provided Professional Development
The LSC teacher questionnaire included several questions about the quality of the science/mathematics professional development available in their district. There was no attempt to focus on LSC professional development programs, but rather teachers were asked to think about district-provided professional development generally.
The composite for quality of district professional development combines teacher ratings of the extent to which they were:
- Involved in planning their professional development;
- Encouraged to develop an individualized professional development plan;
- Given time to work with other teachers and to reflect on what they have learned; and
- Received follow-up support.
Ratings on this composite are relatively low and show a lot of variation among projects; project means ranged from 43 to 65 percent of total possible points. (See Figure 6.)
Roughly one-fourth to one-third of teachers in each cohort reported that they were fairly heavily involved in planning their own science- or mathematics-related professional development and encouraged to develop individual professional development plans to address their needs and interests. Only about 20 percent of the teachers indicated that they are given ample opportunities to work with other teachers or reflect on how to apply what they have learned to the classroom as part of their professional development. Roughly 1 in 3 Cohort 1 teachers, but only 1 in 5 Cohort 2 teachers reported that they were given the follow-up support they needed to implement what they had learned in professional development experiences. (See Table 2.)
Teacher Descriptions of District Professional Development
* Includes teachers marking 4 or 5 on a five-point scale where 1 = "not at all" and 5 = "to a great extent."
||Percent of Teachers*
||Cohort 1 Science
||Cohort 2 Science
||Cohort 2 Mathematics
|Involved in planning my science (mathematics) related professional development
|Encouraged to develop an individual professional development plan
|Received follow-up support in implementation
|Given time to work with other teachers as part of my professional development
|Given time to reflect on what I've learned and how to apply to the classroom
Given current standards for best practice in professional development, there is a great deal of room for improvement in the quality of district-provided professional development in science and mathematics. In the following sections, data from evaluators' observations of LSC professional development activities are used to describe how LSC projects are addressing this challenge.
Description of LSC Professional Development Activities
Evaluators and PIs were asked to decide jointly which project activities would be observed for the core evaluation, selecting sessions to represent the diversity of the project's activities and reflect the extensiveness and importance of the various activities offered. LSC evaluators observed a total of 180 professional development sessions, an average of seven observations per project. The following description of LSC professional development activities is based on these sessions.2
The typical professional development session observed for the LSC core evaluation had between 21 and 50 participants, mostly teachers from a range of the elementary grades. As can be seen in Figure 7, Cohort 1 projects continued to focus on preparing lead teachers (almost three-quarters of Cohort 1 science sessions included lead teachers); far fewer Cohort 2 sessions included lead teacher participants (36 percent). Overall, 7 out of 10 observed professional development sessions included regular (non-lead) teachers and fewer than 2 in 10 included administrators as participants.
Participants in Observed LSC Professional Development Activities
Typically, LSC professional development sessions were conducted by a team, sometimes including lead teachers, but more frequently composed of other personnel both from the district and from outside the district. Almost 90 percent of the observed sessions had at least one female presenter or facilitator, and 17 percent included non-white women. A little over half of the sessions had male presenters, in a few cases including African American or Hispanic males.
Session Characteristics and Foci
Evaluators were asked to indicate the purposes of each professional development session they observed, choosing from twelve possibilities; for most sessions, evaluators checked from 2 to 4 purposes. Nearly half of the observed sessions focused on helping participants create a vision of learning by inquiry, try out activities or curricula for classroom use, or learn science/mathematics concepts. (See Table 3.) Almost 40 percent of sessions centered on working with teachers on pedagogical/classroom management strategies; about a fifth of observed sessions concentrated on building a professional network among educators or developing participants' leadership capacity.
Major Intended Purposes of Observed
LSC Professional Development Sessions
||Percent of Sessions
||Cohort 1 Science
||Cohort 2 Science
||Cohort 2 Mathematics
|Creating a vision of learning by inquiry
|Trying out specific (e.g., newly-adopted) activities/curricula for classroom use
|Learning science/mathematics concepts
|Learning pedagogical/classroom management strategies
|Building a professional network among educators
|Developing leadership capacity of participants
|Orientation to the project
|Developing instructional materials/activities for classroom use
|Aligning district curricula with state/national standards ||12
|Planning future professional development
|Designing/scoring student assessments
|Developing capacity of participants to use the Internet/on-line telecommunications
Nearly half of the observed science sessions included physical science content, 37 percent focused on life science, and 33 percent had an earth-science content focus. (See Figure 8.)
Among projects targeting mathematics, the content areas of focus were most often computation and estimation, numeration and number theory, patterns and relationships, and mathematical problem solving. While a few of the science sessions included emphasis on one or more mathematics topics (especially measurement), none of the mathematics sessions included science content as a major focus. (See Figure 9.)
Major Disciplinary Content Areas of Observed Science Professional Development Sessions*
Major Disciplinary Content Areas of Observed Mathematics Professional Development Sessions*
As can be seen in Figure 10, when asked to describe the "pedagogical content" of the observed session, evaluators in each cohort noted that over half of the observed sessions emphasized instructional strategies. Somewhat fewer sessions focused on providing visions of effective science/mathematics instruction and even fewer on how students learn. Professional development sessions in Cohort 1 were much more likely than those in Cohort 2 to include consideration of assessment strategies and issues related to access, equity, and diversity.
Pedagogical Content Areas of Observed LSC Professional Development Activities
Most LSC professional development sessions included a formal presentation by a facilitator or other presenter. Teachers were involved in field activities, research, or other hands-on activities during 70 percent of the observed sessions; discussions or seminars occurred at a similar number of sessions. Very few sessions (12 percent) involved participants actively exploring uses of technology. (See Table 4.)
Major Activities at Observed LSC Professional Development Sessions
||Percent of Sessions
||Cohort 1 Science
||Cohort 2 Science
||Cohort 2 Mathematics
|Formal presentations by presenter/facilitator||76||78||75||77
|Hands-on/ manipulative/ research/ field activities||70||69||72||65
|Explored assessment strategies||34||44||31||23
|Formal presentations by participants||16||22||14||12
|Explored technology use||12||20||5||19
Evaluator Ratings of the Quality of the LSC Professional Development
Evaluators' overall ratings of the LSC professional development indicate that LSC projects are generally providing high-quality staff development. Each project evaluator was asked to rate the project's professional development system holistically, considering all of the data available, and place their LSC project on a five-level continuum. (See Table 5.)3
Ratings for the two cohorts were quite similar, with mean ratings of 3.8 on the 5 point scale. Eleven out of 25 projects were rated at Level 3 ("Transitioning to Quality Professional Development") by their project evaluators.4 All of the remaining projects were rated even higher on the continuum, indicating that their professional development plan and activities incorporated many features reflective of current standards-based approaches, that the professional development activities were well-implemented, and that in the evaluator's judgment they will likely enhance participants' capacity to provide high quality science/mathematics instruction to their students.
Continuum Ratings for Quality of LSC Professional Development
||Percent of Projects
|Level 1: Predominance of Ineffective Professional Development||0||0||0
|Level 2: Exploring Quality Professional Development||0||0||0
|Level 3: Transitioning to Quality Professional Development||44||50||41
|Level 4: Emerging Infrastructure of Well-Designed Professional Development||36||25||41
|Level 5: Predominance of Well-Designed Professional Development||20||25||18
|Mean Continuum Rating Level||3.8||3.8||3.8
In addition to overall continuum ratings, evaluators provided ratings for each of seven component areas for professional development, including: the design of activities, their implementation, the disciplinary and pedagogical content, the professional development culture, the likelihood that the professional development would build participants' capacity to provide exemplary science/mathematics instruction, and the likelihood it would build their capacity to be leaders in science/mathematics instructional reform. The largest difference between the cohorts was in ratings for capacity-building for instruction, with Cohort 1 projects more likely to be highly rated in this area. (See Figure 11.) No evaluators rated a component area for their project at Level 1, and very few components were rated at Level 2.
Components of Professional Development System
Evaluator Ratings by Cohort
Strengths of the LSC Projects
Evaluators were asked to describe their rationale for their ratings, particularly highlighting strengths of the professional development activities and areas in need of improvement. Aspects of the LSC professional development that were most often described by evaluators as strengths were:
- strategies for professional development;
- the skills, competence, and expertise of facilitators;
- the systemic focus on the broader science/mathematics education system;
- a culture conducive to and supportive of learning;
- relevant, high-quality content;
- models for effective development of teacher leaders; and
- reflection of project staff for program improvement.
Two features that distinguish this group of professional development projects from many past teacher enhancement projects are the systemic focus and the consistency of attention directed toward the use of exemplary curriculum materials. Evaluators commented frequently on these aspects of the LSC projects.
Evaluators of the LSC projects described a number of ways in which projects targeted the broader science/mathematics education system. For example, a requisite commitment for LSC funding is to serve all eligible teachers for a substantial number of hours. Many programs are choosing to begin by "exposing" large proportions of teachers to one or two kits and achieve at least a "mechanical" knowledge of the materials early on. One evaluator pointed out the benefit of this strategy as a step toward building support among stakeholders:
"Through the [teacher workshops], the Partnership is reaching larger numbers of teachers and providing exposure to inquiry-centered instructional approaches that can readily be used in elementary classrooms. These workshops, by increasing the number of teachers involved in the Partnership, are also expanding the cadre of stakeholders involved in the reform of science and mathematics."
Most projects are implementing curricula that the district has chosen or agreed to adopt, reinforcing the alignment of district curriculum and goals. One evaluator made this observation:
"[The district] has recently adopted FOSS (Full Options Science System) as its district science curriculum. Many of the pedagogy sessions and some of the content sessions encourage exploration of specific FOSS kits. By utilizing the district curriculum as a starting point for good hands-on science, [the project] is seen by participants as supporting the district-identified process to improve science teaching."
The LSC projects are also distinguished, as a group, by being highly reflective of current standards for best practice in professional development. Features described most often by evaluators of the LSC projects are described below:
- Relevancy of professional development activities to teachers' work in classrooms
Evaluators found, for example, that activities in which teachers were engaged were closely linked to their needs as classroom teachers. Said one,
"This professional development model establishes a different dynamic between practitioners and their professional growth. There is a relationship between what occurs inside and outside of their classroom. The school adopts math and science curricula considered exemplary and in the best interest of students, and complements the adoption with extended staff development to assist teachers in the implementation process."
A strategy mentioned as particularly successful by another evaluator was allowing teachers to focus on grade-specific activities. This evaluator reported these teacher comments:
"What I got a lot from in [the workshop] was when I sat with other fourth grade teachers and talked about how we implemented the kit"
"The people who did kindergarten [sessions] were actual kindergarten classroom teachers so when they gave information it had been used and tested and worked. Those were very useful, very helpful."
- Modeling effective strategies
Evaluators found that strategies advocated by the projects were modeled in professional development activities. Activities were characterized, for example, by active involvement of teachers and participatory hands-on activities; in many cases teachers worked on activities from modules that they would later use in their classes.
Experiencing instructional strategies firsthand was particularly important for teachers being introduced to inquiry-based strategies, as described by this evaluator:
"In order to teach in an inquiry-based environment, teachers must experience inquiry learning for themselves. The introductory summer institute provides the foundation upon which teachers must stand if they are to transform their teaching approaches to be truly inquiry-based. Two weeks enables participants to thoroughly explore a particular scientific phenomenon, and to come to conclusions of their own about why "x" works, drawing on their own experiential learning. In addition, teachers learn how a real classroom might function as an inquiry-based learning environment. This is often difficult to even imagine, given the relatively fixed structured nature of most elementary schools. In discussions and debates, the institute directors encourage participants to consider how to translate their experience back to their classrooms."
- Provides opportunities for teachers to reflect and talk with peers
Time for reflection was incorporated into many of the projectsi professional development activities sometimes within individual sessions; sometimes as the major focus of entire sessions as in school-based peer group meetings to discuss the implementation of a kit. This comment is reflective of evaluators' (and participants') praise for projects' providing opportunities for reflection:
"Overall, lead teachers, in interviews and in the post-institute evaluation surveys highly praised the academic year sessions-particularly noting the opportunities to share classroom practices, to practice inquiry, and to discuss national standards with peers. Through the [institute] lead teachers have opportunities for reflection, dialogue, and peer support."
- Strong components of support and follow-up
Support throughout the academic year is a consistent feature of the LSC projects, and one that evaluators recognized as a strength. One evaluator, for example, highlighted the importance of providing opportunities for teachers to meet throughout the year for discussions with peers after actually trying out modules in classrooms.
Projects have explored different strategies for providing on-going support to teachers. A number of projects that focus on kit-based curriculum have organized regular meetings in which teachers may gather to discuss the implementation of the kits. Some projects have worked to build the expertise of in-school teacher leaders who are available for individual support to their peers. Another follow-up strategy has been for teachers to develop personal action plans with support provided throughout the year by mentor teachers or project staff.
- Active involvement of teachers
The LSC projects are clearly attentive to actively involving the teachers as key players in their own professional development. Evaluators described a number of mechanisms designed to engage teachers as active participants and leaders of professional development in their districts.
A number of projects incorporate teacher-leader models designed to build capacity of teachers within the schools and districts, and to reach larger numbers of teachers. Other strategies for encouraging teachers' involvement include options for teachers to select from multiple professional development activities; facilitating active sharing, reflecting and collaboration in small groups; fostering school-based implementation built on "school-site learning communities"; and assembling action research teams that offer teachers the opportunity to network and share with each other.
Challenges Faced by LSC Projects
Interestingly, some of the same features highlighted as strengths by some evaluators were identified by other evaluators as areas of challenge for their projects. Aspects of the professional development most often described as needing increased attention for improvement were:
- Time for participant reflection;
- Attention to content;
- Attention to classroom application;
- Consistency in the skills and expertise of facilitators;
- Attention toward equity; and
- Teachers' use of available resources.
The overlap in features most likely to be mentioned as strengths when they are present and as areas in need of improvement when they are not include: time for reflection, attention to classroom application, high-quality content, and skill and expertise among facilitators. The following comments of evaluators are representative of the concerns expressed by some.
Regarding the need for increased time for reflection:
"Although there was active learning going on in this and other activities during the session, there were missed opportunities for reflection, individually and as a group on what was being taught and learned. That is, although teachers were asked good questions, for example comparing actual [data] with predictions and providing what they thought accounts for differences, the discussions tended to be very limited in time and scope."
"Complex ideas were demonstrated in an interesting way, but some of the teachers did not internalize the concepts. The minds-on portion of the workshop was missing. As a result, the teachers may have some difficulty developing assessment tools and questioning skills when introducing the concepts in their classroom. The workshop seemed more of a show than an integration of content, process, reflection and application. The teachers needed to reflect on the demonstrated concepts and perhaps discuss age-appropriate applications of the concept."
Regarding content-particularly finding an appropriate balance in the content and process of inquiry:
"To elevate the quality of the kit workshops still higher the focus needs to expand to include an emphasis not only on the procedures and organization of the kits, but on the science content and the conceptual development embedded within them. Sole focus on the mechanics of the kits does not allow teachers time to reflect on their teaching practice or the rationale for hands-on, inquiry-based science education. This may lead participants to view kit-based science teaching as an end rather than a means to exemplary inquiry-based science instruction."
Regarding attention to classroom application:
"The teachers needed more time to consider classroom applications of resources, strategies, and techniques introduced in the sessions. For example, in one of the sessions observed, the participants viewed a video called "A Private Universe" which illustrates how students often develop misconceptions about science concepts and suggests methods to identify and resolve these misconceptions. The teachers in this session, however, were unclear as to how to use this information in their own classrooms. The method demonstrated in the video, individual student interviews, was not an option for most teachers, and they needed time to discuss other strategies."
Regarding the skill and expertise of facilitators:
In some cases, lapses in quality of the professional development were due to facilitators whose expertise in content areas was greater than their knowledge of appropriate professional development strategies. In other cases the challenge was to achieve consistency among teacher leaders working with peers. For example, one evaluator made this observation:
"The quality of those sessions was somewhat variable. The presentations made in the fall for colleagues were exceedingly well-executed and reflective of best practice, keeping in mind that many of the activities had been modeled for them by [project staff] in their first summer of professional development. When they were on less familiar ground [the] sessions were not always as well-executed."
Regarding increased attention toward equity:
In addition to the general need for equity issues to "gain visibility", evaluators commented on the challenge of establishing equity as a priority for participants:
"A major focus of the professional development sessions during the summer and year is how to successfully implement the science curricula in a manner that is accessible to all students. In addition, this summer's institute had a series of workshops on cooperative learning as a direct attempt to address the issue of access for new and beginning teachers. Although [the project] is accomplishing some things in the area of equity, the associate superintendent for curriculum and instruction believes that much more needs to be done in this area as test scores for the African-American and Latino populations have not showed expected gains."
"One issue that must be dealt with in the future is the perception of teachers that they know all about equity already and they do not understand why time is spent on this topic when there are so many other content, assessment, and procedural topics that they would like to hear about in professional development sessions."
Regarding use of available resources:
Teachers' failure to use existing resources was described as a challenge for a number of the projects. In most cases the issue was that teachers were not taking advantage of opportunities available to them, as illustrated by these evaluators' comments:
"While monthly meetings, content courses, and [teacher] assistance is available to classroom teachers, many do not make use of these resources..."
"Generally, [the project] is providing high-quality professional development activities by teachers are not feeling supported in the classroom implementation of what they have learned. One area where [the project] attempted to address this need was the creation of kit clubs. Unfortunately, not many teachers attended these even though they would have been paid $20 for every two-hour after-school session they attended."
Teacher Ratings of the Quality of LSC Professional Development
As part of the core evaluation, evaluators of Cohort 1 projects were asked to interview a random sample of 10 teachers who had participated in the project's professional development activities. Among the teachers interviewed, over two-thirds of the group had received more than 30 hours of LSC professional development; the others had participated in 10 to 30 hours of professional development. As seen in Figure 12 the majority of teachers had quite positive impressions of the professional development in which they had participated.
Comments of participants who were most positive about the professional development indicate that high-quality curriculum materials were a particularly important component of the professional development. Teachers described curriculum materials as "complete and comprehensive," providing "easy access to materials" and access to "meaningful investigations." Teachers were also very positive about their experiences with inquiry-based science, the support provided through on-going activities, and strategies that involved lead teachers or core teams at their schools.
Teacher Assessment of LSC Professional Development Activities
Even teachers who had overall very positive comments about the LSC professional development sometimes had suggestions for improvement. A number of comments focused on the need for more direct experience with the type of teaching they were striving to implement through the LSC project. A couple of participants commented, for example, that they would like to observe teachers using inquiry-based or constructivist strategies; another participant expressed the desire to have a visiting teacher come to his classroom to demonstrate these instructional strategies.
Only a small proportion of teachers who were interviewed were described by evaluators as either "somewhat" or "mostly" negative, and all of these teachers came from 3 of the 8 Cohort 1 projects. Those teachers who were negative about the professional development they had experienced complained of being unclear about expectations of the district and needing professional development that better met their particular needs with regard to content or pedagogical strategies.
Ratings of Individual Professional Development Sessions
Observations of selected professional development sessions at each LSC site are key to the core evaluation analysis of the quality of the professional development offered by LSC projects; although evaluators were asked to take other factors into account when making continuum and component ratings of their project, observations were the main standardized data source.
The following sections describe elements of professional development that observers were asked to rate for each professional development session observed, including:
- Design of the session;
- Its implementation;
- The disciplinary and pedagogical content;
- The character of the culture of the session;
- The likely impact on participants' capacity for exemplary science and mathematics instruction; and
- The likely impact on participants' leadership capacity.
Each section includes details about the pertinent indicators that contributed to evaluators' ratings of the individual professional development sessions observed for the core evaluation, as well as the percent of sessions receiving overall ratings of 4 or 5 on a 5-point scale.
Design of Professional Development
Evaluators were asked to assess the design of professional development sessions by rating a series of individual indicators based on current understandings of best practice. Several of these indicators received high ratings (4 or 5 on a five-point scale) in the majority of observed sessions. Those indicators that were most often highly rated included the following:
- The appropriateness of strategies used in the session for accomplishing the purposes of the LSC professional development (78 percent);
- How well the session encouraged a collaborative approach to learning and incorporated tasks and interactions consistent with a spirit of investigation (75 percent); and
- How effectively the session built on participants' knowledge of content, teaching, learning, and/or the reform/change process (71 percent).
Somewhat fewer sessions were rated highly on:
Professional Development Session Ratings - Design
- How well the strategies reflected attention to the varied needs and learning styles of the participants (65 percent); and
- Whether the design of the session provided adequate time and structure for reflection (53 percent).
Evaluators were also asked to provide an overall rating for the design of the session. As can be seen in Figure 13, 74 percent of the professional development sessions received overall design ratings of 4 or 5.
Implementation of Professional Development
Evaluators rated several indicators regarding the implementation of the professional development session. Ratings on the implementation tended to be lower than those on design. The indicators most frequently rated 4 and 5 were:
- Whether the facilitator's background and expertise enhanced the quality of the session (82 percent); and
- The appropriateness of the pace of the session for adult learners (69 percent).
Fewer LSC professional development sessions were rated highly on other indicators, including:
Professional Development Session Ratings - Implementation
- How well the session modeled questioning strategies that are likely to enhance the development of conceptual understanding (60 percent); and
- How well the session modeled effective assessment strategies (42 percent).
As can be seen in Figure 14, about two-thirds of the sessions received overall ratings of 4 or 5 on the quality of implementation.
Content of Professional Development
Evaluators rated both disciplinary and pedagogical content of the professional development sessions. The following individual indicators were highly rated in most of the observed sessions:
- The appropriateness of the disciplinary and pedagogical content for the participants' backgrounds (80 percent);
- The extent to which appropriate connections were drawn to real-world contexts (70 percent); and
- The soundness of the disciplinary/pedagogical content and how appropriately it was explored or presented at the session (68 percent).
As can be seen in Figures 15 and 16, 65 percent of projects were rated 4 or 5 on disciplinary content and 67 percent were rated a 4 or 5 on pedagogical content. Note also that while ratings on disciplinary content were similar for the two cohorts, Cohort 1 projects were more likely than those in Cohort 2 to receive the highest possible ratings for pedagogical content.
Professional Development Session Ratings - Disciplinary Content
Professional Development Session Ratings - Pedagogical Content
Professional Development Culture
Most individual indicators for professional development culture were rated highly by evaluators. Indicators that received ratings of 4 or 5 in most of the sessions included:
Whether interactions reflected collaborative working relationships among participants and also between facilitators and participants (77 percent);
Professional Development Session Ratings - Classroom Culture
- How encouraged participants were to generate ideas, conjectures, and propositions and to what extent investigation and risk-taking were valued (74 and 70 percent, respectively);
- Whether participants were intellectually engaged with important ideas relevant to the focus of the session (72 percent); and
- How highly valued intellectual rigor, constructive criticism, and the challenging of ideas were during the session (65 percent).
Figure 17 shows that 73 percent of the sessions received ratings of 4 or 5, with fairly similar ratings for the two cohorts.
Impact on Participants' Capacity for Exemplary Science/Mathematics Instruction
LSC evaluators were asked to rate how likely they felt the session was to have an impact on teachers' capacity for exemplary science/mathematics instruction. Ratings for this area were quite low, with 60 percent or fewer projects receiving ratings of 4 or 5 on any of the indicators:
Professional Development Session Ratings - Capacity for Exemplary Instruction
- How likely the session was to enhance participants' ability to identify and understand important ideas of science/mathematics (60 percent);
- Whether the session enhanced professional networking among participants (60 percent);
- The likelihood that the session would enhance participants' ability to plan and implement exemplary classroom instruction (55 percent); and
- Whether the session was likely to help participants better understand how students learn (48 percent).
As seen in Figure 18, only 50 percent of the sessions received overall ratings of 4 or 5 in this category. Apparently, evaluators often saw individual elements of high quality, but felt that the experience as a whole was not likely to translate into a marked positive improvement in teaching.
Impact on Participants' Leadership Capacity
The 38 observed sessions in which leadership development was a focus were rated on their likelihood to have an impact on participants' capacity to be leaders in science/mathematics instructional reform. In these sessions, the indicators most frequently receiving ratings of 4 or 5 were:
- The likelihood of the session to enhance participants' understanding of the reform/ change process (70 percent); and
- Whether the session would help participants be more confident in their ability as leaders of science/mathematics reform efforts (68 percent).
Fewer of the observed sessions that focused on leadership-building received high ratings on other indicators such as:
Professional Development Session Ratings - Leadership Capacity
- Whether it would enable participants to plan/implement exemplary professional development (54 percent);
- How well the session enhanced professional networking among participants with regard to leadership (50 percent);
- Whether the session would help teachers identify and understand important strategies for reform (44 percent); and
- Whether the session enhanced participants' understanding of adult learners (37 percent).
As seen in Figure 19, 64 percent of the sessions received overall ratings of 4 or 5 in this area; however, there were striking differences in the ratings of Cohort 1 and Cohort 2 projects. Only 5 of the 13 Cohort 1 leadership sessions (38 percent) were rated a 4 or 5 on likelihood of building leadership capacity compared to 18 of the 23 (79 percent) Cohort 2 sessions.
Capsule Ratings of Observed Professional Development Sessions
Evaluators were asked to provide an overall "capsule rating" of each observed session, taking into account the relative impact of each of the component areas. As would be expected, given the high overall continuum ratings assigned by evaluators for the professional development systems, ratings for individual professional development sessions were similarly quite favorable. Very few LSC sessions were rated as ineffective professional development (Level 1), and fewer than 10 percent were rated at Level 2, having quite limited likelihood of helping participants implement exemplary science/mathematics instruction or be leaders in reform. (See Figure 20.) More than 60 percent of the observed professional development sessions received ratings of 4 or 5, with Cohort 1 sessions somewhat more likely than Cohort 2 sessions to be considered "exemplary professional development."
Professional Development Capsule Ratings of Observed Sessions
The vignettes below illustrate a variety of types of sessions observed and some of the strengths and areas of need noted by evaluators.
Professional Development Session #1:
"Learning Content and Pedagogy Through Kit-Based Activities"
Third and fourth grade teachers were engaged in professional development around a kit-based unit on water. One session focused on a lesson involving construction of a water thermometer. There were multiple goals for participants: (a) to experience learning by inquiry, (b) to try out the activity, and (c) to learn about instructional strategies. The facilitator began by questioning the participants to learn what they already knew about thermometers, modeling the kinds of questions and problems teachers might pose with their own students.
Pairs of participants constructed thermometers, and were engaged in making predictions, investigating how those predictions could be tested, and actually carrying out mini-investigations. The implementation was a real strength of the session. The facilitator knew just when to ask thought-provoking questions to keep participants thinking and investigating further.
Participants were encouraged to share their ideas about thermometers--e.g., how they work, what might go wrong, and how to deal with that. These discussions led to generating ideas about how their "hypotheses" might be tested. At appropriate points throughout the session, the facilitator wove in discussions about classroom application and classroom management issues. Overall, the facilitator used a kit-based unit to skillfully model effective content, process, and inquiry.
This session received ratings of 5 in the areas of design and implementation; ratings of 4 on content, professional development culture, and likelihood of enhancing teachers' capacity to implement exemplary science instruction. The session was placed at Level 5 for the overall capsule rating, indicating this was an exemplary professional development session.
|Professional Development Session #2:
"Study Groups: Classroom Observation and Discussion"
A third-grade team of teachers met at the school on a regular school day for a four-hour professional development session focused on the district's recently-adopted mathematics curriculum and aimed at refining teachers' knowledge of the content and pedagogical strategies needed to successfully implement the curriculum. Self-study/discussion teams organized by grade-level, were meeting regularly around each of the modules as they were implemented throughout the school year.
On this day, early in the school year, the entire study team and a lead teacher/facilitator on leave to work with the LSC project would observe a class taught by two of the study-team members. The two teachers had combined their classes, and were team teaching a lesson from the current module on fractions. Prior to the lesson, the group met to discuss the lesson, reflecting on what they hoped students would get from the lesson; potential questions the teachers might ask to prompt student inquiry and reflection; some potential pitfalls that might be encountered; and how they might orchestrate some informal assessment of students within the lesson. The group then gathered in the classroom to observe the lesson.
Following the 45-minute lesson, the study team reconvened to discuss what they had observed. At first the members of the study team were very reserved. They congratulated two team teachers on their implementation, pointing out what they felt were key strengths of the lesson. They discussed specific incidents that were telling demonstrations of what students knew and what they did not know about fractions. There was some confusion and contradiction in different teachers' assessment of where the team should go next with their students.
Particularly critical for setting the tone of the discussion was the non-threatening way in which the teachers discussed the lesson and the willingness of the team teachers to reflect openly and honestly with the group. An especially fruitful part of the conversation occurred when one teacher pointed out to one of the team teachers that she had called only on boys during the whole group discussions. The teacher's initial expression of surprise, followed by acknowledgment of a pattern she had not been aware of, led to an in-depth discussion of that and other issues of equity. The facilitator's role throughout the session was to help keep the discussion focused issues raised by the group, and to prompt reflection about implications of what they had observed.
The session received ratings of 5 on design, implementation, and professional development culture. Content was rated 4. There were a couple of areas in which teachers were clearly struggling with content themselves. Those areas were not addressed sufficiently in this session, but the facilitator suggested they orient their next meeting around a more in-depth exploration of those concepts that were proving problematic for the students and teachers, alike. Overall, this session was described as "exemplary," receiving a capsule rating of 5.
Professional Development Session #3:
"Exploring Student Assessments"
Teachers of grades K-6 met with facilitators form the LSC project staff in the final half-day session of a week-long workshop on assessment. This session focused on a participant-identified need for assessment strategies for cooperative learning groups. The session had a dual purpose of enhancing the participants' expertise in designing/scoring student assessments, and building a network of teachers who could serve as "assessment experts" in the districts.
The day began with a whole group discussion of an article the participants were to have read the previous evening, on the role of cooperative learning in assessing and communicating progress in student leanings. This article was one of many resources available in a packet provided for each participant.
Following the discussion of the article, teachers worked in groups to develop criteria for selecting reporting options to effectively communicate student progress. Group discussions were characterized by active participation and a collaborative approach to addressing the issues. The facilitators circulated among the groups, encouraging participants to use each other as resources and modeling effective questioning techniques in the process.
A third component of the session was an in-depth discussion of equity issues in assessing students' work. Discussions focused on whole group versus individual grading, heterogeneous versus homogeneous ability grouping, single-gender versus mixed gender groups, and the modification of rubrics to accommodate students with special needs.
The week-long workshop was to be followed up with additional activities designed to further enhance teacher understanding of assessment and provide opportunities for continued sharing of experiences and building the network of assessment experts.
This session received ratings of 3 on design and content and ratings of 4 on implementation and professional development culture. The key weakness noted by the evaluator was the "generic" nature of the work around assessment strategies. There was only incidental application of the strategies to the mathematics modules teachers were to implement in their classrooms. The overall ratings for the session was 4, indicating that the session was very effective and likely to enhance the participants' classroom instruction.
Professional Development Session #4:
"An Attempt to Address Science Content"
A professional development session for first-grade teachers was designed to help the teachers learn the content targeted by a kit-based unit on liquids, and to become familiar with the activities included in the kit. Some participants were new to the system; others were veteran teachers who wanted to learn more about the curriculum that they would be implementing in their classes in the coming year. The session was led by two facilitators-a lead teacher in the district and a scientist from a local university.
On the previous day, the lead teacher had taken participants through several activities included in the kit. Most of the two-hour session on this day was focused on introducing the "big ideas" of the unit to the whole group, with the scientist presenting content about atoms, measuring, volume, density, and phase changes. While the presented information was accurate, there was little integration of the content with the activities of the previous day. The presentation ended with an activity in which the entire group role-played the movement of molecules at different temperatures.
In the last half-hour of the session, the teachers met in small groups to discuss the content covered in the morning presentation and to identify questions they may have about the content or the activities included in the unit. The scientist and lead teacher initially circulated among the groups; however, both of the facilitators spent the majority of the remaining time with one of the four small groups to address questions about one of the activities from the previous day. The session ended with an announcement of when the group would meet next for professional development on a second unit to be implemented in all first-grade classes.
This session received ratings of 2 on a five-point scale on its design and implementation; and a rating of 3 on content. Due to the lack of participant engagement and absence of collaborative or participant-centered interactions, the session was rated 2 in professional development culture. Overall, the session received a capsule rating of 2, indicating that there were elements of effective professional development, but that there were some major areas in need of improvement.
In summary, the LSC projects are providing high-quality professional development to their targeted teachers. All projects were rated at level 3 or above on a five-level continuum, indicating that the projects are either transitioning toward, or have already achieved a level of professional development that incorporates many features reflective of current standards-based approaches and that these activities will likely enhance participants' capacity to provide high-quality science and mathematics instruction to their students.
Each of the component areas of professional development-the design, the implementation, the disciplinary and pedagogical content, and the culture of the sessions-received very high ratings, as well. Mean ratings for each of those areas, as well as the likelihood that the project would enhance teachers' instructional capacity were consistently between 3.5 and 4.0 on the five-point scale. Projects that focused on developing the leadership capacity of teachers in the district were rated somewhat lower in that area, with a mean rating of approximately 3.3.
The LSC projects are distinguished from teacher enhancement projects of the past by their systemic focus, including professional development for all teachers of science/mathematics at the targeted grade levels, the implementation of exemplary instructional materials, and the alignment of district policies and practices with the reform vision.
Evaluators found that the professional development sessions generally incorporated state-of-the-art strategies, including activities that were relevant to the teachers' work in classrooms, modeling of effective pedagogical strategies, opportunities for teachers to reflect and talk with peers, strong components of follow-up and support, and active involvement of teachers as key players in their own professional development. The sessions were generally led by competent and skilled facilitators and characterized by very positive cultures that encouraged the engagement of teachers as professionals in learning communities.
- One project in Cohort 2 did not collect professional development observation data in 1996 because of a late start-up date; this project's baseline questionnaire data are included in the cross-site analyses.
- HRI re-rated all 1995-96 professional development observations based on the updated continuum developed for the 1997 observation protocol. Evaluators were asked to verify these ratings and given the opportunity to make changes as needed. Ratings reported here have either been provided by or verified by the evaluators.
- A number of projects reported observing professional development sessions in addition to those rated for the core evaluation; these additional observations were considered by evaluators when making overall ratings of the project's professional development system.
- A project placed at Level 3.5 was included with the Level 3 group in this report.
- The elements of individual professional development sessions discussed in the following sections parallel the components of the professional development system reported earlier (Figures I-6 and I-7). When rating the system components, evaluators had been asked to rate each component holistically, using all the data available to them. In contrast, ratings in the following sections are based solely on individual professional development sessions observed for the core evaluation.