Paper

Evaluators' Assessment of LSC Professional Development Activities

Evaluators were asked to rate several component areas for the overall professional development program, including: the design of activities, their implementation, the content of the program, and the professional development culture.

As can be seen in Table 11, both K- 8 mathematics and 7- 12 mathematics projects received extremely high ratings in each component area, ranging from 92 to 100 percent 4's and 5's. The highest rated area for K- 8 science projects was the professional development culture, with 79 percent of projects receiving ratings of 4 or 5. This is a surprising result, given that K- 8 science teachers gave high ratings to the LSC professional development and, as will be described in Chapter Six, K- 8 science projects appear to have had a greater impact on classroom instruction than have the projects focusing on mathematics.

Key Strengths

LSC project evaluators were asked to assess the key strengths of the professional development programs and to identify aspects that were in need of further attention. Their comments provided greater detail about features that were important to the quality of the professional development programs.

1. LSC projects are framed around national standards, creating a common vision among participants.

The national mathematics and science education standards have been important in providing a broad framework for reform efforts. Several evaluators noted the importance of participants' awareness and understanding of these standards for enhancing their capacity to provide high quality mathematics and science education in their classes.

Cohort 1 projects were in their third year of funding, and a number of evaluators of these projects described an emerging community of learners with a common vision of mathematics and science education. They noted that the projects were characterized by: (1) similar beliefs among participants (e. g., about the importance of inquiry, what constitutes high quality professional development); (2) communication across grade levels (and across districts in multiple- district projects); and (3) involvement in the national reform arena. These features, in turn, helped to establish a strong, supportive professional development culture.

2. LSC professional development sessions are typically reflective of best practice.

Evaluators of the LSC projects consistently noted the high quality of the design and implementation of many of the professional development sessions they observed. A number of features, routinely identified as strengths of the sessions, indicated that the design and implementation of the LSC professional development were frequently reflective of current notions of best practice. The following features illustrate the strengths of the professional development sessions highlighted by the LSC evaluators.

• Tightly linking professional development to exemplary instructional materials
  

A major "finding" of the PIs and demonstration teachers, this past year, has been that teaching the actual curricula to be used in the classroom has been the most effective way to teach the embedded mathematics teachers need. In the past, very good in- service presentations have been offered, but the question of transfer had always loomed large. At least, in the initial phases of [the project], by using the very materials that teachers will use with students, no transfer is required. Because these curricula are complex and demanding, preparing teachers carefully to use them is important, highly valued by the participants and provides sound mathematics education.

Similarly, an evaluator of a K- 8 science project talked about the project's focus on introducing instructional materials and helping teachers learn about their new kits:

Teachers, when interviewed, consistently spoke about the relevance of the training sessions in helping them to know what science concepts the kit covered and how to set up the investigations.

It's always relevant. Lesson plans are well laid- out. You know what to expect in the classroom once you get there. We adapt things as we go, on our own, but [the project] provides a good guideline.

* * *

The training is helpful. They go lesson- by- lesson, notes and ideas on how to do set ups, presentation of information, and in the afternoon of the second day, you get to do 'make and take. ' That's when you make all the charts and things necessary for teaching the module.

• Modeling appropriate instructional strategies
  

Observers were impressed with the amount of hands- on activity built into the professional development sessions, especially those led by teachers. Most of the summer sessions were structured so that teachers worked on tasks in pairs or small groups for a good bit of the time. Several of these sessions were very skillfully planned and implemented. One, in particular, a model of its kind, put teachers through a series of activities, the main objective of which was to have them experience and understand the difference between hands- on activity that is inquiry- driven and hands- on activity that is structured, and between hands- on activity that is driven by competition and one that requires cooperation to work. Its only flaw was insufficient time for teachers to reflect on these differences.

• Making explicit connections to classroom application
  

The implementation of the pedagogy courses goes beyond basic introduction to the [FOSS] kits. The instructors expect the teachers to explore the components and design of the kit activities and to determine how the kit fits within the greater scope and sequence of the [district] curriculum framework. The teachers have an opportunity to work with the kits in collaboration with other teachers and discuss how they might be implemented both in classroom practice and within a school's overall science program.

3. Time for reflection is a critical feature of effective professional development.

Providing time for teachers to consider the classroom applications of what they were learning emerged as a critical feature in defining effective professional development sessions, as well as professional development programs, generally. Providing teachers with the opportunity to reflect on what they were learning, and to share strategies and ideas with each other were features most often praised when included in a project's design, and most often criticized, when absent. These comments are illustrative of evaluators' and teachers' praise for reflection time:

While time [during professional development sessions] is always used to the last minute, adequate time is given to fully develop ideas. A key strength in terms of design and implementation is the use of reflection with all aspects of the project. Both the structure supporting reflection and allowing time to reflect are built into almost every professional development activity in this project. While some teachers in a Summer Institute teased the project staff about "yet, another piece to write," many commented about the value this time to reflect held for them.

* * *

Feedback received from the teachers after site facilitator- led [professional development] at the schools indicated that teachers are having some opportunities to reflect with peers, and value those opportunities highly. One teacher wrote: "The afternoon to work with the ideas and materials pertaining to the morning session were the most productive hours I have spent during an inservice day in as many as I can remember. I pulled out the guide again and shared with my grade- level partner.... It was helpful, professional, productive, logical. It was progress."

4. LSC projects typically provide support as teachers implement what they have learned.

Support as teachers implement what they learned, consistently identified by teachers as a critical feature, was targeted as a key strength by evaluators, as well.

Evaluators described numerous mechanisms for teacher support. Comments such as the following indicated that LSC projects were routinely designed to provide support via ongoing professional development.

The combination of traditional workshops along with ongoing, regularly scheduled classroom assistance from the science specialist clearly holds the greatest potential for the success of the project.

* * *

[The project's mathematics facilitators provide] school- based support by working with individual teachers or with groups of teachers in their classrooms or in school- based meetings.

* * *

Professional development sessions focus on science concepts within one of the three major themes: astronomy and space science (Year 1), life science (Year 2), and physical science (Year 3). Follow- up sessions provide teachers with opportunities to discuss their successes and failures in implementation of science units and activities.

A few projects focused heavily on providing a supportive context for implementing activities, tightly linking teachers' implementation of activities with opportunities for reflection and feedback regarding their classroom experiences. For example, one project scheduled workshops at two- week intervals. Teachers focused on several lessons in a professional development session, tried them out in their classrooms, and returned two weeks later to share and discuss their experiences with colleagues and workshop facilitators. Teachers then explored the next round of lessons and repeated the process of teaching, reconvening, and discussing the activities in two- week cycles.

Teachers in another project used summer school and inter- session periods (i. e., the time between year- round school sessions) to "try out" what they were learning through professional development in an environment that was supportive of risk- taking. Classes were smaller, teachers were able to focus on the content, and they had the support of peer teachers with whom they planned, tried out, and discussed activities prior to teaching them to students.

5. LSC professional development is typically needs- based and responsive.

Evaluators frequently identified the relevance and responsiveness of the professional development programs as a key strength in both the culture of the sessions and the quality of the design and implementation. Projects attended to participants' needs in a variety of ways. Some projects engaged participants in the planning process, e. g., by developing a menu of training topics based on input from teachers who would participate in the training, or by assessing teachers' needs in the course of planning the professional development. This evaluator's description is illustrative of the close match between the professional development and participants' needs in many of the LSC programs.

Expectations were mutually agreed upon. While it was obvious that the presenter had material to share and goals to accomplish, participants also had opportunities to share and chances to articulate and accomplish their own goals. From the discussion of participants' expectations, it was clear that someone had worked prior to the session to listen to third- grade teachers and find out what were their topics of interest and need. There was a close match between what they stated as their hopes and the day's planned agenda. This kind of correlation does not happen without careful needs assessments before the meeting.

It was quite common for evaluators to describe ongoing attention to participant needs, as well. The excerpts below are illustrative of how projects actively solicited participant feedback and modified designs in response:

Teachers complete comment cards at the close of each session in which they can make suggestions and requests for the next session. The facilitators respond to the requests, making adjustments in their planning when necessary. This [was] particularly useful during the summer workshops. As a result, participants feel some ownership and remain actively involved.

* * *

By design, project staff attempted to be responsive to participants' concerns and needs. The design included a discussion session at the end of the first training week and an evaluation survey to provide project staff with feedback. As a result of this feedback, a break- out session was added to the second training week, in which [lead teachers] could participate in discussion groups focused on specific topics or select catch- up time to review the extensive written materials that had been distributed to them. At the participants' request, project staff also included a discussion on how to present the project at their initial department meetings which would be held during the week when teachers returned to school to prepare for the new school year. One participant wrote in the second week evaluation, "A special thanks for actually listening to our needs and responding so quickly and appropriately."

Designs that offered a menu- approach or "multi- tiered" options were particularly noted for effectively meeting multiple teachers' needs. One project, for example, was praised for its design that offered " a diverse range of well- thought out, rich professional development opportunities." The evaluator's description of the program is included in the box below.

6. Most LSC projects encourage a collegial environment.

LSC evaluators were consistently positive about the culture of the professional development programs. Projects were often successful in creating a collegial environment in which teachers and facilitators were routinely respectful of each other as professionals and teachers' contributions were sought and valued. Teachers were typically engaged as active participants in the sessions, fostering a collaborative approach to their professional development.

One evaluator noted, for example, the productive sharing that occurred when teachers attended professional development in small, grade- level groups:

This configuration provided the possibility of sharing and cooperation among teachers from various schools. Observations of these professional development training sessions verify that networking among teachers is common. Some teachers went so far as to bring samples of resources (which included books, bibliographies, posters, and samples of student work) which they had gathered while teaching the kits. They offered materials and support to each other, both formally and informally. Trainers were sensitive to these teacher- generated additions to the agenda. This resulted in formal, as well as informal idea exchanges and swap and shops. These components of the professional development training enhanced the collegial spirit among teachers.

7. Most facilitators demonstrate knowledge and skill in implementing the LSC professional development.

Facilitators played a key role in the implementation of effective professional development. Projects recruited facilitators from a variety of sources: some teamed project staff with scientists, some used facilitators from outside the district, some relied on teacher leaders from within the district. Whatever the source or affiliation of facilitators, evaluators commented on the importance of their expertise-- in the targeted content area, in classroom instruction, or both.

One evaluator, for example, made this comment about the project's facilitation team comprised of the principal investigator, co- principal investigators, project director, district science resource staff, and two faculty members from a state university:

The subject matter and pedagogical expertise provided by [university] science educators, combined with the organizational skills, classroom experience, and ongoing work in curriculum development, professional development, and science education reform, make for a powerful and effective facilitation team. They all receive praise for their fine work. With their commitment to the LSC effort, they are undeterred by barriers or setbacks. They strategize and then proceed to remove, go around, or jump over any roadblocks.

It was also important to have facilitators who were able to guide activities in a way that both fostered a supportive and collaborative environment and offered high quality professional development experiences for the participants, as illustrated in this excerpt from an LSC evaluation report:

Facilitators circulate among the groups, ask questions of the groups who may need a prompt, hand a transparency to one or two groups who may be nearly finished so that they can prepare a presentation, and bring the groups back to the whole group when most are ready for presentations. Facilitators may steer discussion in a particular way in response to a presentation, but usually the other participants are quick to interject their disagreement, discuss points of conflict, offer an alternative presentation, and acknowledge the contribution of their peers.

For teachers, it was especially important to have facilitators with classroom experience with curriculum materials. One evaluator made the following comment, and provided illustrative quotes from several teachers in the project:

Teachers appreciated learning from their peers at the kit implementation workshops. They liked learning "insider tips" from teachers with recent classroom experience of teaching the kit.... The credibility of teachers teaching teachers is much higher than that of outsiders coming to tell the teachers, yet again, how they should be teaching. As one teacher said..." I liked it being taught by teachers who have used it and they tell you what is really feasible and what you might want to change. That is really helpful."

8. LSC projects are enhanced by systemic features that attend to the larger context.

Administrative support for teachers and the reform effort was a feature that evaluators identified as important in both creating a positive environment within the professional development program and contributing to the soundness of the design and implementation of the program. In some projects, administrative support was fairly well- established; in others, the project focused attention on building the capacity of administrative leaders to support the efforts toward improved mathematics and science instruction.

A variety of other support mechanisms was described by evaluators, including school- based support for targeted teachers; support among the teachers within the district who identify themselves as a "community of learners... working on a joint effort"; having a reliable system for kit refurbishment and delivery; and having release time available for teacher leaders' professional development activities.

Evaluators noted the contribution that a variety of systemic features made to the soundness of some programs' designs. For example, in one project, the school district hired recent pre- service graduates to teach summer school classes with guidance from project staff. After class, these beginning teachers discussed the content and pedagogy of their classes with each other and with project staff. In some cases, projects linked their LSC professional development with other district reform efforts, focusing on areas such as literacy and technology. Additional features of program designs that reflected the systemic character of the projects were the integral involvement of a variety of stakeholders in instructional materials review and selection, and encouraging collaboration among districts (e. g., setting up mechanisms for districts to share resources).

The following excerpts from evaluation reports are illustrative of some of the systemic efforts of LSC projects:

The plan for the [LSC] professional development activities was based on the clear, shared goals of the district and schools to adopt a K- 12 mathematics curriculum that the entire educational community-- district personnel, school administrators, teachers, parents, and the community-- could buy into. This need presented an opportunity for the district to develop and elicit the informed participation of its constituents to select mathematics curricula that were consistent across the grades and reflected the frameworks and standards of the NCTM and the content and practices of a reform mathematics curriculum.

* * *

During the kit review process, the Project infused a literacy component for each kit. This literature connection takes the form of a packet of non- fiction books for children, related to the science ideas in the kit. During interviews, teachers said they appreciated having this direct link between science and reading incorporated in the kits. Also, increasingly, the Project is explicitly helping teachers to use journals as a vehicle for: (1) achieving student literacy and (2) as a way of assessing the nature and extent of student understanding. In a district that is heavily focused on basic literacy, it is very important that the science program develop sophisticated strategies for making the two- way linkages between science and reading and writing.

Areas in Need of Further Attention

While teachers were typically quite positive in their assessments of the quality of the LSC professional development programs, they occasionally complained about particular facilitators who lacked appropriate skills; some found the pace of sessions too slow or too repetitive; and some felt their project was not well- balanced in content coverage. One project, in particular, appeared to have alienated a number of teachers by implementing their program in a way that teachers felt was "condescending" and did not build on teachers' experiences as knowledgeable professionals.

Overall, specific weaknesses among the LSC projects were fairly diverse. Areas that were mentioned by evaluators of several different projects are described in the sections below.

1. Some projects need to keep more of a focus on key mathematics and science concepts.

The LSC projects are designed around the use of exemplary mathematics and science curricula, many of which are kit- based. Some evaluators noted a tendency for projects to focus so much on individual activities or the procedural aspects of the units, that the conceptual focus (i. e., the key mathematics and science concepts we want students to learn) was lost.

Several evaluators saw a need to relate the content of professional development sessions to the "big ideas" of mathematics and science.

The content of the professional development sessions was worthwhile and appropriate for the purposes of the session and backgrounds of participants.... An area that might be addressed in future professional development activities is the explicit linking of the content of each individual session to the overall content of the professional development: the big ideas of algebraic thinking.

* * *

According to the professional development observations, there was some question about whether or not the "big ideas" about science were evident during the... workshops. The primary focus of these sessions is, by design, on the procedural aspects of teaching the kits. That is, it is most important that teachers be able to leave the workshop with the "nuts and bolts" knowledge needed to implement the kits in their own classrooms. This, by and large, seems to have happened. However, it also seems that teachers need more than a "worms- eye" view of the kits. They need some sense of the larger vision of inquiry, or the science content that underlies the kits, and possible assessment strategies for the outcomes of the kit.

Evaluators frequently identified the need to deepen teachers' knowledge of content. However, underlying the comments of some evaluators was the belief that teachers' needs for enhanced content knowledge went beyond what the LSC professional development was able to address. For example:

It is unfortunate that the time for content development is limited, as teachers need to be versed in content beyond what they are expected to teach and there is little time to do so. Approximately, 25 percent of the teachers who taught [the curriculum] in 1996- 97 reported not having a degree or concentration in mathematics or mathematics education. This appears to be the case as well with the group who will teach [the curriculum] in 1997- 98. Co- directors need to work on a means to identify participants whose content background may be weak and arrange a means of content support with teacher leaders in the school.

2. Some projects provided insufficient time for teachers to reflect, discuss, and share ideas with one another.

Lack of time for discussion, application, and sharing of ideas was the aspect of professional development sessions most often mentioned by evaluators as needing further attention. Often the source of the problem was an attempt to do too much in the time available. The comments below are typical of evaluators' observations about the professional development they observed:

If something is lacking, it is, at this point, sufficient time for quiet reflection. Professional development has been 'jam packed' with knowledge of mathematics and of leaning theory, pedagogical modeling, and numerous 'tips' for smooth functioning classrooms. There has not yet been much time available to be devoted to thinking quietly, reflecting, on paper or otherwise, on practice.

* * *

Many sessions are designed to cover too much material within the allotted time frame. Facilitators of module- specific sessions often try to describe an entire module, lesson- by- lesson. Evaluating such a session, one participant urged presenters to "slow down, too much information, too fast. I realize we needed to cram a lot into a short time, [but it] seems a lot to comprehend." Because of such extensive coverage, time runs short and activities tend to be rushed and overly directive. Participants have no time to think about task objectives or to consider the potential impact of the task on student learning. The result, according to one participant, is that "teachers do not understand the relationship between activities... and the underlying concepts."

In addition to needing more time within the professional development programs, themselves, time to share ideas within the school setting was also identified as a key need, as noted by this evaluator:

In interviews, teachers were more explicit about the kinds of ongoing support they felt they needed to improve science instruction. Teachers wanted their workshop facilitators to visit their classrooms to validate their efforts, and to see how they were doing with the new curriculum and instruction. They also wanted their peers in the school, or in other schools, to visit their classrooms, and to go visiting themselves to other classrooms. They asked for mechanisms to create the same kind of opportunities in their schools to share ideas and talk about science with their colleagues that they enjoyed in the workshops.

3. Some projects needed to enhance the "systemic- ness" of project design and implementation.

Evaluators of a few projects described areas in need of a "big- picture" perspective, including the need for long- term, strategic planning for professional development in the district, the importance of maintaining a common vision amidst decentralization and staff turnover within the district, and the need to strive toward a common focus among the multiple reform efforts that tend to occur in districts over time.

Student assessment was one area singled out as needing additional attention. One evaluator, for example, described the pressure that teachers felt in a district that used standardized testing to assess the schools' (and teachers') success, and made this observation about teachers' needs for assistance in "negotiating assessment issues" and their need for more appropriate assessment strategies than those modeled in standardized texts.

The challenge facing teachers around assessment is twofold. One challenge is to make sure that they use and supplement [the math curriculum] in such a way that they feel confident that their students are mastering basic mathematical skills. The other challenge is to help teachers have a repertoire of classroom assessment procedures so that they know how well their students are doing with the new curricula. Most teachers currently report using some combination of traditional assessments with new methods encouraged by [the curriculum].... All teachers felt they could use more guidance and/ or training [in the area of assessment].