Paper

Highlights of the Year Two LSC Cross-Site Report

Introduction

LSC projects are expected to align policy and practice within the targeted district(s) and to include:

• A shared comprehensive vision of science, mathematics, and technology education;
• Active partnerships and commitments among stakeholders;
• A detailed self-study that provides a realistic assessment of the current system's strengths and needs;
• Strategic planning that incorporates mechanisms for engaging each teacher in intensive professional development activities over the course of the project; and
• A set of clearly defined, measurable outcomes for teaching, and an evaluation plan that provides on-going feedback for the project.

NSF contracted with Horizon Research, Inc. (HRI) to design and implement an evaluation for collecting data that could be aggregated across projects. The core evaluation system designed by HRI in conjunction with NSF and the project PIs and evaluators includes observations of professional development activities and science and mathematics classrooms; teacher and principal questionnaires; and teacher interviews. Project evaluators use these data, as well as any additional, project-specific data they may have collected, to address the following core evaluation questions:

0. What is the overall quality of the LSC professional development activities?
1. What is the extent of school and teacher involvement in LSC activities?
2. What is the impact of the LSC professional development on teacher preparedness, attitudes, and beliefs about science and mathematics teaching and learning?
3. What is the impact of the LSC professional development on classroom practices in science and mathematics?
4. To what extent are the district and school contexts becoming more supportive of the LSC vision for exemplary science and mathematics education?
5. What is the extent of institutionalization of high quality professional development systems in the LSC districts?

Project data sheets completed by the PIs provide some basic information about the 26 projects included in Cohorts 1 and 2.¹

• Roughly half of the Cohort 1 and Cohort 2 projects are single-district projects; at the other end of the scale, 4 projects involve more than 10 districts, including 1 with 21 districts.
• The 8 Cohort 1 projects and 18 Cohort 2 projects plan to involve a total of 28,239 teachers in 1,312 schools in 121 districts across the United States.
• By the completion of these projects, an estimated 706,000 students will receive instruction from LSC-treated teachers each year.
• 20 projects target K-8 science; 2 projects target K-8 mathematics; and 4 projects target both K-8 science and mathematics.

This report presents highlights from the 1995-96 project year, including results from the 8 Cohort 1 projects that were in their second year of funding and the 18 Cohort 2 projects that were funded in 1996.²

Description of LSC Professional Development

Core evaluation data to assess the quality of LSC professional development activities come from observations of the professional development activities, and to a lesser extent, from teacher questionnaires. Cohort 1 projects augmented these data with teacher interviews. Based on these data, evaluators were asked to rate the quality of the project's professional development in eight component areas and to provide an overall rating of the project's professional development system.

Evaluators observed a total of 180 professional development activities, an average of 7 per project. Using a standard protocol developed by HRI, they described the sessions they observed; they also assessed the appropriateness of the design of each session for accomplishing the stated purposes and the quality of its implementation. As can be seen in Figure 1, nearly half of the observed science sessions included physical science content, 37 percent focused on life science, and 33 percent had an earth-science content focus.

Major Disciplinary Content Areas of Observed Science Professional Development Sessions Figure 1

Major Disciplinary Content Areas of Observed Mathematics Professional Development Sessions Figure 2

Among projects targeting mathematics, the content areas of focus were most often computation and estimation, numeration and number theory, patterns and relationships, and mathematical problem solving. (See Figure 2.) While a few of the science sessions included emphasis on one or more mathematics topics (especially measurement), none of the mathematics sessions included science content as a major focus.

Figure 3 shows evaluators' overall "capsule ratings" of the observed sessions. Very few LSC sessions were rated as ineffective professional development (Level 1), and fewer than 10 percent were rated at Level 2, having quite limited likelihood of helping participants implement exemplary science/mathematics instruction or be leaders in reform. More than 60 percent of the observed professional development sessions received ratings of 4 or 5, with Cohort 1 sessions somewhat more likely than Cohort 2 sessions to be considered "exemplary professional development."

Figure 3

The sample vignettes that follow illustrate two sessions with similar purposes, but judged to be quite different in quality.

Professional Development Session #2: A professional development session for first-grade teachers was designed to help the teachers learn the content targeted by a kit-based unit on liquids, and to become familiar with the activities included in the kit. Some participants were new to the system; others were veteran teachers who wanted to learn more about the curriculum that they would be implementing in their classes in the coming year. The session was led by two facilitators--a lead teacher in the district and a scientist from a local university. On the previous day, the lead teacher had taken participants through several activities included in the kit. Most of the two-hour session on this day was focused on introducing the "big ideas" of the unit to the whole group, with the scientist presenting content about atoms, measuring, volume, density, and phase changes. While the presented information was accurate, there was little integration of the content with the activities of the previous day. The presentation ended with an activity in which the entire group role-played the movement of molecules at different temperatures. In the last half-hour of the session, the teachers met in small groups to discuss the content covered in the morning presentation and to identify questions they may have about the content or the activities included in the unit. The scientist and lead teacher initially circulated among the groups; however, both of the facilitators spent the majority of the remaining time with one of the four small groups to address questions about one of the activities from the previous day. The session ended with an announcement of when the group would meet next for professional development on a second unit to be implemented in all first-grade classes. This session received ratings of 2 on a five-point scale on its design and implementation; and a rating of 3 on content. Due to the lack of participant engagement and absence of collaborative or participant-centered interactions, the session was rated 2 in professional development culture. Overall, the session received a capsule rating of 2, indicating that there were elements of effective professional development, but that there were major areas in need of improvement.

Evaluators' overall ratings of the LSC professional development indicate that LSC projects are generally providing high-quality staff development. Each of the component areas of professional development--the design, the implementation, the disciplinary and pedagogical content, and the culture of the sessions--received very high ratings, as well. Mean ratings for each of those areas, as well as the likelihood that the project would enhance teachers' instructional capacity were consistently between 3.5 and 4.0 on the five-point scale. Projects that focused on developing the leadership capacity of teachers in the district were rated somewhat lower in that area, with a mean rating of approximately 3.3.

Each project evaluator was also asked to rate the project's professional development system holistically, considering all of the data available, and place their LSC project on a five-level continuum. All evaluators judged the LSC projects to be at Level 3 or above on a five-level continuum, indicating that the projects are either transitioning toward, or have already achieved, a level of professional development reflective of current standards for best practice.

Evaluators found that the professional development sessions generally incorporated state-of-the-art strategies, including activities that were relevant to the teachers' work in classrooms, modeling of effective pedagogical strategies, opportunities for teachers to reflect and talk with peers, strong components of follow-up and support, and active involvement of teachers as key players in their own professional development. The sessions were generally led by competent and skilled facilitators and characterized by very positive cultures that encouraged the engagement of teachers as professionals in learning communities.

Areas of strength for some projects were also described as areas in need of improvement for others. For example, some evaluators commented on the need for participants to have more time for reflection, increased attention to issues of classroom application. Others pointed out a need for a better balance in attention to content and pedagogical strategies, or greater emphasis on the content addressed by particular kits or modules. Finally, evaluators noted a need for more consistency in the skills and expertise of facilitators.

Teacher Preparedness in Science and Mathematics Education

The core evaluation design calls for a random sample of teachers in each project to complete questionnaires about their attitudes and instructional practices. In Cohort 2, these teachers are providing baseline data about the status of science and mathematics education. In Cohort 1, the sample includes teachers who have begun to participate in the LSC professional development, but in most projects only a very small proportion of the teachers had received in-depth assistance at the time of questionnaire administration.

Data collected from teachers in the LSC districts suggest that it will take a long and concerted effort to prepare these teachers to teach science and mathematics in a standards-based fashion. While the vast majority of teachers in LSC districts described attitudes in keeping with the standards (e.g., noting that hands-on activities, inquiry-based instruction, providing concrete experiences, and connections with other disciplines are very important for effective science and mathematics instructions), relatively few teachers feel very well prepared to implement these instructional strategies. For example, more than 80 percent of teachers in the LSC districts indicated that it was very important to have students participate in hands-on science activities, but only about 30 percent felt very well prepared in this area. (See Figure 4.)

Figure 4

Teachers are even more likely to recognize inadequacies in their content preparation. While all but a handful of elementary science teachers feel at least somewhat prepared to teach such topics as the human body and ecology, 20 percent or more indicated they were not adequately prepared to teach sound, astronomy, forces and motion, mixtures and solutions, machines, processes of change over time, and electricity, all areas that are integral to the elementary science curriculum.

Elementary mathematics teachers in LSC districts were likely to express confidence in their ability to teach traditional mathematics topics such as computation, numeration and number theory, measurement, and estimation. However, substantial proportions of teachers noted that they were only "somewhat prepared" to teach some of the topics advocated for inclusion in standards-based instruction in the elementary grades, including geometry and spatial sense, data analysis and probability, and algebraic concepts.

	Percent of Teachers
	Not Adequately Prepared	Somewhat Prepared	Fairly Well Prepared	Very Well Prepared
Computation	0	2	26	72
Patterns and relationships	1	8	36	55
Numeration and number theory	1	9	35	55
Estimation	1	9	38	53
Measurement	1	9	38	53
Geometry and spatial sense	2	16	41	40
Data analysis and probability	7	27	40	27
Technology in support of mathematics	11	32	37	20
Pre-algebra/algebra	17	25	32	26