Communication Center  Conference  Projects Share  Reports from the Field Resources  Library  LSC Project Websites  NSF Program Notes
 How to Use this site    Contact us  LSC-Net: Local Systemic Change Network
Educational Reform & Policy

Professional Development

Teaching and Learning

LSC Papers and Reports

Cross Site Reports

LSC Case Study Reports

Papers / Presentations authored by LSC Members

About LSC Initiatives and their impact

Bibliographies

Paper

  New!     

Highlights of the Year Three LSC Cross-Site Report

author: Iris R. Weiss, Diana L. Montgomery, Carolyn J. Ridgway, Sally L. Bond
published in: Horizon Research
published: 02/04/1999
posted to site: 02/04/1999

December 1998
Horizon Research, Inc.
111 Cloister Court - Suite 220
Chapel Hill, NC 27514- 2296


The Local Systemic Change through Teacher Enhancement: YearThree Cross- Site Report was prepared with support from the National Science Foundation under contract number RED- 9255369. These writings do not necessarily reflect the views of the National Science Foundation.


Table of Contents

Acknowledgments

Introduction

Participation in LSC Activities

Quality of LSC Professional Development Programs

Current Status of Teachers' Beliefs and Practices

Impact of LSC Activities

Supportiveness of Context

Sustainability of Reform

Summary and Recommendations


Acknowledgments

The core evaluation of the Local Systemic Change Initiative requires the energy, efforts, and insights of a very large number of people.

Conrad Katzenmeyer in the National Science Foundation's (NSF) Division of Research, Evaluation, and Communication; Joyce Evans, Susan Snyder, and Diane Spresser in NSF's Division of Elementary, Secondary, and Informal Education (ESIE); and Joy Frechtling of Westat, Inc. were instrumental in the design and implementation of the core evaluation system. Numerous other NSF/ ESIE program officers provided valuable assistance throughout the process, as well.

Within HRI, in addition to the authors of this report, Eric Banilower, Alison Bowes, Gail Gellatly, Elizabeth Hammond, Scott Hanrath, Susan Hudson, Ben Kurgat, Joan Pasley, Sheila Richmond, Sean Smith, Eugene Soar, and Claudia Templeton were responsible for various tasks essential to the production of this report, from instrument design, to data processing and analysis, to report production. David Flora and Abigail Panter of the L. L. Thurstone Psychometric Lab at the University of North Carolina at Chapel Hill assisted in the data analysis.

Most notably, this report would not have been possible without the efforts of the LSC project evaluators, whose work formed the basis for the analyses presented here. And special thanks are due to the thousands of teachers throughout the 263 participating districts who took time from their busy schedules to provide information about their mathematics and science teaching.


Introduction

In the spring and summer of 1995, the National Science Foundation (NSF) funded the first cohort of eight projects in a new initiative, the Local Systemic Change through Teacher Enhancement (LSC) program. Eighteen additional projects were funded in 1996 and 20 more in 1997, for a total of 46 projects in Cohorts 1, 2, and 3.

The goal of the LSC program is to improve the teaching of science, mathematics, and technology by focusing on the professional development of teachers within whole schools or school districts. Each targeted teacher in a K- 8 project is to participate in a minimum of 100 hours of professional development; for projects targeting teaching in grades 6- 12, the minimum is 130 hours over the course of the project. In addition to its focus on involving all teachers in a jurisdiction, the LSC initiative is distinguished from previous teacher enhancement efforts by its emphasis on preparing teachers to implement designated exemplary mathematics and science instructional materials in their classrooms.

LSC projects are expected to align policy and practice within the targeted district( s) and to include:

  • A shared comprehensive vision of science, mathematics, and technology education;

  • Active partnerships and commitments among stakeholders;

  • A detailed self- study that provides a realistic assessment of the current system's strengths and needs;

  • Strategic planning that incorporates mechanisms for engaging each teacher in intensive professional development activities over the course of the project; and

  • A set of clearly defined, measurable outcomes for teaching, and an evaluation plan that provides ongoing feedback to the project.

The LSC solicitation indicated NSF's plan to "provide a framework for data collection (including a set of instruments and procedures) that will allow the Foundation to evaluate individual projects, aggregate data and information across projects, and produce a cross- project analysis" (NSF 94-73). NSF contracted with Horizon Research, Inc. (HRI) of Chapel Hill, NC to design the data collection framework, provide technical assistance in its implementation, and prepare a cross- site analysis of the evaluation results.

An Overview of Cohorts 1, 2, and 3

Project data sheets completed by the principal investigators (PIs) provide some basic information about the 46 LSC projects included in Cohorts 1, 2, and 3.

  • In 1996- 97, the LSC initiative included 25 K- 8 science projects, 5 K- 8 mathematics projects, 9 secondary mathematics projects, 4 projects that targeted both elementary mathematics and science, and 3 projects that targeted both elementary and secondary mathematics.

  • Nineteen of the 46 LSC projects are single- district projects; at the other end of the scale, 3 projects involve more than 20 districts.

  • Thirty- four of the 46 LSC projects are five- year projects, 8 are four- year, and 4 are three year.

  • The 46 LSC projects plan to involve a total of approximately 40,000 teachers in more than 2,000 schools in 263 districts across the United States.

  • By the completion of these projects, an estimated 1,356,000 students will receive instruction from LSC- treated teachers each year.

Description of Core Evaluation Data Collection and Analysis

HRI has worked with the National Science Foundation and PIs and evaluators of the LSC projects on the design and implementation of a core evaluation system to allow aggregating information across projects in response to six core evaluation questions.

LSC Core Evaluation Questions

1. What is the overall quality of the LSC professional development activities?

2. What is the extent of school and teacher involvement in LSC activities?

3. What is the impact of the LSC professional development on teacher preparedness, attitudes, and beliefs about mathematics and science teaching and learning?

4. What is the impact of the LSC professional development on classroom practices in mathematics and science?

5. To what extent are school and district contexts becoming more supportive of the LSC vision for exemplary mathematics and science education?

6. What is the extent of institutionalization of high- quality professional development systems in the LSC districts?

Data collection activities for the projects' 1996- 97 Core Evaluation Reports were conducted from September 1, 1996 through August 31, 1997. Cohort 3 projects were collecting baseline data for their first year of funding; this was the second year of data collection for Cohort 2 projects and the third year for Cohort 1 projects. Data collection activities included the following:

1. Observations of professional development activities

The core evaluation calls for projects to conduct 5- 8 observations of professional development sessions each year and record their observations on standardized protocols. Evaluators were to consult with PIs on what professional development experiences were planned throughout the data collection year, and select a sample that was representative of the diversity of the project's activities. Program- wide, a total of 276 observations of professional development sessions were conducted.

2. Classroom observations

HRI provided the lead evaluator of each project with a list of 10 randomly selected teachers for each targeted subject. These teachers were to be observed in the spring of 1997. There was a total of 517 classrooms observed, including 299 classes taught by teachers who had participated in at least 20 hours of LSC professional development, and 218 classes as baseline for Cohort 3 projects.

3. Teacher questionnaires

Each project was asked to administer teacher questionnaires developed for the core evaluation to a sample of 300 teachers per targeted subject; the median response rate was 84 percent. A total of 10,054 teacher questionnaires were returned to HRI, including 6,126 from K- 8 science teachers; 2,347 from K- 8 mathematics teachers; and 1,581 from 7- 12 mathematics teachers.

4. Principal questionnaires

Projects were also asked to administer questionnaires to the entire population of principals of targeted schools. Return rates on the principal questionnaire were generally higher than for the teacher questionnaire, with a median response rate of 92 percent; a total of 1,905 principal questionnaires were returned.

5. Teacher interviews

Evaluators of each Cohort 1 and Cohort 2 project were asked to interview a sample of 10 teachers who had participated in at least 20 hours of professional development activities in that project. A total of 249 interviews were conducted among the 26 projects. About two- thirds of the interviews were conducted by phone, and the remaining one- third in person.

Project evaluators were asked to report their findings using guidelines developed for the core evaluation system, including responding to the six core evaluation questions. Evaluators were also asked to provide overall ratings of the quality of professional development activities, the supportiveness of the context, and the sustainability of high- quality professional development systems.

Participation in LSC Activities

LSC projects are responsible for serving all of the teachers who teach the targeted subject and grade range, rather than only those who volunteer to participate. Data on participation in LSC professional development activities come from teacher and principal questionnaires.

Evaluators report that most LSC projects are on schedule for providing in- depth professional development to all teachers in the targeted subjects and grades. To date, the population of elementary teachers participating in LSC professional development activities closely mirrors the overall targeted population in these districts in terms of teacher gender, race/ ethnicity, and course background preparation. In contrast, secondary mathematics projects appear to be starting with relatively less well- prepared teachers in the middle rather than the high school grades.

Based on their responses to core evaluation questionnaires, a substantial proportion of principals is neither involved in nor knowledgeable of LSC activities. However, a number of evaluators pointed out that principals they know to have been involved indicated otherwise on the core evaluation questionnaires, perhaps not associating those activities with the term "LSC program."

A more serious concern is the fact that a few projects appear to have redefined targeted teacher population to mean those who are willing to participate, which is clearly inconsistent with the intent of the LSC initiative.

Quality of LSC Professional Development Programs

Project evaluators were asked to observe 5- 8 professional development activities for the core evaluation. Evaluators and PIs were to decide, jointly, which activities would be observed, selecting sessions to represent the diversity of the project's professional development program and to reflect the extensiveness and importance of the various activities offered.

Description of Observed Sessions

Evaluators documented a number of descriptive features of each professional development session, providing information across all projects about targeted participants, presenters/ facilitators, purposes and content focus, and the major types of activities that characterized the sessions.

The typical professional development session observed for the LSC core evaluation had between 21 and 50 participants. Some sessions exclusively targeted lead teachers (22 percent); other sessions targeted non- lead teachers (47 percent); and some sessions targeted both lead and nonlead teachers (27 percent). A total of 12 percent of the sessions included principals or other administrators.

LSC professional development involves presenters/ facilitators from a variety of settings. Lead teachers served in this capacity in 40 percent of the observed sessions, while only 27 percent of the sessions included university faculty as presenters or facilitators. Two- thirds of the presenters/ facilitators were female, 87 percent were white, 8 percent African- American, 3 percent Hispanic, and 2 percent Asian or Pacific Islander.

Evaluators were asked to indicate the major intended purposes of each observed session based on information provided by the session facilitators. As can be seen in Table 1, the most frequently cited purposes were working on classroom pedagogy issues and enhancing teachers' understanding of science/ mathematics concepts.

Table 1
Major Intended Purposes of LSC Professional Development Sessions

Percent of Sessions
All Sessions K- 8 Science K- 8 Mathematics 7- 12 Mathematics
Working on classroom pedagogy issues* 75 77 69 75
Learning about specific instructional materials 38 37 31 51
Learning pedagogical/ classroom management strategies 36 37 39 30
Creating a vision of learning through investigation 35 39 27 30
Designing/ scoring student assessments 9 11 6 4
Considering issues of access, equity, and diversity 6 7 4 4
Increasing teacher mathematics/ science content knowledge 40 37 39 49
Promoting reflective practice 17 18 10 19
Developing teacher leaders 15 14 15 15
Building professional networks among educators 14 14 12 15
Orientation to the project 14 13 23 11
Developing capacity of participants to use technology 5 5 2 9

* Percents add to more than total for category because sessions could include more than one purpose.

When sessions focused on one or more disciplinary content areas, evaluators were asked to categorize that content. In K- 8 science projects, evaluators reported that nearly half of the sessions that had a disciplinary content focus dealt with physical science concepts (46 percent), nearly that many with life science content (41 percent), and somewhat fewer with concepts from the earth and environmental sciences (32 percent).

In projects targeting K- 8 mathematics, the most heavily emphasized topics were patterns and relationships (32 percent of the sessions that dealt with disciplinary content), numeration and number theory (29 percent), and data collection and analysis (23 percent).

The most commonly emphasized topics in sessions for 7- 12 mathematics teachers were patterns and relationships, algebra, and geometry/ spatial sense, each the focus of about 30 percent of the sessions that dealt with disciplinary content.

Most of the observed LSC professional development sessions included several different instructional strategies. Most sessions included discussions or seminars (83 percent). Roughly two- thirds of sessions included formal presentations and a similar proportion included investigative/ problem solving activities.

Observer Ratings of LSC Professional Development Sessions

In order to assess the quality of professional development sessions, observers were asked to rate a number of components for each session they observed.

1. Design

For design of the professional development sessions, the indicators that most often received high ratings were: the extent to which the session design reflected careful planning and organization; how well the session encouraged a collaborative approach to learning and incorporated tasks and interactions consistent with a spirit of investigation; and the appropriateness of strategies used in the session for accomplishing the purposes of the LSC professional development.

2. Implementation

In the area of implementation of professional development activities, indicators most frequently rated high were: whether the facilitator's background and expertise or management style enhanced the quality of the session and the extent to which the session incorporated instructional strategies appropriate for its purposes and the needs of adult learners.

3. Content

Observers were asked to rate either the quality of the disciplinary content of the observed session, its pedagogical content, or both, depending on the focus of the session. Disciplinary content was rated in 176 of the 276 sessions; these sessions were most likely to receive high ratings for: the appropriateness of the disciplinary content for the purposes of the session and the background of the participants; the extent to which the facilitators displayed an understanding of mathematics/ science content; and the soundness and appropriateness of mathematics/ science content.

Observers rated 198 of the 276 observed professional development sessions on the quality of their pedagogical content. As was the case for quality of disciplinary content, these sessions tended to be highly rated for: the appropriateness of the pedagogical content for the purposes of the session and the background of the participants and the extent to which the facilitator displayed an understanding of pedagogical concepts.

4. Culture

The literature on effective staff development emphasizes the importance of establishing a professional development culture where teachers can explore content and pedagogy in a collegial, risk- free environment, and most observed LSC professional development sessions were successful in this regard. Indicators that were most likely to receive high ratings included: whether active participation of all was encouraged and valued; the extent to which there was a climate of respect for participants' experiences, ideas, and contributions; the extent to which interactions reflected collaborative working relationships among participants and between facilitators and participants; and whether participants were intellectually engaged with important ideas relevant to the focus of the session .

5. Leadership

Most of the LSC projects have chosen to develop teacher leaders as part of their strategy for reaching large numbers of teachers, and 43 of the 276 observed sessions focused specifically on leadership content, such as planning and implementing effective professional development sessions. These sessions were most likely to receive high ratings for: the extent to which the facilitators displayed an understanding of leadership concepts; whether the leadership content was sound and appropriately presented; and the extent to which the leadership content was appropriate for the purposes of the session and the backgrounds of the participants.

Overall ratings for individual professional development sessions were quite favorable. Only 1 percent of observed LSC sessions were rated as ineffective professional development (Level 1), and 7 percent were rated at Level 2, having quite limited likelihood of helping participants implement exemplary mathematics/ science instruction or be leaders in reform. Overall, 71 percent of the observed professional development sessions received ratings of 4 or 5, indicating that those sessions were skillfully facilitated, engaging participants in purposeful work that would likely lead to enhanced capacity to implement exemplary instruction. (See Figure 1.)

Figure 1

Teacher Ratings of LSC Professional Development Programs

While observations were the main source of information on the quality of LSC professional development, information was also gathered from the participating teachers via questionnaires and interviews. On teacher questionnaires, LSC participants described the extent to which a variety of features characterized the LSC professional development, rating each item on a scale ranging from 1, "not at all" to 5, "to a great extent." Table 2 shows the percent of teachers rating the LSC professional development high (4 or 5 on the five- point scale), medium (a rating of 3), and low (a rating of 1 or 2) on each of several items. Note that the LSC received highest marks for providing both opportunities for professional development and support for teachers as they implement what they have learned; still only about half of the participating teachers gave the LSC ratings of 4 or 5 on these items.

Table 2 Teacher Ratings of LSC Professional Development*

Percent of Teachers
High Medium Low
Adequate opportunities are available to me for mathematics/ science- related professional development. 54 31 15
I receive support as I try to implement what I've learned. 46 31 23
I am encouraged to develop an individual professional development plan to address my needs and interests related to mathematics/ science education. 35 29 36
I am involved in planning my mathematics/ science- related professional development. 33 26 41
I am given time to work with other teachers as part of my professional development. 26 27 47
I am given time to reflect on what I've learned and how to apply it to the classroom. 25 30 45

* Only teachers who had participated in LSC professional development were included in these analyses.

It is interesting to note that the more time teachers had spent in LSC professional development, the more highly they rated it. (See Figure 2.)

Figure 2

Evaluators' Overall Ratings of the LSC Professional Development Programs

Using the observation results, as well as teacher questionnaire and interview data, evaluators then rated the overall quality of the professional development program, noting particular strengths and areas in need of additional attention. Evaluators indicated the following as key strengths in the programs:

  • LSC projects are frequently framed around national standards, creating a common vision among participants.

  • LSC professional development sessions are typically reflective of best practice, tightly linked to exemplary instructional materials, modeling appropriate instructional strategies, and making explicit connections to classroom practice.

  • Most facilitators demonstrate knowledge and skill in implementing the LSC professional development.

  • LSC projects typically provide a collegial learning environment and support as teachers implement what they have learned.

  • Many LSC projects are enhanced by systemic features that attend to the larger context.

Evaluators also mentioned areas that needed further attention:

  • Some projects provided insufficient time for teachers to reflect, discuss, and share ideas with one another.

  • Some projects needed to keep more of a focus on key mathematics and science concepts.

  • Some projects needed to enhance the "systemic- ness" of project design and implementation.

It is interesting to note that while observers typically gave high ratings to the quality and likely impact of individual leadership development sessions, they were less impressed with the overall quality of the project's handling of leadership content. Only 40 percent of the projects were rated highly (4 or 5 on a five- point scale) with regard to leadership content, compared to more than 70 percent of projects receiving high ratings in each of the other component areas.

Evaluators identified a number of key components that were important for effective professional development for teacher leaders. They included:

  • Providing a supportive culture;
  • Clearly communicating expectations of teacher leaders;
  • Balancing attention to disciplinary, pedagogical, and leadership content;
  • Giving teachers opportunities to practice leadership roles;
  • Providing teacher leaders with ongoing support;
  • Broadening teacher leaders' professional experience; and
  • Fostering administrative support for teacher leaders.

As a culminating task in their analysis, evaluators were asked to place each project on a five- level continuum describing the quality of the professional development program. As can be seen in Table 3, none of the 46 projects was rated at Level 1 (Predominance of Ineffective Professional Development), and only 1 project was considered to be at the Level 2, "Exploring" stage. Nine projects were rated at Level 3 (Transitioning to Quality Professional Development). Twenty- six projects were rated at Level 4 (Emerging Infrastructure of Well- Designed Professional Development), indicating that their professional development plan and activities incorporated many features reflective of current standards- based approaches, that the professional development activities were well- implemented, and that in the evaluator's judgment they will likely enhance participants' capacity to provide high quality mathematics/ science instruction to their students. The final 10 projects (22 percent) were rated at Level 5, indicating consistently excellent quality.

Table 3
Continuum Ratings for Quality of LSC Professional Development

Percent of Projects*
All Projects K- 8 Science K- 8 Mathematics 7- 12 Mathematics
Level 1:
Predominance of Ineffective Professional Development
0 0 0 0
Level 2:
Exploring Quality Professional Development
2 3 0 0
Level 3:
Transitioning to Quality Professional Development
20 28 8 0
Level 4:
Emerging Infrastructure of Well- Designed Professional Development
57 48 67 75
Level 5:
Predominance of Well- Designed Professional Development
22 21 25 25
Mean Continuum Rating Level 4.0 3.9 4.2 4.3

* Projects that address two subject areas are included in each subject, but counted only once in the total of all projects.

 next page