page 10 of 13
Chapter Seven
Impact of Local Systemic Change Activities
Two of the 6 core evaluation questions address the impact of the LSC on teachers and teaching.
One focuses on the impact on teacher preparation, attitudes, and beliefs; and the other on
classroom practices. This chapter looks at impact from several different points of view: those of
the teachers, their principals, the classroom observers, and the lead evaluators who synthesized
and interpreted the data for each project.
Impact From the Teachers' Point of View: Questionnaire Composites
Teachers' perceptions of impact come from two sources-- teacher questionnaires and
interviews-- the most comprehensive of which is the teacher questionnaire. As noted earlier,
many of the teacher questionnaire items have been combined to form composites.1
These composites group together responses to related questions. For example, the "Attitudes Toward
Teaching" composite combines questionnaire data on how teachers feel about reform- minded
instructional strategies. This grouping of responses into composites allows the data to be
analyzed thematically, rather than item-by-item.
Most of the LSC projects have identified a subset of the targeted teacher population as lead
teachers. These teachers are likely to be in the group who report most intensive involvement in
the LSC, but it is likely that many of them were already fairly aligned with the LSC vision prior
to their participation in the project. To avoid biasing the results, teachers who indicated they
have been designated as lead teachers in the LSC have been omitted from the comparisons of
teacher attitudes and classroom practices by level of participation in the LSC.
This section shows the results by level of participation for each of the composites relevant to
classroom impact. As noted in Chapter Two, K- 8 teachers who have and have not participated
in the LSC professional development are similar both demographically and in terms of course
background. Accordingly, comparisons of the results between these two groups are used as an
indication of the effect of the LSC projects on teachers and teaching.2
Composite: Attitudes Toward Teaching
The "Attitudes Toward Teaching" composite describes participants' beliefs about a number of
reform-oriented strategies. It includes items such as the perceived importance of:
- Providing concrete experience before abstract concepts;
- Developing students' conceptual understanding of mathematics/science;
- Having students participate in appropriate hands- on activities; and
- Engaging students in inquiry- oriented activities.
As can be seen in Figure 28, teacher beliefs are quite well- aligned with the tenets of mathematics
and science reform, and they remain fairly constant regardless of hours of LSC professional
development. While the results show a small positive effect for K- 8 science and a small
negative effect for K- 8 mathematics, LSC involvement appears to have relatively little impact on
teachers' already positive attitudes toward the reform strategies.
Figure 28
Composite: Pedagogical Preparedness
While the previous composite focused on the perceived importance of various reform strategies,
this composite reflects how prepared teachers feel to implement reform strategies. These include
instructional strategies such as cooperative learning and use of manipulatives, as well as
recognizing and responding to student diversity.
Comparisons by treatment level, shown in Figure 29, indicate that teachers who participated in
LSC professional development report increasingly high levels of pedagogical preparedness. The
3 percentage points difference between untreated and highly treated K- 8 mathematics teachers
constitutes a small effect (. 23 standard deviations), while the 8 percent difference for K- 8
science (. 60 standard deviations) is a medium effect.
Figure 29
Composite: Content Preparedness
The mathematics and science versions of this composite reflect how well prepared teachers feel
to teach a number of topics within their discipline. As can be seen in Figures 30 and 31, teachers
who have participated in LSC professional development are more likely to report feeling well
prepared in the various content areas; however, due to fairly large standard deviations for these
variables, the effect sizes are small.
Figure 30
Figure 31
Composite: Investigative Culture
The "Investigative Culture" composite examines strategies used by teachers to facilitate
exploration and investigation on the part of students, including such practices as:
- Arranging seating to facilitate student discussion;
- Using open- ended questions;
- Requiring students to supply evidence to support their claims; and
- Encouraging students to consider alternative explanations.
Again, K- 8 science teachers show a steady increase in this area with level of treatment. (See
Figure 32.) The nine- point difference between the untreated and highly treated K- 8 science
teachers constitutes a medium effect, while the one point difference for K- 8 mathematics
represents a very small effect size.
Figure 32
Composite: Investigative Practices
The "Investigative Practices" composite looks more closely at what students actually do in the
classroom. It includes such instructional strategies as having students:
- Engage in hands- on mathematics/ science activities;
- Work on models or simulations;
- Work on extended investigations; and
- Write reflections in a notebook or journal.
As can be seen in Figure 33, the means for this composite tend to be lower than in other areas,
likely reflecting the fact that while the classroom may always be set up to facilitate group work,
even the most highly investigative classes do not do field work or design their own investigations
on a daily basis. Participation in the LSC appears to have a fairly substantial impact on
classroom practice in K- 8 science (. 69 standard deviations), and a smaller impact in K- 8
mathematics (. 32 standard deviations).
Figure 33
Composite: Investigative Practice, Use of Calculators and Computers
The factor analysis yielded a composite on the frequency of use of calculators and computers in
mathematics instruction:
- For learning or practicing skills;
- To develop conceptual understanding; and
- As a tool, e. g., spreadsheets, data analysis.
Two results in Figure 34 are noteworthy. First, comparing only mathematics teachers who have
not yet participated in the LSC, calculator and computer usage is much higher in grades 7- 12 (69
percent of total possible points) than in grades K- 8 (49 percent). Second, K- 8 mathematics
teachers who have participated in at least 40 hours of LSC professional development are more
likely to use calculators and computers in their instruction than are their untreated peers. (The 10
percentage point difference between the two groups is .47 standard deviations, categorized as a
medium effect.)
Figure 34
Table 20 summarizes the effect sizes for the classroom impact composites, using 0 for no effect,
a single "+" to indicate a small positive effect, and so on. Note that the LSC professional
development appears to have had a marked impact on K- 8 science teachers, with small effects
on 2 of the composites, and medium effects on the other 3, including both establishing an
investigative culture and implementing investigative practices in the classroom. The impact on
K- 8 mathematics teachers appears less substantial, with small effects on four composites, a
medium effect only on use of calculators and computers, and a small negative effect on teacher
attitudes towards reform.
Table 20
Summary of the Impact of LSC
Professional Development on Teachers and Teaching
|
Effect Size |
|
K-8 Science |
K-8 Mathematics |
Composite |
Number of Standard Deviations |
Effect Size |
Number of Standard Deviations |
Effect Size |
Attitude Toward Teaching |
.24 |
+ |
.25 |
- |
Pedagogical Preparedness |
.60 |
++ |
.23 |
+ |
Content Preparedness |
.35 |
+ |
.33 |
+ |
|
|
|
|
|
Investigative Culture |
.66 |
++ |
.11 |
+ |
Investigative Practices |
.69 |
++ |
.32 |
+ |
Use of Calculators and Computers |
-- |
-- |
.47 |
++ |
Teacher Interviews
As described earlier, the LSC core evaluation uses interviews to collect more in- depth
information than provided in the teacher questionnaires, but from a smaller number of people.
Approximately 10 teachers from each Cohort 1 and Cohort 2 project were interviewed about
their beliefs, level of confidence, and the way they approached teaching science or mathematics
prior to the LSC, as well as the impact of the LSC in these areas.
When asked about the impact of the LSC, most of the teachers were able to provide specific
examples of ways in which the program had affected their teaching. Interviewers interpreted
these comments and rated the impact of the LSC on a five- point scale from "1, little or no
impact" to "5, extensive impact." Table 21 shows the percentage of teachers with varying levels
of involvement in the LSC describing extensive positive impact (rated level 4 or 5 by the
interviewer).
Table 21
Teachers Reporting Extensive Positive Impacts of the LSC in
Each Area by Level of Involvement in LSC Professional Development
|
Percent of Teachers |
|
All Teachers |
Less Than 10 Days |
10 Days or More |
Instruction |
65 |
63 |
69 |
Curriculum |
63 |
60 |
68 |
Confidence |
58 |
55 |
63 |
|
|
|
|
Beliefs |
53 |
52 |
55 |
Assessment |
48 |
43 |
52 |
Leadership |
38 |
20 |
56 |
Two facts about this table are noteworthy. First, the majority of teachers interviewed indicated
that the LSC had had considerable impact, particularly in the areas of curriculum and instruction.
Second, the amount of impact generally increases with increasing time spent involved in LSC
activities. It is also interesting to note that impact on leadership ability seems to depend on
having a high level of involvement in the LSC.
Teachers' descriptions of their former and current practices highlight the impact LSC projects
are having on many teachers.
I just taught multiplication to my second graders and rather than teach them the
multiplication formulas, I let them explore multiplication first.
* * *
These activities have provided me ongoing science instruction and really
heightened my interest in science. I had no clear direction before. I feel more
knowledgeable and secure. I know what to teach and there is a plan.
* * *
I have learned that I should not automatically give answers. I now do more
questioning. I used to teach for the right answer!
However, not all respondents considered the LSC impact to be a positive one. The following
comments illustrate some of the difficulties a number of teachers are experiencing.
I teach it because it is the mandated curriculum. The kids in my class are not
getting it. Problem solving is new and hard for them.... I'd like to see more of the
basics.
* * *
The workshops give teachers strategies for implementing inquiry science in the
classrooms, but it's in the classroom during actual implementation where the
weaknesses of the training are seen and need to be addressed. I realize that a lot
of strategies and techniques discussed in the workshops don't work in the
classroom.
Principals' Impressions of Impact
All principals in schools targeted by LSC projects were asked to complete questionnaires.
Among other items, they were asked to rate: (1) the level of their school's current involvement
in the LSC, (2) the extent to which their teachers were using the LSC- specified instructional
materials, and (3) the extent to which teachers were using the LSC- advocated instructional
strategies with their students. When either the materials or strategies variable is plotted against
level of involvement, the results are seen to be highly correlated: the higher the level of the
school's involvement in the LSC, the more likely the teachers will be using exemplary materials
and teaching strategies.3
Principals were also asked to describe their school's progress in moving toward excellence in
mathematics and science education. Slightly more than half of the principals involved in K- 8
mathematics and science projects in Cohort 1 and Cohort 2 felt that their schools were at least
"well along in improving" their science education.
Impact as Measured by Classroom Observers
By looking at the synthesis and capsule ratings of the three different cohorts, it is possible to get
an indication of impact of the LSC programs. While Cohort 1 and Cohort 2 contained teachers
that have received at least 20 hours of LSC professional development, Cohort 3 teachers have not
yet received 20 hours of professional development. As can be seen in Figure 35, lessons of
teachers in all cohorts were rated similarly for the quality of their implementation. Cohort 1 and
Cohort 2 teachers were rated higher in all other aspects than were Cohort 3 teachers.
Figure 35
Lead Evaluators' Analyses of Impact
Each project's lead evaluator submits a core Evaluation Report to Horizon Research, Inc. every
year. These reports make clear that different projects are dealing with distinct populations of
teachers and are beginning at different points. To quote from one report:
[In many ways,] teachers are just like students, all over the place in their abilities
and attitudes. What, of course, reform efforts of all kinds have substantiated in
recent years is that teachers are not "like" students-- they are students, adult
students and what is known about learning applies to them as well as to young
people. No longer would anyone expect all children of a certain age to learn to
read or to add fractions in exactly the same way at the same time, in lock- step
fashion.... Both teachers' attitudes and beliefs, and their behaviors, are not going
to be uniform at this or any point in the project.
Although teachers enter the LSC projects with a wide range of abilities and knowledge,
evaluators frequently commented on participants' growing self confidence as they engage in
LSC activities. Most Cohort 1 and Cohort 2 evaluators also reported that there had been impact
on teachers' classroom practices, pointing out that they noticed frequent usage of the designated
materials in both mathematics and science. However, they often observed a rather mechanical
level of implementation of activities. An evaluator of a science project described the situation as
follows:
[Teachers] are using the kits but are not yet at the point of selecting and tailoring
appropriate kit activities to meet their goals or being able to extend beyond the
preplanned suggestions that come with the kits. In a sense some teachers are
using the kits with a "recipe" approach.... One positive effect of the program is
that teachers recognize the difference between hands- on activity in the science
classroom and authentic, rigorous investigation. They realize that they are
becoming more capable in the hands- on part of science exploration, but they also
recognize that they are less fluent with the minds- on part of exploration.
Another project evaluator described teachers as being at the "getting- it- going" level of classroom
implementation, and not yet to the next level of "getting- it- good." Similarly, a project evaluator
commented:
It appeared to us that many teachers were working through the [mathematics]
lessons carefully and thoroughly, step by step.
These comments emphasize that many teachers are conscientiously trying the new materials and
strategies but have yet to grasp the larger conceptual picture. This situation points to the need for
continued mentoring and support to encourage teachers to see beyond the mechanics of
implementation.
Specific areas of classroom practice mentioned by evaluators as being in need of attention
include lesson closure or "sense- making," reflection by students, portraying science as a dynamic
discipline, and making connections to other disciplines. One evaluator commented that although
science was portrayed as dynamic and investigative in the professional development sessions,
this conception was not passed on to students. This "dilution effect" was mentioned by several
other evaluators as well, including comments that the professional development experience
affected the lead teachers' classroom practices much more than that of the targeted teachers.
A concern noted by several evaluators as limiting the impact of the LSC was testing policies at
both state and district levels, particularly in mathematics. Evaluators reported that many teachers
felt the need to revert to more traditional teaching practices to prepare their students to do well
on these standardized tests. This tension pulls at teachers: they are held accountable, in many
cases, for how well their students perform in relation to others on these norm- referenced exams.
Yet covering the curriculum to be tested may not support the more in- depth investigations
advocated by the LSC.
Comparison of Impact Data
Data from a wide variety of sources-- at the individual teacher level, at the school level, and at
the project level-- all suggest that the LSC projects are having a positive impact on both teachers'
feelings of competence to teach mathematics/ science and their ability to actually do so at the
classroom level. LSC- supported curricula and methods are being implemented across a wide
spectrum of classrooms, schools, and districts. Principals, teachers, and evaluators agree on this.
Differences of opinion occur at a finer level and center around the nature of the impact.
As noted earlier, the majority of principals view their schools as being well along in reform.
This contrasts with the view of the evaluators who generally see the teachers as just beginning to
change. How can these differences in opinion be reconciled assuming that the principals and the
evaluators see the same types of activities occurring in the same types of classrooms.
The difference may originate in the level of subject matter expertise the two groups bring to the
observation. Evaluators are generally knowledgeable about mathematics and science content.
Furthermore, they have spent considerable time thinking about and discussing with PIs and other
evaluators during LSC meetings just what exemplary mathematics or science instruction might
look like. They have examined videos for benchmarking purposes and considered indicators.
Because of this reflection, evaluators are apt to see exemplary instruction as more that just
implementation of curriculum. As one evaluator noted, "getting- it- going" is not the same as
"getting- it- good." Principals, on the other hand, are responsible for all academic subjects; their
knowledge of mathematics/ science instruction may be less deep. To them, adoption of a
standards- based curriculum and seeing these materials used on a regular basis in the classroom
with students may represent not just a first step, but being well along on the road to reform.
What implications does this have for the LSC projects? If principals are to support teachers in
their reform efforts beyond the level of following the directions that come with the modules, they
will need to become aware of what might occur beyond this level. Like classroom teachers, they
may need to develop images of exemplary instruction that go beyond theory. They need
opportunities to observe and discuss with thoughtful, knowledgeable persons a variety of strong
mathematics/ science lessons being taught in the context of classrooms similar to those in their
schools. As projects struggle with means to move teachers to the next level of expertise, they
will need to include the principals in developing a shared vision of exemplary instruction.
Summary
- LSC is having an overall positive effect on teachers' beliefs, feelings of preparedness, and
classroom practices, with these effects being greater among K- 8 science teachers than K- 8
mathematics teachers. The impact of LSC professional development on their beliefs is
relatively small, in part because most teachers report positive beliefs prior to the LSC.
- Teachers participating in LSC activities are more likely to be using LSC recommended
materials and strategies than their peers; many have yet to grasp the "big picture," and need
mentoring and support to see beyond the mechanics of implementation.
- The level of impact increases with increased hours of participation in LSC activities.
Leadership appears to require more hours of professional development for impact to occur
than other areas.
- Most principals perceive their schools as being well along the road to reform, whereas most
evaluators see change as being in the beginning stages. the support of principals might be
enhanced by encouraging positive views of reform, while improving their understanding of
exemplary mathematics and science instruction.
Footnotes
1 See pages 7 and 8 for a description of how composite scores were calculated.
2 Results by treatment level are presented separately for K- 8 science and K- 8 mathematics teachers. The effect size is calculated as the
difference between the "0 hours" and "40 or more hours" group means, divided by the standard deviation of the population. Following standard
conventions, effect sizes of .2 are considered small effects, .5 medium effects, and .8 large effects (Jacob Cohen, Statistical Power Analysis for
the Behavior Sciences, Hillsdale, NJ: Lawrence Erlbaum Associates, 1988). Since all of the 7- 12 mathematics projects included in this report
were funded as part of Cohort 3, the number of teachers participating in LSC professional development at the time of data collection was too
small for separate analysis by treatment level; consequently for 7- 12 mathematics, results are presented for untreated teachers only.
3 Spearman' rho correlation of .67 and .64, respectively.