Core Evaluation Question IV
What is the impact of the LSC professional development on classroom practice in science and mathematics?
A number of core evaluation data sources contributed information about classroom practices. The teacher questionnaire data provide self-report information from a sample of all LSC-targeted teachers about the frequency with which they use various instructional strategies in their classes. The classroom observations provide an external look at the quality of classroom instruction, including information related to equity, classroom culture, and student engagement that are difficult to measure with questionnaire items.
Teacher Descriptions of Classroom Practice
The design of the core evaluation calls for administration of questionnaires to a random sample of teachers who are targeted for LSC professional development regardless of whether they have actually participated in the LSC at that point. Therefore, in the early years of the LSC initiative, when only a small subset of teachers have participated, the questionnaire data are more useful in describing the general status of science and mathematics instruction in these districts than in evaluating the impact of the LSC.
One finding from these questionnaires is that in LSC districts, as is the case nationally, science is taught less frequently than is mathematics at the elementary level. Whether targeted for professional development in science or mathematics, these elementary teachers were most likely to report teaching reading/language arts each day of the last five (nearly 90 percent in each group), followed by mathematics (80 to 86 percent). Only about 1 in 4 teachers in each group reported teaching science or social studies on a daily basis. (See Figure 38.)
Teachers were also asked to respond to a series of items about instructional practice, indicating how often they used each of a number of instructional strategies in their science and mathematics classes. Factor analysis of these responses was used to create two composites of related items. The "student-centered" composite includes the frequency with which the teacher arranges seating to facilitate student discussion; encourages students to explain concepts to one another; allows students to work at their own pace; and involves students in such activities as formal presentations to the class, student-led discussions, and small group work. The "investigative-culture" composite includes the extent to which the teacher uses open-ended questions; requires students to explain their reasons or supply evidence to support their claims; and involves students in science and mathematics investigations.
Teachers Reporting Instruction in Each Subject on All 5 of the Last 5 daysFigure 38 Based on teacher reports, science and mathematics classes were similar in the extent to which they used a student-centered approach, with a mean of approximately 70 percent of total points possible on that composite in each group. As can be seen in Figure 39, there was some variation among projects, with means ranging from 65 to 77 percent in Cohort 1 science, from 62 to 74 percent in Cohort 2 science, and from 67 to 73 percent in Cohort 2 mathematics projects. The composite scores for "investigative culture" were also similar for the three groups, averaging just over 60 percent in each case. (See Figure 40.)
Figure 39 Figure 40 While the overall composite scores were similar, an examination of individual items reveals a number of differences between science and mathematics instruction. Table 12 shows the percent of science and mathematics teachers who reported that each of a number of instructional activities occurred at least once a week. Note that mathematics classes were more likely than science classes to include such activities as teacher presentations of content information, work on solving real-world problems, and homework review at least once a week.
Table 12
Classes Participating in Each Activity at Least Once a Week
|
Percent of Classes |
|
Cohort 1 Science |
Cohort 2 Science |
Cohort 2 Mathematics |
Teacher-led discussions | 81 | 82 | 88 |
Work in cooperative learning groups | 74 | 70 | 69 |
Teacher presentation of content information | 73 | 86 | 91 |
Hands-on activities | 64 | 53 | 74 |
Share ideas or solve problems in small groups | 54 | 51 | 59 |
Write reflections in a notebook/journal | 38 | 32 | 25 |
Review homework | 30 | 37 | 74 |
Work on solving real-world problems | 30 | 32 | 59 |
Work on extended investigations | 26 | 25 | 9 |
Work on portfolios | 16 | 14 | 13 |
Student presentations | 18 | 21 | 22 |
Table 13 shows the percent of classes that ever participated in particular instructional activities. While mathematics lessons were more likely to include reading from a textbook, science lessons were more likely to include reading other non-textbook materials. Both Cohort 1 and Cohort 2 science classes were more likely than Cohort 2 mathematics classes to include having the students conduct extended investigations, work on portfolios, and write reflections in a notebook or journal some time during the semester.
Table 13
Activities Ever Occurring in Science and Mathematics Classes
|
Percent of Classes |
|
Cohort 1 Science |
Cohort 2 Science |
Cohort 2 Mathematics |
Read from non-textbook materials in class | 93 | 93 | 60 |
Conduct extended investigations | 88 | 87 | 73 |
Participate in field work/field trips | 88 | 83 | 83 |
Write reflections in a notebook/journal | 86 | 82 | 64 |
Work on portfolios | 69 | 65 | 51 |
Read from a textbook in class | 57 | 63 | 69 |
Table 14 shows the percent of classes where teachers report using each of a number of instructional strategies to provide student understanding. Interestingly, science instruction was more likely than mathematics instruction to include the use of open-ended questions, while students in mathematics lessons were more likely to be allowed to work at their own pace and to be asked to explain their reasoning or supply evidence to support their claims.
Table 14
Instructional Strategies Used at Least Once a Week
|
Percent of Classes |
|
Cohort 1 Science |
Cohort 2 Science |
Cohort 2 Mathematics |
Use open-ended questions | 83 | 81 | 68 |
Arrange seating to facilitate discussion | 77 | 71 | 71 |
Allow students to work at their own pace | 72 | 66 | 81 |
Encourage students to explain concepts to one another | 64 | 62 | 72 |
Encourage students to consider alternative explanations | 65 | 66 | 72 |
Require students to explain their reasons/supply evidence to support their claims | 59 | 56 | 85 |
Read and comment on student journals | 33 | 32 | 31 |
Finally, the teacher questionnaire focused on the use of various strategies to assess student understanding of science and mathematics. As can be seen in Figure 41, the mean percent of total points possible on the use of assessment strategies composite was approximately 50 percent for each group. Project means varied quite a bit, ranging from 41 to 53 percent in Cohort 1 science, from 42 to 58 percent in Cohort 2 science, and from 42 to 58 percent in Cohort 2 mathematics.
The relatively low means in the assessment composite are to be expected. While a very high score on student centered strategies is desirable, a score of 100 percent would mean that a class wrote reflections in a notebook or journal, worked on portfolios, took tests requiring constructed responses or essays, and engaged in performance tasks for assessment purposes on a daily basis. Even if much of that assessment were embedded in instruction, such a class would likely be spending far too much time finding out what students know and far too little time moving their conceptual understanding forward. With that understanding, the assessment composite is useful for looking at trends over time.
Based on teacher reports, the majority of both science and mathematics classes engage in performance tasks for assessment purposes at least once a month. (See Table 15.) Note that mathematics classes are considerably more likely than science classes to take short-answer tests, whether publishers' tests or those developed by the teachers. In contrast, science classes are more likely to have students write their reflections on a fairly frequent basis.
Figure 41
Table 15
Assessment Activities in Science and Mathematics Classes
Using Each Type of Assessment at Least Once a Month
|
Percent of Classes |
|
Cohort 1 Science |
Cohort 2 Science |
Cohort 2 Mathematics |
Write reflections in a notebook/journal | 66 | 59 | 47 |
Engage in performance tasks for assessment purposes | 62 | 56 | 63 |
Work on portfolios | 40 | 41 | 36 |
Take teacher-developed short-answer tests | 32 | 45 | 58 |
Take tests requiring constructed responses or essays | 31 | 33 | 38 |
Take publishers' tests or end-of-chapter tests | 21 | 33 | 65 |
Evaluator Descriptions of Classroom Practice
Additional information on science and mathematics instruction comes from the classroom observations and accompanying interviews, including information about the composition of classes, available resources, and the frequency of various instructional activities.
The lead evaluator of each LSC project was given a list of 10 randomly selected teachers and asked to observe a class in targeted subject during the period March through May 1996. For Cohort 1 projects, the sample to be observed was drawn from a list of teachers who had already participated in LSC professional development.
Student Demographics
- The typical class had 21-25 students in class at the time of the observation; only 7 percent of observed classes had more than 30 students.
Class Size
Figure 42 - Minority enrollment in LSC classes is quite high; in half of the observed classes, more than one-half of the students were members of minority groups.
Minority Enrollment in Observed Classes
Figure 43
Adequacy of the Physical Environment
Observers found a "normal distribution" in terms of classroom resources, with 83 percent of the classes rated at 2, 3, or 4 on a five-point scale from 1, "sparsely equipped" to 5, "rich in resources."
There appeared to be a difference between Cohort 1 and Cohort 2 classes in terms of adequacy of classroom space, with more than two-thirds of Cohort 2 science and mathematics classes, but fewer than one-third of Cohort 1 science classes judged to have adequate space (4 or 5 on a five-point scale, where 1 is "crowded.")
Purposes and Content of the Lessons
Prior to each observation, the teacher was asked about the purposes of the lesson. As can be seen in Figure 44, there were major differences in reported purposes of science and mathematics lessons. For both Cohort 1 and Cohort 2 science classes, developing conceptual understanding was the most frequently-cited purpose of the lesson, with more than 60 percent of observed teachers (compared to only 36 percent in mathematics) indicating conceptual development was a major purpose. Similarly, about 20 percent of the science lessons had identifying student prior conceptions as a major intended purpose, compared to fewer than 10 percent for mathematics.
Purposes of Science and Mathematics LessonsFigure 44 In contrast, mathematics lessons were much more likely to focus on reviewing concepts: review was cited as a major purpose for 51 percent of the observed Cohort 2 mathematics lessons, compared to 11 percent for Cohort 1 science and 23 percent for Cohort 2 science. In fact, 38 percent of the Cohort 2 mathematics lessons had reviewing concepts as the only major purpose listed, compared to 2 percent of the Cohort 1 science lessons and 7 percent of the Cohort 2 science lessons.
In terms of disciplinary content, both Cohort 1 and Cohort 2 science observed lessons were much more likely to focus on life science than on either physical science or earth/space science; only a handful of lessons focused on engineering and design principles.
Focus of Science LessonsFigure 45 Mathematics lessons most frequently focused on computation, followed by measurement, numeration and number theory, and mathematical patterns and relationships. Very few mathematics lessons included a focus on data analysis and probability or algebraic concepts.
Focus of Cohort 2 Mathematics LessonsFigure 46 Classroom Activities
As can be seen in Table 16, a typical observed lesson included both a formal presentation by the teacher and some kind of hands-on/investigative activity by the students. Teacher presentations were particularly prevalent in the baseline mathematics classes, with 89 percent of the observed lessons including a formal presentation by the teacher, compared to 78 percent of baseline science classes and 71 percent of Cohort 1 science classes.
Table 16
Instructional Activities in Observed Classes
|
Percent of classes |
|
Cohort 1 Science |
Cohort 2 Science |
Cohort 2 Mathematics |
Formal presentation by teacher | 71 | 78 | 89 |
Hands-on/investigative activity | 73 | 60 | 72 |
Following detailed instructions | 56 | 44 | 57 |
Record/analyze/represent data | 44 | 33 | 18 |
Work on models/simulations | 19 | 14 | 6 |
Design/implement own investigations | 7 | 2 | 3 |
Play game to develop/renew knowledge/skills | 2 | 11 | 14 |
Field work/field trip | 1 | 3 | 0 |
Class discussion | 45 | 40 | 18 |
Whole class, teacher led | 38 | 37 | 15 |
Small groups | 12 | 10 | 3 |
Whole class, student led | 3 | 1 | 0 |
Reading/writing/reflection | 65 | 59 | 45 |
Writing reflections in notebooks/journals | 22 | 8 | 1 |
Reflecting on activities individually or in groups | 17 | 15 | 5 |
Answering textbook/worksheet problems | 12 | 24 | 38 |
Reading about science/mathematics | 11 | 17 | 7 |
Practicing routine computations | 1 | 3 | 12 |
Identifying patterns | 1 | 2 | 5 |
In both science and mathematics classes, investigative activities tended to be ones in which students followed a detailed set of instructions rather than designing and implementing their own investigations. Both Cohort 1 and Cohort 2 science lessons were more likely than Cohort 2 mathematics lessons to include having students work on models or simulations and be involved in recording or analyzing data.
Similarly, class discussions were more commonly observed in science lessons than in mathematics; in both subjects these were much more likely to be whole class discussion led by the teacher than either small group discussion or student-led whole group discussion.
Relatively few observed classes in either Cohort involved students in reading about science or mathematics. In contrast, answering textbook/worksheet questions was fairly common, especially in the baseline classes, with 12 percent of Cohort 1 science classes, 24 percent of Cohort 2 science classes, and 38 percent of Cohort 2 mathematics classes involved in that activity. Finally, science lessons of Cohort 1 treated teachers were more likely to have students writing their reflections in a notebook or journal (22 percent compared to 8 percent of Cohort 2 science lessons and 1 percent of Cohort 2 mathematics lessons.)
|