Annual Report Overviews

Assessment in the Service of Standards-Based Teaching: Goals and Objectives of the CRSEP Initiative

NSF Award 9911868

Assessment in the Service of Standards-Based Teaching
Overview of Activities and Findings

The goal of the Capital Region Science Education Partnership (CRSEP) project continues to expand and deepen content and pedagogical knowledge of teachers through profession development with a primary focus on assessment in the service of standards based teaching.

Professional Development Project

Activities-Education

Between November 1, 2001 and October 31, 2002, CRSEP staff provided high quality professional development activities designed to provide teachers with the content knowledge and pedagogy necessary to improve their students' understanding of science and improve their students' performance on state mandated science tests administered at grades 4 and 8.

Alignment of districts' elementary science curricula with the New York State Elementary Core Curriculum and the Elementary Science Program Evaluation Test (ESPET) and the alignment of the districts' middle level science curricula with the New York State Middle Level Core Curriculum and the Middle Level Science Test (MLST) are necessary conditions for improving student performance on the ESPET and ILST. Consequently, considerable professional development time has been devoted to developing teachers' ability to do valid alignment. While alignment may seem a mechanical process, it is not. Good alignment requires considerable knowledge of psychology and science. As teachers work to interpret the meaning of the standards and assess the conceptual demands of test items and tasks, they are not only applying what they know but also expanding their understanding.

The alignment work has revealed gaps in the science curricula and in teachers understanding of the science content students are expected to learn. This summer several workshops were presented to design activities and assessments to fill the curriculum gaps and to provide content information to teachers in the form of concept maps and the big ideas to be uncovered in STC Units. This fall teachers are continuing this work by compiling this information in Teacher Resource Guides.

In addition, CRSEP staff provided professional development focused on the design of valid end-of-unit and end-of-year assessments as well as the use of item statistics. Teachers administered tests they designed and are in the process of analyzing the data from the field test of the assessments. Other teachers are using item level statistics from the state mandated tests to identify content on which district students are performing poorly and plan academic interventions using that information.

We continue to explore the integration of English Language Arts with science in workshops focused on writing in science. We have also engaged teachers in the design of science games as a way of developing their science content knowledge and strategic science thinking.

Activities-Education

Action research is the essence of this project. Our knowledge of adult learning, science learning, curriculum, test design, and data analysis is applied to our understanding of the needs of the students, teachers, and districts in which we work, to plan professional development that will meet the needs and apply the best educational theories available to us. We assess the effects of the workshops primarily based on our own critical observations and on feedback we collect from the participants. We redesign our professional development based on our observations and participant feedback in a cycle of implementation, feedback, and redesign.

Unfortunately, we have learned that many constraints operate on our professional development activities over which we have no control. Among these are the size of stipends the project can offer, the overriding focus on mathematics and reading motivated by the NCLB legislation, the varieties of professional development available to teachers requiring less time and effort than the CRSEP activities and unwillingness of teachers to leave their classrooms or give up time in the summer for professional development. All our observations, positive feedback from teachers, and attempts to think out of the box have not provided us with insights into how to address these stark realities of the demands on the students, teachers, and administrators in the CRSEP project.

Materials Development

Between November 1, 2001 and October 31, 2002, Materials Development staff continued to work on the same activities as reported in November 2001, while exploring new directions and adding new materials.

The planned distribution of our materials through the National Science Teachers Association did not work out, due to circumstances beyond our control. Thus, we reverted to our original dissemination plan of making our materials available through the Internet. To that, we added the idea of designing and piloting graduate-level, on-line or partially on-line, professional development courses that make use of the materials and knowledge we have gained in our CRSEP work. Once the courses have been developed, they could be disseminated to other institutions or other providers of science education professional development. The courses would be incorporated into usual graduate teacher education programs and, thus, would become self-sustaining.

We met with our newly hired materials development/web development consultant, Eileen O'Connor, to discuss requirements for a wide range of issues with regard to materials development. Topics of discussion included web site development relevant to mission; possible web support for presenting our material in graduate level courses; content to be communicated; information organization; course development and design issues; general technology issues, and considerations of an in-person course as a pilot for online courses. Dr. O'Connor worked on two reports, one on web courses, server and design issues and server requirements for and the feasibility of discussion boards within courses. The second report was on defining issues and questions to help determine technological capacity requirements. We also worked with Dr. O'Connor in the design and posting of two "under development" web sites: one was designed initially to demonstrate various possible uses for a web site by CRSEP and to serve as an "idea" template for CRSEP. The second site was designed as an initial framework for an online course that could demonstrate course organization and could serve as a template for either a stand-alone course or a model for a program design. Sample course content was placed on the site for design review by CRSEP staff. One result of our investigations was learning that the relational concept maps developed with Inspiration software do not easily transport to online documents. We anticipate having to reconstruct all concept maps using a different format such as power point.

We continue the work on the activities reported in November 12, 2001, and added to that work with Professional Development Associates and other teachers involved in the project on supplemental teacher guides and resource materials to support or fill gaps in the science content "big ideas" in the STC units.

Activities-Research

Materials Development staff analyzed aspects of the use of relational concept maps and reported on that analysis at a regional research conference in Maine.

High Stakes Testing Study

Activities-Research

We continue to collect and analyze data on the testing policy context in NYS. We have begun an historical examination of the Regents Exams, as this seems a critical element in the contemporary meanings stakeholders make of the current state tests. In addition, we continue to collect stories and accounts in the media about NYS testing.

We have also begun to examine the "test resisters" in New York State. This is a small group who contribute to the public discourse on testing. We have interviewed a number of individuals associated with the Alternative Schools Coalition in the state as well as other teachers, parents and school administrators. We are using a snowball approach to sampling and anticipate we will continue to interview resisters throughout the upcoming year.

The case study fieldwork is the major component of our research. We completed two case studies and four are in progress. These case studies entail a field researcher being in the school at least one day per week, although often more. Data are collected through classroom observations, informal interviews with school personnel, focus groups with teachers and parents, and attendance at appropriate school events (faculty meetings, open houses, and so on).

Student Achievement Study

Activities-Research

Approval to conduct the Student Achievement Study was received by the University's Institutional Review Board (IRB) in February 2002. Student Achievement staff worked with partner school districts and BOCES centers that process state test data to arrange for student scores to be sent in a usable format, and to request lists of students that science teachers instruct each year. This activity has to be coordinated with each individual school since there is no uniform structure for the data reporting and analysis. For example, one district submits scores by homeroom teacher, another uses science classrooms as an organizer, and a third lumps all students together. While transferring data to our computer systems, we discovered import problems and had to import some data by hand.

Teachers were contacted via e-mail or letters and were asked to provide required background data to be used as control variables in the studies. To date we have had an 11% return on that data. One of the challenges of formally working with schools is that of working within their hierarchical structure. Teacher contact is made through school/CRSEP coordinators; thus, we are dependent on them. We were able to contact teachers more directly in two districts via their school e-mail addresses. This means has not proven any more fruitful than having school coordinators hand-deliver our questionnaires to teachers. Our next approach is to send representatives to faculty meetings to distribute the questionnaires and make a plea for a response.

As we work with the initial data, it has become clear that our computers do not have the capacity to handle the volume of data (400+ teachers and 13,034 students). We also discovered that the university does not have a site license for the software needed to do the analysis. The Department of Educational Theory and Practice in the School of Education at the University at Albany is replacing our 3-computer network with top-level computers and assisting in the purchase of the needed software. The systems and software are scheduled to arrive the week of November 11.

FINDINGS:

CRSEP Initiative Findings

The four components of the CRSEP Initiative provide evidence at the project level of the systemic nature of achieving the changes necessary to provide all students the opportunity to achieve the New York State Learning Standards for Science. Each component provides findings that inform the work of the others. For instance, parent and community impressions of state testing uncovered by the High Stakes Testing study allow the Professional Development component to provide teachers with insights as to how high stakes testing can be beneficial to classroom practices and to understand why sharing those insights with parents and community leaders is important. Our experience with teachers in the Professional Development component have helped us understand better the differences in the cultures of teaching and professional development. We then translate that understanding into strategies we use in the Materials Development component to help teachers move their thinking from the classroom and its responsibilities to the professional development arena and its responsibilities.

Professional Development Findings

Despite the documents developed to help teachers provide students with the opportunity to meet the New York State Learning Standards, translating the expectations conveyed in the standards to classroom practice presents teachers with an insurmountable challenge. The shear volume of words is overwhelming. The messages have evolved and consequently changed over time. Suggestions in the documents are poorly aligned with the standards and with the state mandated tests. Consequently, teachers follow their instincts and teach to the tests. However, without the mediating effects of thinking behind the design of the tests and their links to the standards, teaching to the test becomes more vocabulary, and drill and practice than the development of understanding of science principles intended by the standard setters.

As previously mentioned under Professional Development Activities-Research section of this report, we have learned that many constraints operate on our professional development activities over which we have no control. Size of stipends, the focus on mathematics and reading motivated by NCLB legislation, the varieties of professional development available to teachers requiring less time and effort than the CRSEP activities, the unwillingness of teachers to leave their classrooms, and the reluctance to participate in summer professional development activities constrain our work. Observations, positive feedback from teachers, and attempts to think out of the box have not provided us with insights into how to effectively address these stark realities of the demands on the students, teachers, and administrators.

The rate of teacher turnover in the CRSEP districts is significant. Data for teacher participation in CRSEP professional development have been based on four sampling frames. The original composed in May 2000, and modifications made in November 2000, November 2001 and November 2002. The percent of teachers who were in the original sampling frame and are not in the November 2002 sampling frame is 50%. These changes have resulted from teachers whose responsibilities have changed and are no longer eligible for professional development or teachers who have retired or taken leave and not returned.

Materials Development Findings

Our findings for this current year have not changed substantially from the November 2001 report, except with the addition of the following. Many teachers are not practiced or adept at designing their own professional development, thus we have added a unit to our work on designing a professional development plan.

High Stakes Testing Study Findings

Although perhaps self-evident, our research demonstrates the power of state mandates to influence local practices, although how that influence is manifest in individual schools varies. The following statements highlight some of our findings to date:

• In large part, the accountability measures in New York State are driven by the Commissioner of Education and critical perspectives on testing and accountability within the NYS Education Department and in most schools are cautious and muted.
• For the most part, school districts criticize state mandated tests for being excessive but have adopted assessment models that mimic the state's to ensure their students do well on the state tests, sometime out of fear of punishment.
  
  
• State tests have pushed schools to focus primarily on academic tasks at the expense of other content and psychosocial connections considered essential to students' self-esteem and successes.
  
  
• Most teachers believe that students can learn and do well in school. We have found this to be especially true in urban contexts where there are many poor and minority students.
  
  
• Teachers feel a sense of disenfranchisement as professionals as state mandated testing usurps some control over curriculum and pedagogy.
  
  
• Teachers experience ongoing ethical quandaries as they try to be good teachers - preparing students to do well on tests the teachers do not necessarily believe in - and following rules established by NYSED even when the rules are unclear or disadvantage children.
  
  
• The strategies used by schools for meeting the expectations of the state assessment system tend to follow and extend already existing practices (whether or not these have been shown to be effective) rather than implement new ones.
  
  
• The need to provide remedial services to children who perform poorly on state mandated tests has, in some schools, created a chaotic array of uncoordinated academic services.
  
  
• The curriculum in elementary grades, especially 4th grade which is most heavily tested, has taken on an annual, rather than daily, rhythm dictated by the dates each state test is administered: language arts, followed by math, followed by science.
  
  
• Teachers interpret the accountability measures as both necessary to raising standards and as being invalid indicators of quality education.

Student Achievement Study Findings

Some teachers' days are such that even taking time to complete a one-page questionnaire on their background is a burden.

Some elementary teachers have had no college-level semesters in science content (we do not have a percentage to report because data collection is not complete).

Science performance on NY state exams at the elementary level is not reported in the same way that performance on mathematics or English Language Arts is reported. For English and mathematics, the percent of students scoring at each of four levels is reported, with scores ranging from 603 to 800 for English and 602 to 810 for mathematics. Elementary level science scores are not leveled but are broken into components (multiple-choice and hands-on/inquiry performance) with a much smaller range of possible scores 0-45 for the multiple choice items and 0-29 for the performance component. The only level reported is the percent of students above the State Designated Level (30 out of the 45 points on the multiple-choice items). Students below the SDL must receive mandatory intervention services. These differences serve to place science behind English and mathematics in terms of teacher's professional development and teachers' willingness to participate in a study of student performance in science.