ABC on Scoring Rubrics Development for Large Scale Performance Assessment in Mathematics and Science
3. Options for Scoring Rubric Development
Scoring rubric development requires teamwork among developers and users, and it is time consuming. Depending on the feasibility and resources, there are three options for selecting a scoring rubric: adopt, adapt, and do it yourself.
The easiest way to develop a rubric is to adopt a currently existing scoring rubric that matches exactly what you need in an item or a product. This approach is most likely to work with general scoring rubrics, and can be used for both analytic and holistic procedures (e.g., speech presentation rubrics). The advantage is that adoption is quick and easy and does not require much expertise in rubric development. The disadvantage is the difficulty of finding the exact match.
When existing rubrics are not available, an alternative is to adapt an existing rubric by any of the following methods:
3.3 Do It Yourself
Districtwide or statewide standardized assessments often require specific scoring to their performance assessment items or tasks, but there is simply no existing scoring rubric available. Every item is new and measures specific skills based on the local standards. To align with the local assessment standards, scoring rubrics need to reflect their contents. Consequently, test developers have to develop scoring rubrics from scratch. In the next section, we will describe step-by-step procedures about how to develop a scoring rubric. Again, the focus is on performance assessment items or tasks in mathematics and science in standardized assessments.
4. Development of a Scoring Rubric for Performance Assessment Items in Mathematics and Science
This section will describe briefly the common types of performance assessment items in standardized mathematics and science assessments, as well as general procedures to develop scoring rubrics for such items. Tips for developing a scoring rubric will be suggested also.
4.1 Common Types of Performance Assessment Items in Standardized Mathematics and Science Assessments
Performance assessment tasks can be used to assess a variety of learning outcomes. Higher order thinking, connecting, and integrating knowledge abilities are some of the typical examples for mathematics and science assessments. Depending on the goals they serve, performance tasks may be short or lengthy, and may employ different types of item formats. In mathematics and science particularly, the most common types of performance tasks in standardized assessments include, but are not limited to, the following:
4.2 General Procedures for Scoring Rubric Development
1. Based on the test blueprint, members of item development committee need to make a preliminary decision on the dimensions of the performance or product to be assessed.
Scoring rubric development is a team activity, especially for standardized assessments. Theoretically and practically, item writing and scoring rubric designing is an integrated process. Based on the test blueprint and assessment standards, item writers should work together with the content advisory committee to decide on the dimensions of each performance or product to be assessed. For mathematics and science, a holistic scoring procedure is usually used. Each item has specific scoring rubrics that are usually developed together with item writing, and this development is considered as part of the item writers responsibilities.
2. Look at some actual examples of student work to see if you have omitted any important dimensions.
Before designing scoring rubrics it would be helpful to look at some examples of student work to get a range of performance of what students are actually able to do. When the first draft of a rubric is done, looking at examples of student work again will help item writers verify the dimensions and degree of detail that each rubric measures against.
3. Refine and consolidate your list of dimensions as needed.
This action is a natural followup to steps 1 and 2. It will serve as the base for formal scoring rubrics writing. For holistic scoring in mathematics and science, the number of dimensions for an item does not usually exceed two to three. Mostly it is one.
4. Write a definition of each of the dimensions.
It is important to define the boundary and meaning of each dimension or trait to be assessed because the definition will provide the frame of skills and knowledge to be measured.
5. Develop a scale or a checklist on which you will record the presence or absence of the attributes of a quality product/performance.
In this critical step, scoring rubric developers need to decide such characteristics as whether the scale is numerical, qualitative, or a combination of the two; maximum possible scores; and a description for each level of scored performance.
When the total number of performance levels or maximum score points is decided, a description of performance at each score level will substantiate the meaning of the assigned score. The description can be as short as a sentence and as long as a small paragraph. The following strategies can be considered when developing a performance scale:
6. Evaluate your rubric using the following questions.
7. Peer review and pilot test the rubrics on actual samples of student work.
Always have your peers review the rubrics together with the item to make sure that they cover what was intended. Piloting rubrics will help establish that the rubric is practical to use and that scorers can generally agree on what scores they would assign to a given piece of student work. When pilot testing a rubric, it is best to use examples of work that span the entire continuum from very poor to very good on a scale.
8.Revise the rubric and try it out again.
It is unusual to get everything right the first time, so be prepared to revise a rubric. The analysis of the pilot test result and the criteria stated above can serve as the guides.
9. Share the sample rubric with teachers, students, and parents.
This may entail rewording the rubrics in less technical language. Give teachers and students a clear target to aim for by letting them know what constitutes quality work that meets the standards. Training teachers to use the rubric to score student work can be a powerful instructional tool. Sharing the rubric with students and parents will help them understand what is expected from students.
However, when test items are secure materials, sharing item-specific rubrics will disclose the items. A solution to this dilemma is to provide teachers some exemplars of items with rubrics so that they can have an idea of assessment expectations.
4.3 Tips on Scoring Rubric Development
Perhaps the most common challenge is avoiding unclear language. Take the phrase "creative solution" in a rubric for a science investigation, for example. The word creative is difficult to define. If a rubric is to teach as well as evaluate, such terms must be defined for students. One possible approach is to discuss what the term "creative solution* means. Another is to actually list ways in which students could meet the criterion, thereby providing valuable information to students on how to begin to develop a solution and avoiding the need to define elusive terms.
A second challenge in rubric development is avoiding unnecessarily negative language. Again, use the "creative solution" as example. The rubric should avoid words like boring by describing what was done during a so-so solution and implicitly comparing it with the highest level of quality. Thus, students know exactly what their problem is and how they can do better next time, not just that their solution lacked fresh ideas.
Articulating gradations of quality is often a challenge. It helps if you spend adequate time thinking about criteria and how best to combine them before going on to define the levels of quality. One helpful approach is trying them out as described in step 7 above before the actual operation application.
4.4 Checklist of Scoring Rubric Development
This checklist, which summarizes the procedures of rubric construction, will help you verify each step in the process.
5. Sample Sources of Additional Information for Scoring Rubrics Development
With the increasing use of scoring rubrics in various assessments and evaluations, more and more scoring rubrics are developed to serve different purposes. Much information on this topic has been published on line and off line. Below are sample sources available for scoring rubric developers as references. They contain many examples of scoring rubrics for mathematics and science items.