Paper

Scoring rubric development is a team activity, especially for standardized assessments. Theoretically and practically, item writing and scoring rubric designing is an integrated process. Based on the test blueprint and assessment standards, item writers should work together with the content advisory committee to decide on the dimensions of each performance or product to be assessed. For mathematics and science, a holistic scoring procedure is usually used. Each item has specific scoring rubrics that are usually developed together with item writing, and this development is considered as part of the item writers’ responsibilities.

Before designing scoring rubrics it would be helpful to look at some examples of student work to get a range of performance of what students are actually able to do. When the first draft of a rubric is done, looking at examples of student work again will help item writers verify the dimensions and degree of detail that each rubric measures against.

This action is a natural followup to steps 1 and 2. It will serve as the base for formal scoring rubrics writing. For holistic scoring in mathematics and science, the number of dimensions for an item does not usually exceed two to three. Mostly it is one.

It is important to define the boundary and meaning of each dimension or trait to be assessed because the definition will provide the frame of skills and knowledge to be measured.

In this critical step, scoring rubric developers need to decide such characteristics as whether the scale is numerical, qualitative, or a combination of the two; maximum possible scores; and a description for each level of scored performance.

When the total number of performance levels or maximum score points is decided, a description of performance at each score level will substantiate the meaning of the assigned score. The description can be as short as a sentence and as long as a small paragraph. The following strategies can be considered when developing a performance scale:

• Start with the highest level of performance. Describe an outstanding product/performance. What characterizes the best possible performance of the task? This description will serve as the anchor by defining the highest score point on your rating scale.
• Then work on the lowest level of performance. Describe the worst possible product/performance. What characterizes the worst possible performance of the task? This will serve as a bottom line of performance on your rating scale.
• When the ceiling and bottom levels of performance are defined, it will take less effort to describe characteristics of products/performances that fall at the intermediate points of the rating scale. Often these points will include some major or minor flaws that prevent the product/performance from receiving a higher rating.
• Avoid ambiguity in wording of description. Keep the description succinct.
• Avoid overlapping in characteristics of different levels of performance. This point is very important in that it will allow you to categorize students of different abilities to the appropriate level. It will also help scorers in the process of hand scoring.

• Does the rubric relate to the outcome(s) being measured? Does it address anything extraneous?
• Does the rubric cover important dimensions of student performance by standards?
• Do the criteria reflect current conceptions of "excellence" in the standards?
• Are the levels of performance well defined?
• Is there a clear basis for assigning a score at each scale point?
• Can the rubric be applied consistently by different scorers?
• Can teachers, students and parents understand the rubric?
• Is the rubric developmentally appropriate?
• If it is a general rubric, can it be applied to a variety of tasks?
• Is the rubric fair and free from bias?
• Is the rubric useful, feasible, manageable, and practical?

Always have your peers review the rubrics together with the item to make sure that they cover what was intended. Piloting rubrics will help establish that the rubric is practical to use and that scorers can generally agree on what scores they would assign to a given piece of student work. When pilot testing a rubric, it is best to use examples of work that span the entire continuum from very poor to very good on a scale.

It is unusual to get everything right the first time, so be prepared to revise a rubric. The analysis of the pilot test result and the criteria stated above can serve as the guides.

This may entail rewording the rubrics in less technical language. Give teachers and students a clear target to aim for by letting them know what constitutes quality work that meets the standards. Training teachers to use the rubric to score student work can be a powerful instructional tool. Sharing the rubric with students and parents will help them understand what is expected from students.

However, when test items are secure materials, sharing item-specific rubrics will disclose the items. A solution to this dilemma is to provide teachers some exemplars of items with rubrics so that they can have an idea of assessment expectations.

A second challenge in rubric development is avoiding unnecessarily negative language. Again, use the "creative solution" as example. The rubric should avoid words like boring by describing what was done during a so-so solution and implicitly comparing it with the highest level of quality. Thus, students know exactly what their problem is and how they can do better next time, not just that their solution lacked fresh ideas.

Articulating gradations of quality is often a challenge. It helps if you spend adequate time thinking about criteria and how best to combine them before going on to define the levels of quality. One helpful approach is trying them out as described in step 7 above before the actual operation application.

This checklist, which summarizes the procedures of rubric construction, will help you verify each step in the process.

With the increasing use of scoring rubrics in various assessments and evaluations, more and more scoring rubrics are developed to serve different purposes. Much information on this topic has been published on line and off line. Below are sample sources available for scoring rubric developers as references. They contain many examples of scoring rubrics for mathematics and science items.