Are you ready?

TASK

PURPOSE

Determine the standard(s) to be assessed and how they should be interpreted (by unpacking/deconstructing)

• Agreement by stakeholders on which standards or components of the standard are important.

• Not all standards are equal in terms of developing student understanding of curriculum content.

• Most standards are subject to multiple interpretation by teachers, therefore, if agreement is not reached, students in different classes will not have the same educational experience (for better or worse, depending on the teacher)

• Need to find agreement on the essential elements of a standard (knowledge, skills/understanding, reasoning)

Locate released items from state and nationally normed tests.

• These items have typically undergone a rigorous field test and review before being accepted into an item bank

• Many of the questions are written to assess more than knowledge and comprehension (require higher order thinking)

• Many state and national tests (college board, AP, NAEP, SAT) have been written by professional test writers

• Be careful when using item banks provided by a publisher…these questions typically have not undergone a rigorous quality control. They typically focus on lower order think skills with an emphasis on comprehension of academic terminology and/memorization of terms.

Analyze each released item for alignment to standards and its cognitive level.

• Trains the mind to become more critical in analyzing and selecting questions that are written at higher cognitive levels (Bloom’s), vs. those that focus on lower levels.

• Helps participants negotiate and come to agreement in terms of what is important to assess vs. what is simply easy to assess

• Helps participants to learn how to write quality prompts (assumes that released items are decent models) and distracters

• Improves the participant’s ability to write test items.

Determine the schematic for the assessment tool to be designed:

• Establishes the number of questions per standard – allotting more questions to more important standards or objectives

• Establish a level of rigor students will be expected to perform, rather than allowing the availability of questions to set the standard for rigor by default.

• Determine the format to be used (# of Multiple Choice questions, Open-ended, short answer etc.)

• Standardize the “look” of the assessment tool – Consider formatting the tool similar to the state or national assessment your students will be expected to take.

• Layout the answer key to ensure equal distribution of distracters and the absence of a predictable pattern.

• Consider the first design to be a draft. It is extremely rare that one can design both a valid and reliable tool the first time. That’s why it takes most testing companies 2-3 years to develop a decent tool for evaluating student achievement. Yet teachers and schools routinely evaluate and make judgments about student achievement based on test items that may be flawed and poorly designed as a result of limited, or no, quality control.

• Select appropriate test items from released items collected earlier in the process and modify as needed.

• Write test items to fill in gaps where no released items are found. Use criteria and guidelines provided by LLL for writing new items.

• Unless you have considerable training and experience writing high quality selected response questions (i.e. you do this for a living), accept the fact that you will probably write a few questions that have poor prompts, unclear distracters, are cognitively inappropriate, poorly worded, or worse, simply do not assess what they are suppose to assess. You need your colleagues and their “eyes” to help you see some of these shortcomings. Nobody’s perfect!

• If your colleagues are going to use the assessment tool you have designed they should at least have some opportunity for input.

• The field test provides student data that will be used to identify questions that might require additional revision

• The results of the field test will be helpful to establish performance bands (performance standard for mastery) during the next step of the process.

• Provides initial data a teacher can use to identify re-teaching targets, despite the fact that some of the questions may later be determined to be invalid (in other words, it isn’t a wasted effort).

• Reveals patterns in student responses, and therefore, their thinking

• Reveals items that may need to be revised or eliminated.

• Reveals re-teaching targets (what needs to be re-taught, and who).

• Data needed to establish performance bands (determine how to interpret scores, for example, “what score is needed to be considered “proficient”?)

• Establishes “cut scores” based on (1) professional agreements among users of what constitutes mastery

• Establishes “cut scores” based on expectation rather than arbitrary scores (i.e. 70%=C)

• Establishes “cut scores” based on the rigor of the test, conceptual difficulty, importance of the standard relative to other standards, and student performance.

• Eliminates artificial grading scales that assume all assessments are qualitatively identical.