|
A.
Introduction to Essay Questions | B. Writing and Scoring Essay
Questions
|
B. Writing And Scoring Essay Questions
|
Essay questions are supply or constructed response type questions and can be the best way to measure the students' higher order thinking skills, such as applying, organizing, synthesizing, integrating, evaluating, or projecting while at the same time providing a measure of writing skills. The student has to formulate and write a response, which may be detailed and lengthy. The accuracy and quality of the response are judged by the teacher. Asking students to simply generate a list of the planets or a definition of a spelling word from memory is not an essay question. There is not a specific category for such questions, but they would more nearly fit the Short Answer category than Essay. Essay tests or
questions should be aligned with objectives and instruction, as
are Because of the time needed to answer and score essay questions and the limited amount of content that can be covered in them, essay questions should be used only when other types of questions cannot measure accomplishment of the objective(s). They are particularly appropriate when there is some concern about test security and when the number of students being tested is small. They can be useful when there is little time to prepare the assessment but more time in which to grade it. The teacher can write essay questions fairly quickly. Fewer essay items can be completed by the students in the same length of time as other types of questions, so fewer essay questions need to be written. One essay question typically replaces several questions of other types, so sampling of the subject matter is more limited with essay questions. Essay questions eliminate the possibility of the students' guessing the correct answer. However, essay questions are generally more time-consuming to grade than other types of questions and there are several factors that can influence the grade awarded. To respond to essay questions, students need to have writing skills and may be unfairly penalized because of handwriting, spelling, grammar, neatness, vocabulary, sentence structure, organization, etc. if they are not part of the content being assessed. To prevent this from happening, we suggest that separate grades or scores be given for essay content and writing skills. Also, longer answers (which may contain irrelevant information) may be given better scores than shorter ones. There is a risk
that the grading of essay responses can be subjective and unreliable.
The concept of reliability is commonly applied to the results of
tests and measurement instruments. When consistent results can be
obtained with an assessment, we say that the instrument is reliable.
In the case of essay questions, reliability is When students are directed to take a stand on a controversial issue, the scorer must be careful to insure that the evidence and the way it is presented is evaluated, not the position taken by the student. If the student takes a position contrary to that of the scorer, the scorer must be sensitive to possible bias in scoring the essay because the student's position differs from that of the scorer. Brookhart (1999) developed a scoring rubric for an essay test based on three criteria: thesis and organization, content knowledge, and writing style and mechanics. Thesis and organization was judged on the extent to which it was defensible, clearly stated, appropriateness of supporting facts and concepts, and logical use. Macmillan (1997) suggests the following criteria for scoring arguments: distinguishing between facts and opinions, judging the credibility of a source, identifying relevant material, recognizing inconsistencies, and using logic. Essay questions designed to elicit lengthy responses are sometimes referred to as Extended response items. Restricted response items, on the other hand, include in the directions a limitation on the length of the answer (such as "in 75 words or less," "in one paragraph," or "in no more than two pages,"). As is true with other types of questions, one or more essay questions can be based on content supplied as part of the question: a poem, a chart or graph, a paragraph, etc. One author refers to this particular structure as an interpretive exercise. |
|
other
types of assessments. If the teacher has not taught students what
is meant by "compare" and "contrast" during
the course of instruction, assessment in which they are called upon
to do so in a new situation may be a test of their understanding
and interpretation of the terms rather their ability to demonstrate
the higher level skills involved. In other words, instruction should
prepare students for essay questions.
dependent
on the scoring of the question. For the scoring to be reliable,
there should be consistency among scorers. Two individuals independently
scoring the same set of papers should arrive at the same scores.
Unreliability may be shown by the teacher's awarding higher grades
to students who have a history of performing at higher levels when
their answers do not justify the better marks. The order in which
papers are graded can also have an impact on the grades that are
awarded. A teacher may grow more critical (or more lenient) after
having read several papers, thus the early papers receive lower
(or higher) scores than papers of similar quality that are scored
later. Also, if the scorer becomes tired, judgment can be affected.
| When deciding whether or not to use essay questions in your assessment and when developing the questions, consider the following guidelines. | |
| * | Determine the level at which thinking is to be assessed (refer to your objectives and instructional methods). |
| * | Keep in mind the reading and writing levels of the students. |
| * | Use essay questions only for outcomes that can't be measured by other types of questions. |
| * | Two or more questions that are more specific and shorter are preferable to a single longer question. |
| * | Define the task as specifically as possible without giving away the answer(s). From your directions students should be able to tell how the item will be scored (what you are looking for in the answer) and how much weight will be given to essay answers in determining the total test score. |
| * | Provide more than one essay item and let the student select which one to answer. The items may not be of similar difficulty. Good students may select the difficult question but not do as well as they might have on the easier question. Some students may not know which question they can answer best. |
| * | Suggest a (reasonable) time or page limit for each essay question. |
| * | Decide in advance what you are seeking in the answer. Write a model response and/or develop a scoring system that includes the information you will reward in the answer. |
|
Some of the tasks that are typically addressed by essay questions,
and item shells provided by some authors (Ebel, 1979 and Linn &
Gronlund, 2000) are listed below: Show understanding, explain Explain in your own words the meaning of . . . Explain in your own words what is meant by . . . Describe an example of . . . that you have witnessed or become aware of in your job. Explain what . . . meant when he said ". . . . " Why did the . . .
Predict, recognize cause and effect What are the major causes of . . .? What would be the most likely effects of . . .?
In light of the facts presented, what is most likely to happen when . . .? How would (Senator X) be likely to react to the following issue?
Using the principle of . . . as a guide, describe how you would solve the following problem situation. Describe a situation that illustrates the principle of . . .
Describe the reasoning errors in the following paragraph. List and describe the main characteristics of . . .
Describe a plan for proving that . . . Write a well-organized report that shows . . .
Which of the following alternatives would you favor, and why? Explain why you agree or disagree with the following statement. . . Write a letter to the County Commission to get increased funding for school computers.
In your own words, summarize . . . . State the main points included in . . . Briefly summarize the contents of . . .
Formulate several valid generalizations from the following data. State a set of principles that can explain the following events.
Using the criteria developed in class, write an evaluation of . . . Describe the strengths and weaknesses of . . . Attached are a portfolio and the rubric to be used to evaluate the portfolio. Evaluate the portfolio using the rubric. Explain your scores by citing examples from the portfolio.
List as many ways as you can think of for . . . Make up a story describing what would happen if . . .
|
| 1. | Write a paragraph about after-school employment for students during the school year. | |
| 2. | Write a paragraph about after-school employment for students during the school year. Your paragraph should include: | |
| a. | A topic sentence | |
| b. | At least four supporting sentences | |
| c. | A concluding sentence | |
| 3. | Write a paragraph for or against after-school employment for students during the school year. Your paragraph should include: | |
| a. | A topic sentence | |
| b. | At least four supporting sentences | |
| c. | A concluding sentence | |
| 4. | Write a paragraph for or against after-school employment for students during the school year. Your paragraph should include six to ten sentences. Be sure to include: | |
| a. | A topic sentence (5 points) | |
| b. | At least four supporting sentences (8 points) | |
| c. | A concluding sentence (4 points) | |
| You will have 20 minutes to write and revise your paragraph. | ||
The
basis for scoring should be determined by the teacher prior to the
administration of the assessment. Directions to the student should
specify if penalties will be assessed for errors in spelling, punctuation,
and grammar. The following suggestions can help increase the validity
and reliability of your essay questions and the reliability of the
scoring. |
|
| * | Review the text and your class notes; list the main points to be covered in the essay response. |
| * | Develop a model answer first to determine what you are looking for. |
| * | Score one essay item at a time for all students to increase consistency in your scoring. (Score everyone's answer to essay question 1 before going on to essay question 2.) |
| * | Rearrange the order of papers randomly after scoring each item, and attempt to have students' identities hidden from you when grading papers. |
| * | If possible (and particularly if the score is important), have another person independently grade the answers to see if their scores match yours. |
| * | One author notes that a diversity of responses probably indicates ambiguity in the question. |
|
For analytic scoring, the score is based on the extent to which predetermined essential components are present. Because it is more structured than holistic scoring, scores are easier to justify because students can determine specifically where they made errors or wrote inadequate or incomplete responses. In analytic scoring, essential parts of the answer are identified, then parts are scored individually. The main component(s) and subcomponents of the desired response are each assigned a specific point value. If all are weighted equally, a checklist may suffice. If some components are worth more than others, a form may be needed to show the student the points earned for the various components. An example of a point system (or checklist, if all elements to be scored have equal value) for use in evaluating the students' paragraphs in Examples 1 through 4 might resemble the one below (Example 5) . In this case, a point system is appropriate because of the different point values. After scoring the papers, there is still a question of how to convert the scores into meaningful information for the students. How many points does it take to earn an "A," or are the points converted to percentages (13 of 13 possible points = 100%). |
| Name ______________________ | Date _____________ | Score Total (13) _______ |
| I. | Indentation (1 point) | _____ | ||
| A. | Paragraph is indented | |||
| II. | Topic sentence (1 point each - 2 points total) | _____ | ||
| A. | Topic sentence is included | |||
| B. | Topic sentence is appropriately placed | |||
| III. | Supporting Sentences (2 points for each attribute for each of 2 sentences - 8 pts. total) | _____ | ||
| A. | They relate directly to the topic | |||
| B. | They are logically sequenced | |||
| IV. | Concluding Sentence (2 points each) | _____ | ||
| A. | Concluding sentence is included | |||
| B. | Sentence is appropriately placed | |||
| The response is scored as a whole in comparison with characteristics of answers representative of the pre-established score levels. Holistic scoring tends to be more subjective than analytic scoring. | |
| 1. | The first step in holistic scoring is to estimate the number of categories to which papers will be assigned. The characteristics of an answer in each category should be described. (This can be done through a rubric.) |
| 2. | Next, the scorer reads a few (5) papers to sample the quality of the responses (and perhaps make adjustments in the scoring system). |
| 3. | All papers are then read and sorted into stacks representing the categories used in grading. (Excellent, good, fair, poor; A, B, C, D, F; etc.) |
| 4. | The papers in each stack are read, moving papers to other stacks if necessary. |
| 5. | Finally, the scores are assigned to the papers. |
| Example 6 (below) offers an alternative to Example 5 for assessing the work of the students in Examples 1-4. While the example shows letter grades being awarded, the teacher could just as easily have determined that the highest category ("A" grade) earned 13 points or 50 points, or 100 points, the "B" grade papers earned 10 points, 40 points, 90 or 95 points, etc. | |
| A. | Very well organized and easy to follow; contains all required components. The student's position is clearly stated and well supported without the inclusion of unrelated material. |
| B. | Well organized, containing all required components. The student's position is clear and supported although extraneous, unrelated material weaken the student's position. |
| C. | Adequate organization containing all required components. The student's position is clear but not well supported. |
| D. | All required components are included. There is some attempt at organization. The student's position is poorly communicated and unclear. |
| E. | Not all components are present. The student's position is unclear. Statements appear to be in random order. |
| 1. | Is there a penalty for incorrect statements in the answer, although the correct answer is there? |
| There may be good reason to create a penalty for incorrect parts of an answer, if the correct answer is actually dependent on factors that are covered in supporting statements, or if incorrect statements represent knowledge of skills that have been addressed in instruction and should have been learned. | |
| 2. | Is credit for extraneous correct information (not related to answer) included? |
| Awarding credit for extraneous information is generally not wise. To do so raises additional issues; e.g., "What extraneous information is acceptable and what is not?" In addition, this award of "extra credit" suggests that students can take a shotgun approach to answering your questions - throwing in anything they think might have value with some hope of receiving a good score or grade. |
developing
good writing prompts.| 1. | Let the writer draw on experience - topics students know something about |
| 2. | Permit choice of topics or ways of addressing the content (Make up a story using ONE of these words . . . ; think of a time you felt . . . ) |
| 3. | Choose a prompt that will catch the imagination of the student. (It could involve use of a prop or object.) |
| 4. | Don't give too many suggestions or directions. At one point Spandel cautions against making suggestions that the teacher doesn't intend students to follow literally because students may perceive them as requirements rather than suggestions. |
| 5. | Avoid building a positive or negative connotation in the prompt (think of the best . . . ) |
| 6. | Avoid topics that are very controversial or might infringe on a student's privacy. |
| 7. | If the purpose is important, make it clear to the students. Directions might include such phrases as "tell the story" or "tell what happened" if the objective is to write a story. "Explain" should generate a different type of paper, while a paper written in response to directions to "Persuade or convince someone to agree with you" would differ from either of the others. |
| When constructing writing tasks as essay assessments, don't forget to include at appropriate places in the curriculum the kinds of writing tasks that are required in the state's writing assessment. Study the information provided to schools and teachers about the requirements of the writing assessment at different grade levels: | |
| * | What type(s) of writing is/are required? |
| * | What are the time limits imposed? (How long does the student have to produce the writing sample?) |
| * | What are the elements on which the writing sample is scored? |
|
|
|
Copyright
© 2001-2002 Alabama Department of Education
All Rights Reserved. |