In Defense of Utah Write
By this time, most students here have experienced the
writing program called Utah Write, and most parents have noticed the lab fee for
it at registration. Utah Write is a computer program that evaluates and
“scores” certain traits of student writing.
Every year the program becomes more sophisticated, and the state of Utah is now using the
Utah Write algorithms to score both the Direct Writing Assessment and the
English CRT, two of the “high-stakes” tests all students have to take. Despite that, Utah Write is also the cause of
most of the complaints I have heard over the past couple of years. I understand both sides of the issue, and I
have given a great deal of thought to how this program will be used in my
classroom. In the interest of addressing
some of the common complaints and preparing students (and parents) for what
this year will hold in store, I offer this defense of Utah Write.
Because it is a computer program and not a human being, Utah Write can't actually understand the content of a student’s writing. This is the basis of most of the complaints I have heard in recent years: “How can a computer grade writing?!” This is a perfectly legitimate and correct criticism, and in an ideal world, I would have plenty of time to read everything my students write, evaluate it completely, and offer timely and instructive feedback. But this is not an ideal world when it comes to the resources available for public education.
Every year of my 24 years in teaching, there have been more students in my classes than the year before. Last year, I had 241 English students, an average of just under 35 per class. If I were to personally offer constructive feedback on even a single-paragraph written by each of those students, it would take a week. The core curriculum calls not just for single paragraphs but for 1000+-word essays, many of them, over the course of the year. To personally score everything my students are required to write and give constructive feedback on all of it would take more time than there is in a school year and I would do little else, even after school and overnight. Furthermore, by the time the feedback got to the students whose papers were at the bottom of the pile, it would no longer be meaningful. (Even using Utah Write, I often have to resort to a system of evaluation codes and general suggestions to score term papers before the term ends). So, please understand that the big reason I started using Utah Write was because I wanted my students to have some kind of feedback on all their writing, which would be impossible given current (and, probably, future) class loads. If I want to have my students complete more than one writing assignment per term (which I do), there is no way it can happen if I have to personally score and give feedback on every assignment. Mind you, this isn't because I am shirking my responsibilities as a teacher (although I have had parents accuse me of that, some more subtly than others). Fact: There are too many students in the system to accommodate a personal touch on every writing evaluation. No human could read and score that much writing in a reasonable amount of time. So, I am left with a choice: I can have an admittedly imperfect computer program help me evaluate and give immediate feedback on the writing, or I can have the students write less often. Since I believe that students should write as much as possible, I have chosen to use technology to make that happen. Practice may not make perfect, but it beats no practice. Also, now that the state-mandated tests are using the same algorithms to evaluate students, it seems only fair that students get familiar with the program.
If your philosophical opposition to Utah Write is so strong that you don’t want to see it used in schools, please contact the state legislature and campaign for two things: reasonable class sizes and no high-stakes writing tests that are scored by computers. You will likely be told, as I have been, that the expense of these suggestions makes them impossible dreams, but perhaps you have more clout than I do. Until such a change is voted into reality, Utah Write is with us and it is going to be used to evaluate and score student writing in this class and on the final state-mandated exam.
Now, back to that complaint: “How can a computer grade writing?!” I know that this question is usually asked rhetorically (and angrily), but I would like to try to answer it. While Utah Write can’t actually “know” what a piece of writing is saying or evaluate the logic of the argument contained within it, the program DOES determine if verbs are used correctly, how long and varied the sentences are, the grade-level of the vocabulary, whether or not transition words and phrases are employed, if the writer depends on clichés, the degree of repetition, and whether or not punctuation marks are used correctly. Utah Write can also check grammar and spelling and look at the statistics of the writing (word count, paragraph length, etc.) just like Microsoft Word does. It is not perfect, but it is feedback...and it can generate that feedback to every student in less than one minute. This is why I use it. If Utah Write indicates that a student’s organization is flawed or his fluency needs work, at least it is a place for the student to start a revision. Utah Write is part of the process, not just the result. I generally find that students who object most strongly to the program are those who pay no attention to the process and don’t really focus on their writing until Utah Write has scored it poorly; they just hurry through a draft in a hopeful attempt to earn a good score with very little effort, much like many students who don’t read will make a desperate, last-minute attempt to write a book report that will fool a teacher into giving them points. For some students, that method might work part of the time…until I get around to judging the quality of the argument (i.e., the content). However, most of the time, Utah Write – by identifying the elements of good writing that it IS able to evaluate – gives a legitimate evaluation of most traits of a student’s writing. To the degree that it is possible, Utah Write recognizes good writing. I have used the program long enough to know that those who address all the elements of an effective argumentative essay will get a good score. Lazy or sloppy efforts will not. So, while it can’t make a judgment about the quality of logic used in a paper, Utah Write does a pretty good job with all the other writing traits.
Because I am fully aware of the limitations of Utah Write, I go to great pains during class to make sure that students have the benefit of every conceivable resource to help them improve their writing. I tell them from the beginning that Utah Write is imperfect in many ways, but they may revise and resubmit as many as 50 times to achieve a score they can live with, and they have more than a week between the time the rough draft is reviewed and the time the final draft is due. Furthermore, almost every step of prewriting and drafting can be completed during class time in the lab (if the time is used efficiently). While they are writing, I use a program called Net Support to watch students work – from my computer, I see what they are doing at their computers – so I have some idea where help is needed. Granted, because there are 35+ kids in every class, such help often comes in the form of general statements to the entire class when I notice, for example, that many of them are writing incomplete introductions or unclear thesis statements. Again, one-on-one help is rare when there are so many of them, but I try. First and foremost, I provide (on the website and/or in class) lots of good examples. For every writing assignment, I display and analyze a well-done example and encourage students to mimic the strategies of the successful writer. When appropriate, I run the good example through Utah Write so students can see how effective writing earns good scores even from an artificial intelligence. Sometimes I will also show and analyze non-examples, poorly done essays that earned low scores. In other words, I try to make it crystal clear to students what to do and what not to do in their own work. With few exceptions, those who pay attention and revise their own writing with care earn top marks both from Utah Write and from me.
To this end, I once had the Honors English classes do a comparative analysis of papers scored by Utah Write, and we looked specifically at how to raise a score. The results were surprising, and you can see them here. In developing this list, we learned that while Utah Write may not "know" what the writing is about in the way a human reader would, the algorithm is sophisticated enough to identify certain qualities of good writing and to differentiate between good writing and mediocre writing in many respects. In other words, Utah Write can provide helpful suggestions for improvement, and this list of student-generated observations is actually full of excellent suggestions for raising scores as well as making writing noticeably better even to the humans who read it. It might be possible to fool Utah Write into giving a high score to a lengthy piece of written nonsense, but that would be far more difficult than just writing a good paper to begin with and revising it carefully, following the suggestions offered by the program. (Did I mention that Utah Write has built-in writing tutorials that are focused on the student's individual writing weaknesses? Well, it does, and if students like the one I am about to introduce would bother to complete those tutorials, I don't think I would get any complaints.) In short, the suggestions developed by the Honors English classes support good writing in general, not just when using Utah Write, and they are always available to students who are working on a Utah Write assignment in my class.
Here is the whole story behind the most recent complaint I had about Utah Write. On April 29th, “Bob” submitted 15 "revisions" (i.e., hit the "save" or "submit" button) in less than an hour and a half. Many of those “revisions” included nothing more than correcting a single misspelled word or changing the spacing. These were not truly revisions but more editing and “polishing” changes. (Utah Write marks such errors in red, similar to what Microsoft Word does, so they are easy for students to identify, but unless there are hundreds of such errors in a paper, fixing only the redlined suggestions will not raise a score very much.) On May 8th, Bob submitted seven more “revisions” between 6:02 and 6:24 P.M. That’s about one new draft every three minutes. Obviously, no serious changes could be made in such a short time period; in fact, most of those changes were simply substitutions of larger vocabulary words (found in a thesaurus) rather than the addition of any new content or support for the written argument. Given the benefit of Utah Write's scores and feedback, the built-in tutorials, the student-generated suggestions, and the in-class conversation/analysis, he should have been able to make more significant revisions that would have raised the score. In fact, the first "Rule" on the list of student-generated suggestions is "Revise! Don't just edit and resubmit." Small changes will not profoundly affect a score. Sometimes restructuring the entire essay is in order, and students should not feel tied to the five-paragraph format. Sometimes more logical evidence (i.e., more content!) is needed to support and warrant a claim. After the discussion and analysis, many students made big changes and rewrites, and they got significantly higher scores. Bob was mad that he couldn't raise his score by changing only the simplest and most obvious things. I know that Utah Write is not perfect, and it can't judge the effectiveness of an argument, but as a general rule, better writing gets better scores. Students who are willing to invest the time in legitimate revision, examine their writing carefully, and address the writing traits individually almost always see improved scores. In fact, with practice, most students learn to get an A on the first submission. And students in my classes will get lots of practice. Because I have tried to be critically meticulous in my use of this technology, I bristle when I receive comments like this one from Bob's mom:
"...it appears that the actual paper was only graded on Utah writes and not by a real person. I found his paper better than B-. B work. (unless in all his fiddling trying to get utah writes to give him a better score he screwed it all up). Either way I believe that Bob told me that he was told that to better the Utah writes score more actual words are needed to change the score - not substance. I have issue with this if I am understanding it correctly. I know how many hours my son worked on this and would have been nice to have some way to direct him to get a better Utah writes score if that is all he is graded on."
The first problem, as often happens, is that Bob's mom doesn't know the whole story. She knows what Bob told her, and she knows what score she sees when she checks Bob's grades, weeks after the assignment was submitted. While it is true that Utah Write was used in the drafting process for the paper in question, I gave the final score that appeared on Bob's midterm, and I agreed with Utah Write: It was a B paper at best. (Actually, I probably would have given it a C+, but since Utah Write had given it a better score, I gave Bob the benefit of the doubt.) Second, Bob's mom has no conception of how long students worked on the paper and how much class time was devoted to writing and revising it. It was a term paper involving research, and it took much of the term to complete. Most students spent not only hours in class working on it but also hours at home revising and resubmitting until they earned a score they could live with. As noted above, all of Bob's "revisions" were last-minute and, frankly, careless attempts to raise a score without making any significant changes to the paper. If you were to compare Bob's first draft to his final paper (another thing that Utah Write allows students to do, which I love!), you would see that as far as the validity of the argument, the quality of the evidence and research, the clarity of the warrants, and the effectiveness of the paper as a whole, the two drafts were identical. The fact is that Bob is not telling his mom the whole story...and she isn't asking to hear it. Third, her comment about more words (not substance) being needed is obviously how Bob interpreted for her the first suggestion on the list generated by the Honors classes. Notice: He made no mention of any of the the other ones, all of which would have improved his paper. Also, as noted previously, I graded the "substance," what little there was of it, not Utah Write. This brings us to her personal evaluation of the writing. She says she found it "better than B.B- work." Maybe she is a writing teacher, but based on the quality of writing in her e-mail, I am guessing not. Since I was the one who shepherded the students through a six-week writing process, complete with checklists, lab time, and lots of models, I think I am more qualified to assign the final grade. Also, I have the advantage of seeing the work of 240 of Bob's classmates, so I have a pretty good idea for how his writing compares. She says she knows how many hours her son worked on this paper. Actually, so do I. Remember: Utah Write shows the time on every change and allows students (and parents, if they so desire) to compare one to the next. Bob didn't spend anywhere near as much time as most of the class. She says it would have been nice to have some way to help him get a better score, at which point I have to excuse myself to go puke. Are you kidding?! Having provided a website full of resources, handouts for students with lists of revision suggestions, tutorials built into the program itself, more examples than I can count, and lots of time to work on it during class, I think this criticism is patently unfair.
Given this experience with Bob and a few others like it, I think most of the complaints about Utah Write originate in a grade-grubbing culture that values high GPAs over actual learning. “It doesn't matter if I become a better writer as long as I get an A.” The fact is that any paper I would give an A to would also get an A from Utah Write. Not only that, but in most cases Utah Write is far more lenient than I am. It's the process (of learning and self-discovery) that is important, not just meeting every requirement in order to get points. I know plenty of "A students" who don't learn very much, or, worse, never appreciate what they do learn because they don't believe that matters...as long as they get an A. I usually find that the learners who are truly engaged and challenging themselves get good grades without counting points and begging for extra credit. And they don’t blame a computer program for their own failings as writers.
I hope you won’t either.
Because it is a computer program and not a human being, Utah Write can't actually understand the content of a student’s writing. This is the basis of most of the complaints I have heard in recent years: “How can a computer grade writing?!” This is a perfectly legitimate and correct criticism, and in an ideal world, I would have plenty of time to read everything my students write, evaluate it completely, and offer timely and instructive feedback. But this is not an ideal world when it comes to the resources available for public education.
Every year of my 24 years in teaching, there have been more students in my classes than the year before. Last year, I had 241 English students, an average of just under 35 per class. If I were to personally offer constructive feedback on even a single-paragraph written by each of those students, it would take a week. The core curriculum calls not just for single paragraphs but for 1000+-word essays, many of them, over the course of the year. To personally score everything my students are required to write and give constructive feedback on all of it would take more time than there is in a school year and I would do little else, even after school and overnight. Furthermore, by the time the feedback got to the students whose papers were at the bottom of the pile, it would no longer be meaningful. (Even using Utah Write, I often have to resort to a system of evaluation codes and general suggestions to score term papers before the term ends). So, please understand that the big reason I started using Utah Write was because I wanted my students to have some kind of feedback on all their writing, which would be impossible given current (and, probably, future) class loads. If I want to have my students complete more than one writing assignment per term (which I do), there is no way it can happen if I have to personally score and give feedback on every assignment. Mind you, this isn't because I am shirking my responsibilities as a teacher (although I have had parents accuse me of that, some more subtly than others). Fact: There are too many students in the system to accommodate a personal touch on every writing evaluation. No human could read and score that much writing in a reasonable amount of time. So, I am left with a choice: I can have an admittedly imperfect computer program help me evaluate and give immediate feedback on the writing, or I can have the students write less often. Since I believe that students should write as much as possible, I have chosen to use technology to make that happen. Practice may not make perfect, but it beats no practice. Also, now that the state-mandated tests are using the same algorithms to evaluate students, it seems only fair that students get familiar with the program.
If your philosophical opposition to Utah Write is so strong that you don’t want to see it used in schools, please contact the state legislature and campaign for two things: reasonable class sizes and no high-stakes writing tests that are scored by computers. You will likely be told, as I have been, that the expense of these suggestions makes them impossible dreams, but perhaps you have more clout than I do. Until such a change is voted into reality, Utah Write is with us and it is going to be used to evaluate and score student writing in this class and on the final state-mandated exam.
Now, back to that complaint: “How can a computer grade writing?!” I know that this question is usually asked rhetorically (and angrily), but I would like to try to answer it. While Utah Write can’t actually “know” what a piece of writing is saying or evaluate the logic of the argument contained within it, the program DOES determine if verbs are used correctly, how long and varied the sentences are, the grade-level of the vocabulary, whether or not transition words and phrases are employed, if the writer depends on clichés, the degree of repetition, and whether or not punctuation marks are used correctly. Utah Write can also check grammar and spelling and look at the statistics of the writing (word count, paragraph length, etc.) just like Microsoft Word does. It is not perfect, but it is feedback...and it can generate that feedback to every student in less than one minute. This is why I use it. If Utah Write indicates that a student’s organization is flawed or his fluency needs work, at least it is a place for the student to start a revision. Utah Write is part of the process, not just the result. I generally find that students who object most strongly to the program are those who pay no attention to the process and don’t really focus on their writing until Utah Write has scored it poorly; they just hurry through a draft in a hopeful attempt to earn a good score with very little effort, much like many students who don’t read will make a desperate, last-minute attempt to write a book report that will fool a teacher into giving them points. For some students, that method might work part of the time…until I get around to judging the quality of the argument (i.e., the content). However, most of the time, Utah Write – by identifying the elements of good writing that it IS able to evaluate – gives a legitimate evaluation of most traits of a student’s writing. To the degree that it is possible, Utah Write recognizes good writing. I have used the program long enough to know that those who address all the elements of an effective argumentative essay will get a good score. Lazy or sloppy efforts will not. So, while it can’t make a judgment about the quality of logic used in a paper, Utah Write does a pretty good job with all the other writing traits.
Because I am fully aware of the limitations of Utah Write, I go to great pains during class to make sure that students have the benefit of every conceivable resource to help them improve their writing. I tell them from the beginning that Utah Write is imperfect in many ways, but they may revise and resubmit as many as 50 times to achieve a score they can live with, and they have more than a week between the time the rough draft is reviewed and the time the final draft is due. Furthermore, almost every step of prewriting and drafting can be completed during class time in the lab (if the time is used efficiently). While they are writing, I use a program called Net Support to watch students work – from my computer, I see what they are doing at their computers – so I have some idea where help is needed. Granted, because there are 35+ kids in every class, such help often comes in the form of general statements to the entire class when I notice, for example, that many of them are writing incomplete introductions or unclear thesis statements. Again, one-on-one help is rare when there are so many of them, but I try. First and foremost, I provide (on the website and/or in class) lots of good examples. For every writing assignment, I display and analyze a well-done example and encourage students to mimic the strategies of the successful writer. When appropriate, I run the good example through Utah Write so students can see how effective writing earns good scores even from an artificial intelligence. Sometimes I will also show and analyze non-examples, poorly done essays that earned low scores. In other words, I try to make it crystal clear to students what to do and what not to do in their own work. With few exceptions, those who pay attention and revise their own writing with care earn top marks both from Utah Write and from me.
To this end, I once had the Honors English classes do a comparative analysis of papers scored by Utah Write, and we looked specifically at how to raise a score. The results were surprising, and you can see them here. In developing this list, we learned that while Utah Write may not "know" what the writing is about in the way a human reader would, the algorithm is sophisticated enough to identify certain qualities of good writing and to differentiate between good writing and mediocre writing in many respects. In other words, Utah Write can provide helpful suggestions for improvement, and this list of student-generated observations is actually full of excellent suggestions for raising scores as well as making writing noticeably better even to the humans who read it. It might be possible to fool Utah Write into giving a high score to a lengthy piece of written nonsense, but that would be far more difficult than just writing a good paper to begin with and revising it carefully, following the suggestions offered by the program. (Did I mention that Utah Write has built-in writing tutorials that are focused on the student's individual writing weaknesses? Well, it does, and if students like the one I am about to introduce would bother to complete those tutorials, I don't think I would get any complaints.) In short, the suggestions developed by the Honors English classes support good writing in general, not just when using Utah Write, and they are always available to students who are working on a Utah Write assignment in my class.
Here is the whole story behind the most recent complaint I had about Utah Write. On April 29th, “Bob” submitted 15 "revisions" (i.e., hit the "save" or "submit" button) in less than an hour and a half. Many of those “revisions” included nothing more than correcting a single misspelled word or changing the spacing. These were not truly revisions but more editing and “polishing” changes. (Utah Write marks such errors in red, similar to what Microsoft Word does, so they are easy for students to identify, but unless there are hundreds of such errors in a paper, fixing only the redlined suggestions will not raise a score very much.) On May 8th, Bob submitted seven more “revisions” between 6:02 and 6:24 P.M. That’s about one new draft every three minutes. Obviously, no serious changes could be made in such a short time period; in fact, most of those changes were simply substitutions of larger vocabulary words (found in a thesaurus) rather than the addition of any new content or support for the written argument. Given the benefit of Utah Write's scores and feedback, the built-in tutorials, the student-generated suggestions, and the in-class conversation/analysis, he should have been able to make more significant revisions that would have raised the score. In fact, the first "Rule" on the list of student-generated suggestions is "Revise! Don't just edit and resubmit." Small changes will not profoundly affect a score. Sometimes restructuring the entire essay is in order, and students should not feel tied to the five-paragraph format. Sometimes more logical evidence (i.e., more content!) is needed to support and warrant a claim. After the discussion and analysis, many students made big changes and rewrites, and they got significantly higher scores. Bob was mad that he couldn't raise his score by changing only the simplest and most obvious things. I know that Utah Write is not perfect, and it can't judge the effectiveness of an argument, but as a general rule, better writing gets better scores. Students who are willing to invest the time in legitimate revision, examine their writing carefully, and address the writing traits individually almost always see improved scores. In fact, with practice, most students learn to get an A on the first submission. And students in my classes will get lots of practice. Because I have tried to be critically meticulous in my use of this technology, I bristle when I receive comments like this one from Bob's mom:
"...it appears that the actual paper was only graded on Utah writes and not by a real person. I found his paper better than B-. B work. (unless in all his fiddling trying to get utah writes to give him a better score he screwed it all up). Either way I believe that Bob told me that he was told that to better the Utah writes score more actual words are needed to change the score - not substance. I have issue with this if I am understanding it correctly. I know how many hours my son worked on this and would have been nice to have some way to direct him to get a better Utah writes score if that is all he is graded on."
The first problem, as often happens, is that Bob's mom doesn't know the whole story. She knows what Bob told her, and she knows what score she sees when she checks Bob's grades, weeks after the assignment was submitted. While it is true that Utah Write was used in the drafting process for the paper in question, I gave the final score that appeared on Bob's midterm, and I agreed with Utah Write: It was a B paper at best. (Actually, I probably would have given it a C+, but since Utah Write had given it a better score, I gave Bob the benefit of the doubt.) Second, Bob's mom has no conception of how long students worked on the paper and how much class time was devoted to writing and revising it. It was a term paper involving research, and it took much of the term to complete. Most students spent not only hours in class working on it but also hours at home revising and resubmitting until they earned a score they could live with. As noted above, all of Bob's "revisions" were last-minute and, frankly, careless attempts to raise a score without making any significant changes to the paper. If you were to compare Bob's first draft to his final paper (another thing that Utah Write allows students to do, which I love!), you would see that as far as the validity of the argument, the quality of the evidence and research, the clarity of the warrants, and the effectiveness of the paper as a whole, the two drafts were identical. The fact is that Bob is not telling his mom the whole story...and she isn't asking to hear it. Third, her comment about more words (not substance) being needed is obviously how Bob interpreted for her the first suggestion on the list generated by the Honors classes. Notice: He made no mention of any of the the other ones, all of which would have improved his paper. Also, as noted previously, I graded the "substance," what little there was of it, not Utah Write. This brings us to her personal evaluation of the writing. She says she found it "better than B.B- work." Maybe she is a writing teacher, but based on the quality of writing in her e-mail, I am guessing not. Since I was the one who shepherded the students through a six-week writing process, complete with checklists, lab time, and lots of models, I think I am more qualified to assign the final grade. Also, I have the advantage of seeing the work of 240 of Bob's classmates, so I have a pretty good idea for how his writing compares. She says she knows how many hours her son worked on this paper. Actually, so do I. Remember: Utah Write shows the time on every change and allows students (and parents, if they so desire) to compare one to the next. Bob didn't spend anywhere near as much time as most of the class. She says it would have been nice to have some way to help him get a better score, at which point I have to excuse myself to go puke. Are you kidding?! Having provided a website full of resources, handouts for students with lists of revision suggestions, tutorials built into the program itself, more examples than I can count, and lots of time to work on it during class, I think this criticism is patently unfair.
Given this experience with Bob and a few others like it, I think most of the complaints about Utah Write originate in a grade-grubbing culture that values high GPAs over actual learning. “It doesn't matter if I become a better writer as long as I get an A.” The fact is that any paper I would give an A to would also get an A from Utah Write. Not only that, but in most cases Utah Write is far more lenient than I am. It's the process (of learning and self-discovery) that is important, not just meeting every requirement in order to get points. I know plenty of "A students" who don't learn very much, or, worse, never appreciate what they do learn because they don't believe that matters...as long as they get an A. I usually find that the learners who are truly engaged and challenging themselves get good grades without counting points and begging for extra credit. And they don’t blame a computer program for their own failings as writers.
I hope you won’t either.
posted by MRT at 8:00 AM
0 Comments:
Post a Comment
<< Home