Editor’s note. The articles below appear substantially as written. You will find several unavoidable references to “outcomes.” Like learning objectives, they refer to observable behaviors we can assess for. We are now calling them “objectives” in order to comply with external mandates. Don’t let the terminology confuse you.
You are capable, competent, creative, careful. Prove it ("Fortune").
SEPTEMBER 2004. This year SCI's Assessment Committee is going to put a new emphasis on faculty development, the kind of nuts-and-bolts advice this newsletter promises. But I'll admit it: I didn't get my inspiration from the professional literature on assessment. I got it in large part from a Chinese fortune cookie.
At the very beginning of fall semester, before we could buy lunch in what used to be known as the Quiet Lounge (should we emulate the rock artist and call it "the Lounge Formerly Known as Quiet?"), I got some Chinese take-out from one of the restaurants on North Grand Avenue and with it the fortune cookie quoted above. Typically, fortune cookies don't promise me wealth and happiness; instead, they warn me to shape up, get back to work and try harder. So this one, I liked. Besides, it reminded me of our work on the Assessment Committee. We're good, and our students are learning. So let's prove it. We're required to, anyway, so let's do a good job of it. And let's do it in a way that actually helps us in the classroom.
That was part of my inspiration. The other inspiration came when former Academic Affairs Dean Eldon Brown visited campus a couple of weeks ago. I remember Eldon drafted me for the Assessment Committee in 2000 because assessment is basically a government mandate, and I have a background in political journalism and public relations. Well, I'd waded through a lot of government gobbledygook in newspapering days, but I was dumbstruck when I started dealing with pedagogical gobbledygook. All right, OK, I thought, I understand I'm required to do this for accreditation, but what in blue blazes do they want me to do? In a word, what I needed most was a good stiff dose of faculty development.
We've got a lot of new faculty this year. Just glancing at mailboxes in Dawson Hall, I count at least a dozen over the last year or so in the traditional program. And almost two dozen more in New Horizons and the Benedictine University programs. So it's a good time to go back to some of the basics.
We'll begin with a brief session during October's faculty meeting. Southern Illinois University at Edwardsville has an excellent introduction to Classroom Assessment Techniques (CATs for short) linked below. We'll take a look at it. And in this newsletter, which is named Nuts & Bolts after all, we'll go into some of the nuts and bolts of specific CATs as the new school year goes along.
More important than specific techniques, though, is the underlying attitude. Why do we do assessment? Well, for one thing, we're required to by outside stakeholders like government and accrediting bodies. But that's not the real payoff for us as educators. "First and historically, assessment is what we faculty members can do in order to demonstrate to ourselves that we actually do what we say we do," says Douglas Eder of SIUE. "It is our source of in-process feedback." By assessing what our students actually learn, Eder explains, we can analyze "the curriculum (or an assignment, class, or course) into component parts." By doing it as we go along, we can see where we're going with a class and change course as needed.
We'll have more to say about specific techniques in later issues of the newsletter. In the meantime, here's one that gets a lot of favorable mention on our faculty assessment questionnaires. It's called the "one-minute essay," and it's more informative and less cumbersome than some of the other CATs.
Here's the minute essay prompt they use at SIUE, with some variations they suggest in brackets: "In concise, well-planned sentences, please answer the two questions below: 1. What are the two [three, four, five] most significant [central, useful, meaningful, surprising, disturbing] things you have learned during this session? 2. What question(s) remain uppermost in your mind?" I'll read the questions at the end of class, and have students write their answers. I've also used the questions in end-of-semester evaluations.
Here's what I like about one-minute essays: They allow me to compare what I think my main point was with what my students think it was. Let's just say it's not always the same!
I've been talking about classroom techniques here, but assessment is not just for teachers. "In both student and academic affairs," says Eder, "I believe assessment should ask: Who came, what did they learn, what are their impressions, and what subsequent behavior has resulted?" Eder commended Miami University of Ohio, which assesses "such questions as, why do you go to class, why do you drop out, where do you go for help, what did you come in with, how do you learn best, how do your parents feel, how do you spend your time, who is NOT here, how do minority students fare, how do at-risk students fare, are mid-term warnings effective, and what connections do you see between general education and the world beyond the classroom?"
If you think about it, that's a lot like what the fortune cookie said. We're capable, competent, creative and careful at SCI. Let's prove it.
Eder, Douglas. "Assessment, Thermodynamics, and the Myth of Sisyphus." Keynote Address, Virginia Assessment Group, Nov. 6, 1998. http://www.siue.edu/~deder/assess/vagkey.html
Fortune. Chicago: Sam's Fortune Cookies Inc., n.d.
Southern Illinois University-Edwardsville. Classroom Assessment Techniques. 2004. http://www.siue.edu/~deder/assess/catmain.html
Mission, goals, IAI drive classroom assessment
OCTOBER 2004. At the first of the month I was invited to speak on classroom assessment techniques (CATs for short) to a faculty meeting at Ursuline Academy. As so often happens, I came away from the session with a better understanding than I had going in. Partly it proves the old saying you don't really learn a subject till you try to teach it. But partly I think it has something to do with common values and a shared mission. If you get a group of teachers together in a room, sooner or later the conversation is going to turn to what's best for the kids.
When I was asked to speak to Ursuline's faculty, I wasn't sure what I could say to a group of high school teachers. K-12 school reform has been politicized and crippled by unfunded government mandates to an extent we can't imagine -- so far -- at the college level. But, as it turned out, we had a lively, informative exchange. They brought laptop computers; we visited a Southern Illinois University-Edwardsville website that has a lot of good information about CATs (I'll be sharing some of it at an SCI faculty meeting this month); and we talked about what works, and what doesn't work, in the classroom. It turned out we had a lot in common: What K-12 teachers do with lesson plans and learning standards handed down by the State Board of Education, we're learning to do in higher ed with detailed syllabi and competencies set by the Illinois Articulation Initiative. I learned a lot.
Again, I think it gets back to our mission as educators. It's not by accident that good assessment practice begins with an institution's mission statement, goals and values. The American Association for Higher Education says: "Assessment is a goal-oriented process. It entails comparing educational performance with educational purposes and expectations -- those derived from the institution's mission, from faculty intentions in program and course design, and from knowledge of students' own goals" ("Nine Principles"). In practice, we use all of these things in deciding what we need to do in our classroom and what methods we'll use to assess how well we did it.
Some of our senior faculty at SCI like to base lessons on the "Goals of the College" available on line in SCI's assessment website. Especially useful is the goal of: "Preparing individuals to live full, satisfying, and responsible lives in the complexities of the modern world by developing skills in communication, computation, and problem solving; by developing powers of reflection and critical judgment, and by developing a global awareness of historical and current events." We like it because it gives us specific, measurable goals we can carry into the classroom. In conjunction with the course-specific outcomes mandated by the Illinois Articulation Initiative, we can translate it into measurable learning outcomes -- the specific things our students need to learn, like the Modern Language Association form for an in-text citation, what caused the War of the League of Augsburg or how the integration of computer hardware, software and peripherals help create and combine artistic and design ideas. So all right, already, you're thinking now ... but what do I do?
[Editor’s note: The above passage was written before adoption of SCI’s Common Student Learning Objectives. We can now use the Common SLOs to develop specific measurable things we want our students to learn about the League of Augsburg, the MLA stylebook or whatever we’re teaching. The principle and the procedure are the same.]
I'd start by looking back over the syllabus, not just the schedule of assignments but especially the goals, objectives and desired outcomes for the course.
Then I would take a good, hard look at the IAI's website -- called iTransfer and available on line at http://www.itransfer.org/.
At SCI we've been bringing our syllabi into line with IAI for several years now, and it's an important part of our assessment plan. (IAI, as you probably know, is an agreement "identifying common curriculum requirements across associate and baccalaureate degrees" at Illinois colleges and universities. There's more about it in the SCI catalog [53-55], and a lot more at iTransfer.) Our emphasis on IAI means our courses are designed to give our students what they need to know in order to transfer successfully to Benedictine University or another four-year school. So it's important.
Once you're on the iTransfer home page, you can find the course you teach by going to the pull-down menu for faculty and advisers. Click on "IAI course descriptions," and you will go to a webpage with links to GECC (General Education Core Curriculum) and IAI Majors. To find your course, match the IAI prefix on your syllabus (or in our catalog) with the corresponding prefix on the iTransfer course descriptions page. For example, English 111 carries the designation IAI C1 900 at the end of our catalog entry. Click on the link under "IAI GECC" for "Communication (C)" core courses, and you'll find it. Other courses you'll find under "IAI Majors." My introductory mass communications course, for example, is designated IAI MC 911, and I can get to it by clicking on the "Mass Communication (MC)" link.
Almost all of the IAI descriptions include a statement along these lines: "Upon successful completion of the course, the student will be able to ..." Find it. When you do, study that list of things your students will be able to do on completing the course. Those are the learning outcomes you want to: (1) adapt as needed to conform to our statements of mission, goals and values; (2) teach; (3) use assessment techniques to measure how well your students mastered them; and (4) document the changes you make in your teaching as a result of doing the assessment. That's the part of assessment that makes the whole system work: It makes us better classroom teachers.
OK, OK, you're probably still saying, so now I've read IAI, so what do I do now? But this month's issue of Nuts & Bolts is getting much too long to read at one sitting, so I'd better wind it up now. I hate to leave it at that. I really do. It's too much like the old movies where the heroine was left tied to the railroad tracks at the end of each episode. But, wait, we've got a faculty meeting coming up.
I hope to see you all Oct. 28 at the faculty meeting. We won't screen The Perils of Pauline, but we will talk briefly about where CATs fit into our overall assessment and reaccrediation picture at SCI so you'll know what we're all expected to do. And I'll introduce you to a website with some very good, practical information about classroom assessment.
‘Ready, aim ...’: 1-minute papers help focus
NOVEMBER 2004. Here's a technique I wish I'd known about 10 years ago ...
Like so many college instructors, I never studied teaching methods. (In fact, in grad school I had what I now realize was a snotty attitude about education majors.) So when I came to SCI as an adjunct instructor in 1993, I had no idea what I was doing. About the only thing I had going for me was I've always been a sucker for movies like Up the Down Staircase, the 1967 classic starring Sandy Dennis, and books about idealistic young teachers in beleaguered city schools.
So while I was floundering around wondering how I could possibly fill a 50-minute class period, I noticed teachers in New York had something they called an "Aim" for each day's lesson. "When Jessica Siegel writes the word 'Aim' on the blackboard," I read in one inner-city classroom saga, "the woman in the far left corner [a beginning teacher] shudders ... because 'Aim' is the word she saw in her nightmare. The word was floating in the air, daring her, mocking her, challenging her, for it is the word that by Board of Education edict must begin every single lesson in every single school in New York City" (Freedman 168). Typically, the Aim is stated as a question: "How Does 'Passage from Africa' Reveal How Racial Attitudes Are Established?" in a high school English class or "How do we round large numbers?" in eighth-grade remedial math (Freedman 171; Sachar 17). Feeling a little mocked and challenged myself as a beginner, I latched onto the idea for dear life. It helped me focus, and it helped me stay on track.
Turns out there's a Classroom Assessment Technique that does the same thing. It's called a One-Minute Paper, or a Minute Paper, and it's often considered the most helpful CAT of all.
Here's how it works: At the end of class, you take a minute or two. Hence the name. Ask the students two questions: (1) What was the most important thing you learned during this class? (2) What important question remains unanswered? Let the students write their answers, take up their papers, analyze them and report back to the class what you learned from the exercise. That last step, reporting back is important. It completes the loop, and over time it helps the students realize learning is a two-way street and they're responsible for what and how they learn.
According to assessment gurus Thomas Angelo and Patricia Cross, "no other Classroom Assessment Technique has been used more often or by more college teachers" (148). It's not hard to see why.
"The great advantage of Minute Papers is that they provide manageable amounts of timely and useful feedback for a minimal investment of time and energy," explain Angelo and Cross. "By asking students what they see as the most significant things they are learning, and what their major questions are, faculty can quickly check how well those students are learning what they are teaching. That feedback can help teachers decide whether any mid-course corrections or changes are needed and, if so, what kinds of instructional adjustments to make. Getting the instructor's feedback on their Minute Papers helps students learn how experts in a given discipline distinguish the major points from the details. The Minute Paper also ensures that students' questions will be raised, and in many cases answered, in time to facilitate further learning."
And, as the Southern Illinois University-Edwardsville assessment website accurately points out, "[i]t really does take about a minute."
Angelo and Cross have a seven-step procedure for using the Minute Paper. SCI faculty who want to bone up on all seven can find them in Classroom Assessment Techniques: A Handbook for College Teachers (151-52). It's in Becker Library, and the call number is 378.125 A584. Given my history of floundering around in the classroom and getting my pedagogical inspiration from movies, I especially like their first step: "Decide what you want to focus on ..." Using the Minute Paper helps me do that, and going back to it helps me stay on track.
Pre-/post-tests and classroom assessment ...
DECEMBER 2004. [...] As I've said before, I didn't go to education school, and I felt like I was in the earliest -- and shakiest -- stages of on-the-job training when I started drafting assessment plans in 2000. Nor could I honestly see how classroom assessment answered any larger purpose than confirming my in-class hunches about why my students looked bewildered at times. But I see more and more now how it all fits together.
Pre- and post testing. One of the buzzwords in education these days is "value-added assessment." In plain English, it means looking for evidence our students know more when they finish a course than they did before they took it. One way to do that is to give a test at the beginning of school and give the same test again at the end. Hence the name pre- and post-testing.
We've talked a lot about pre- and post-testing at the classroom level over the years at SCI, but I haven't been able to find much information about it on the Internet. Keyword searches on "pre," "post," "test," "classroom assessment techniques" and "CATs" got me a lot of hits on institutional assessment, not to mention pre- and post-operative procedures for small animal veterinarians, but nothing about using pre- and post-tests in the classroom. Nor is the subject mentioned in Thomas A. Angelo and K. Patricia Cross, Classroom Assessment Techniques (1993), the standard reference on CATs. I did, however, find a brief discussion in an outcomes assessment manual put out by the Office of the Provost for the University of Wisconsin.
"Pre-test/post test assessment is a method used by academic units where locally developed tests and examinations are administered at the beginning and at the end of courses or academic programs," the UW Provost's Office says. "These test results enable faculty to monitor student progression and learning throughout prescribed periods of time. The results are often useful for determining where skills and knowledge deficiencies exist and most frequently develop. Academic departments at other research institutions currently using this form of assessment to measure student learning include communications, economics, geography, linguistics, theatre, and dance."
So there you have it. I've never had much luck with pre- and post-tests myself, and I suspect they work better for institutional assessment and the evaluation of programs at Research I universities like Wisconsin than they do in my classroom. But there are excellent teachers who swear by them.
And their experimental design is elegant in its simplicity: (1) Give a test; (2) give the same test again; and (3) compare the scores. Here's how I'd do it if I were using the technique, by way of example, in my introductory mass communications class. I would start with the goals, objectives and outcomes, which include: "To articulate the complexity of practices in the industries that make up the media, as well as the trend toward concentration of ownership." Like many freshman-level courses, my masscom survey is partly an introduction to the jargon of the field. So I would choose several key terms and ask students to define them. I might ask them the definition of "media conglomerate," for example, since it relates to the quoted objective. Or I might ask them to identify Rupert Murdoch or News Corp. (his conglomerate). For other questions, I would consult other goals, objectives and outcomes. Whatever questions I asked, I'd ask them twice -- once in August as the semester begins and once in December as it ends. If the students did better on the post-test, I could chalk it up to value added by my teaching. If they didn't, I would have something to work on in my next cycle of continuous improvement!
CATs: Formative feedback a 2-way street
January 2005. As I've been writing this series and reflecting on my own teaching, I've mentioned several inner-city teachers whose writing inspired me when I returned to the classroom 10 years ago. Now as the series is coming to an end, I'd like to share some wisdom from Jaime Escalante, the California math teacher who was featured in the 1988 movie Stand and Deliver. In 2002, he told Government Technology magazine how he got low-income black and Latino high schoolers in East Los Angeles to learn advanced calculus.
"You have to have the [cognitive] domain of the subject," he said, and you have to know how to get students to relate to that knowledge. He added, "Third, you have to understand human relations. You have to look at the kid as a person, and you respect the kid. And that way you motivate them. And you develop that gradually over a whole semester or two weeks or three weeks, that good relationship. And if you do that, when you have the feedback from the student mathematically speaking, then the kid speaks back, and you know he is learning."
So what teaching boils down to, Escalante suggested, is communication. And communication is a two-way street. So it is with classroom assessment. It tends to be "formative," to use a $13.95 word favored by educators. Here's a little background and a translation: Assessment comes in two flavors, (1) summative; and (2) formative. The provost's office at Central Michigan University, which has an excellent electronic "toolkit" on assessment, defines summative assessment by noting that it "is comprehensive in nature, provides accountability and is used to check the level of learning at the end of the program." On the other hand, formative assessment is defined by Carol Boston of the Center for the Study of Assessment Validity and Evaluation at the University of Maryland as the "diagnostic use of assessment to provide feedback to teachers and students over the course of instruction."
Boston says as the feedback goes from teacher to students, and back from students to teacher, it helps everybody. "Feedback given as part of formative assessment helps learners become aware of any gaps that exist between their desired goal and their current knowledge, understanding, or skill and guides them through actions necessary to obtain the goal." She adds what all classroom teachers soon learn, detailed comment on student work "encourages students to focus their attention thoughtfully on the task rather than on simply getting the right answer."
Of the eight classroom assessment techniques we're tracking at SCI, seven are wholly formative in nature. And the eighth, pre- and post-testing, can be used formatively when teachers use pre-test results as a prior knowledge inventory or background knowledge probe and plan lessons accordingly. (Southern Illinois University-Edwardsville's classroom assessment website explains how to do a background knowledge probe.) The rest of the CATs we're tracking boil down to asking students to think about what they're learning and share their thoughts with the instructor. To that extent, they can be considered as variations on the "minute essay" discussed in the November issue of Nuts & Bolts. They are:
For teachers, the benefits of formative assessment are obvious. We learn what we can do better while we still have time to do it. But Boston says it works the other way, too, when students are invited to give feedback on how well they are meeting the goals and objectives of a course.
"While feedback generally originates from a teacher, learners can also play an important role in formative assessment through self-evaluation," Boston says, citing research that suggests "students who understand the learning objectives and assessment criteria and have opportunities to reflect on their work show greater improvement than those who do not."
So, in the end, it all comes back where it began ... with goals, objectives and outcomes.