Nuts&Bolts Jan. 2003

 nuts 
& 
bolts

Dec.-Jan. 2003

Vol. 3 No. 5

How do I assess thee? In plain English, please

"How do I assess thee? Let me count the ways," Elizabeth Barrett Browning might ask, if she were alive today and teaching poetry instead of writing it. As a poet, however, she might not recognize some of the answers as being in a language she understood. As educators, we don't always write like, well, like poets. Or like native speakers of English, for that matter.

When it comes to the literature on assessment, there's something else at work, too. Many of us first developed our interest in it at the suggestion of the folks who draft and interpret federal government regulations, another group not always praised for its crisp, clear language. And some of the key concepts derive from TQM (total quality management), one of the more arcane theories of industrial organization. The result: Alphabet soup.

As you revise your syllabi for spring semester, please remember our first CAAP assessment day at SCI is Wednesday, April 2. Assessment testing is now mandatory for all our students, and the CAAP test is a requirement for graduation.

But there's an article by Andrea Leskes, vice president for education and quality initiatives of the Association of American Colleges and Universities, that helps us (if I'm not straining the metaphor) to strain the soup. It's on line in Peer Review, an AAC&U quarterly, and SCI head librarian Susan Full found a copy on a higher education Listserv. It's titled "Beyond Confusion: An Assessment Glossary," and the title suggests its usefulness.

Here are a few of Leske's definitions, starting with "value added" assessment, which is the theme of the Peer Review issue in which her article appears. There are a lot of parentheses, by the way; they're all Leske's:

Value added: "the increase in learning that occurs during a course, program, or undergraduate education. Can either focus on the individual student (how much better a student can write, for example, at the end than at the beginning) or on a cohort of students (whether senior papers demonstrate more sophisticated writing skills - in the aggregate - than freshmen papers). Requires a baseline measurement." Pre- and post-tests are one way of getting at it.
Assessment for improvement: "assessment that feeds directly, and often immediately, back into revising the course, program or institution to improve student learning results." At SCI we are working with a concept, adapted from the TQM literature, known as assessment for continuing improvement with learning outcomes data fed back into our planning processes.
Assessment of individuals: "uses the individual student, and his/her learning, as the level of analysis. ... Would need to be aggregated if used for accountability purposes. Examples: improvement in student knowledge of a subject during a single course; improved ability of a student to build cogent arguments over the course of an undergraduate career." Grades, obviously, are part of this. But we need to use multiple measures to double-check our grades. And we need to study them in the aggragate, to see what patterns might exist and how they reflect our mission in an overall sense.
Assessment of programs: "uses the department or program as the level of analysis. ... Ideally program goals and objectives would serve as a basis for the assessment. Example: how sophisticated a close reading of texts senior English majors can accomplish (if used to determine value added, would be compared to the ability of newly declared majors)." The question we're wrestling with on the Assessment Committee: How do we do this at SCI? In which programs? When? Who does it?
Assessment of institutions: "uses the institution as the level of analysis. ... Ideally institution-wide goals and objectives would serve as a basis for the assessment." Here's where the mission statement comes into play. What do we do specifically to realize its broad objectives? Not only teaching but the entire range of student services needs to be assessed, since learning depends on a wider matrix of activity than the classroom alone.
Embedded assessment: "a means of gathering information about student learning that is built into and a natural part of the teaching-learning process. Often uses for assessment purposes classroom assignments that are evaluated to assign students a grade. Can assess individual student performance or aggregate the information to provide information about the course or program. ... Example: as part of a course, expecting each senior to complete a research paper that is graded for content and style, but is also assessed for advanced ability to locate and evaluate Web-based information (as part of a college-wide outcome to demonstrate information literacy)."

Leske's article defines other useful terms, including the difference between qualitative and quantitative measurements, and the "high-stakes" assessment we read about in the papers that uses standardized testing to "set a hurdle" for promotion or graduation in the public schools. We don't do it at SCI, but it's good to know the difference between what we do and the testing that has created controversy in states like Massachusetts and Texas. The whole article is a good primer on higher ed assessment.

Says Leske, "College professors regularly employ with comfort some types of assessment; they rightly point this out when challenged, explaining how they always evaluate student learning, using tests or homework assignments to do so. Assessment of this sort normally occurs within the confines of individual courses and can provide important information to both the student and the professor. However, higher education has less of a history of examining accomplishments that build cumulatively, over time, throughout a student's entire undergraduate career. Yet we acknowledge that many of the goals of college education are exactly these accomplishments (e.g., effective communication, ethical judgement, analytical acuity)." That's a large order, but we're making a start at SCI. Articles like Leske's help us better understand the context for what we're doing.

Links and sources. Leskes's article is available on line in the Winter/Spring 2002 issue of Peer Review at http://www.aacu.org/peerreview/pr-sp02/pr-sp02contents.cfm. The entire issue centers on a theme of value-added assessment, a term that is developed most fully by the national Value Added Assessment Initiative (VAAI) coordinated by the RAND Corporation's Council for Aid to Education (RAND-CAE). If it whets your appetite for more poetry, Elizabeth Barrett Browning's poem "How Do I Love Thee?" is available on line, along with other links and information, at the Academy of American Poetry's website at http://www.poets.org/poets/poets.cfm?prmID=153. If you're reading this in hard copy, you can get to the December-January Nuts & Bolts (and the links above) by clicking on "Policies & Procedures" in the Resources menu at www.sci.edu; the link to SCI's learning outcomes assessment website is at the bottom of the page.

What should students know about the CAAP test?

On Wednesday, April 2, all SCI sophomores will be required to take the reading component of the Collegiate Assessment of Academic Proficiency (CAAP) test as part of SCI's General Education assessment plan (Gen Ed testing is one of three parts of our overall assessment plan). It is required for graduation, but it is not "high-stakes" (please see above) in the sense that students need to get above a cut-off score in order to graduate. I ordinarily ask my students to do their best on the assessment measures I give them, because that way they can help me become a better teacher. And that's about what I plan to tell my sophomores before April 2. Assessment Committee members Kevin Broeckling, Susan Full and Kathleen Killion are working on other motivators, and you'll be hearing from us again soon.

The CAAP reading test lasts 40 minutes and measures learning in two areas: (1) reading skills, including the ability to "recognize main ideas of paragraphs and passages, to identify important factual information, and to identify relationships among different components of textual information;" and (2) reasoning skills, including "ability to determine meaning from context, to infer main ideas and relationships, to generalize and apply information beyond the immediate context, to draw appropriate conclusions, and to make appropriate comparisons." Readings are drawn from prose fiction, humanities, social studies and natural science. Visit http://www.act.org/caap/ for sample questions and more information.

-- Pete Ellertsen, editor, Nuts & Bolts

Nuts & Bolts is an electronic newsletter published by the Assessment Committee of Springfield College in Illinois.

Members of the Assessment Committee are: Bob Blankenberger; Susan Full; Alice Gutierrez; Kathleen Killion; Penny Leonhard; Scott McCullar; Steve Stowers; Jeff Mueller and Kevin Broeckling (ex officio); and Pete Ellertsen, chair. If you have information, comments or feedback, please contact any committee member or editor Pete Ellertsen, in Becker L-16A on the SCI campus, 525-1420 ext. 519 or by e-mail at ellertsen@sci.edu.