nuts 
&
bolts

First of a 2-part series

We've got results back from the CAAP (Collegiate Assessment of Academic Proficiency) reading test now, and SCI continues to look like Minnesota Public Radio host Garrison Keillor's fictional town of Lake Wobegon, where "all the women are strong, the men are good-looking and the children are above average." Well, make that about average. SCI students' scores on the norm-referenced test were ever so slightly above average last year, and slightly below average this year. Statistically, it's a wash.

Here are the overall scores, expressed in terms of cumulative percentages for SCI students taking the test (n = 66 students last year and 69 this year) and the nationally normed reference group of all second-year students in two-year colleges (n = 24,701 last year and 26,647 this year) taking the reading module of the CAAP test nationwide:

Since our data are based on a small population, the differences are statistically insignificant. CAAP reading module scores are broken out into subscores for arts and literature and for social science. Here are the arts/lit subscores for SCI students and the national norm, based on 18 items in prose fiction and humanities, and expressed in terms of a 25-point scale:

The arts and literature subscore differences are not statistically significant, either. The other subscores, for social and natural sciences, are based on 18 items in "the Social Studies and Natural Science." The scores are:

Again, the differences are too small to be statistically significant. In fact, at SCI we have to be very careful in interpreting all of our statistical data because the universe - our total population of students - we're measuring is small. Let's put it this way: When you've only got 60-70 students taking a test, something as inconsequential as a beer party the night before could skew the results. So, for that matter, could a clanking radiator in the testing room or an ill-timed freight train blocking 5th and 6th streets at North Grand Avenue. But when the scores are complied over two years, you have larger numbers and better data to work with.

ACT Inc., which produces the CAAP tests, suggests: "Generally, the larger the sample size, the more useful the data. ACT recommends a minimum of 100 students tested for each module used." That means, finally, we've got a large enough set of scores - 135 over a two-year period, to be exact - to give us a set of statistics we can use as a baseline.

So now we've got the stats, what do they tell us? Not much, would be the most honest answer. Not yet. They're baseline figures, and we don't have a whole lot to compare them to. But the consistency of our CAAP test scores so far is still important - it suggests we're basically on the right track.

"We now have a system in place for a national normative test," says Dean of the College Jeff Mueller. ""It's the first hard evidence we've had of our students' reading levels."

And taken together the test scores would tend to confirm that our students read at about the same level as their peers nationwide. But qualtitative data such as CAAP scores, like any other assessment measure, have to be evaluated in terms of other indices of student learning.

"[The CAAP test] is not our sole method of assessment," Mueller added. "We are putting multiple measures in place. We are also expecting teachers to do assessessment in the classroom, and we are working diligently with program assessment in various curricula."

Here's why: No one measure tells us all we need to know. Even ACT Inc. warns us: "Scores on students' score reports provide one way to estimate their level of educational development in college; the measurement of educational development is complex, however, and no single indicator should be considered as definitive proof of what students have learned." Hence the need for multiple measures.

A reminder of where standardized test scores fit into assessment comes in this month's issue of The Illinois School Board Journal. In an address to the National School Boards Association reprinted in the Journal, consultant Debbie Silver suggests quantitative test scores have been blown out of proportion by policymakers who have sold the public on "the idea that we only need to assess what is easily measured (standardized objective tests covering isolated skills and facts) rather than examining more deeply what is actually going on with our students." While she spoke of K-12 assessment, the basic principles apply to higher education as well.

Silver doesn't discount standardized test scores entirely. She just wants to see them evaluated in perspective. "When I supervise teachers' classrooms, I look at their students' test scores along with their students' products as a piece of the overall picture," she said. "I watch the teacher interact with the students, and I note how students interact with each other. I ask to look at written notes from the teacher to the students as well as the students' notes to their teacher (or Valentines, pictures, cards, etc.). I interview the students and ask them what it feels like to be in that class.

"Only by examining both quantitative data and qualitative information am I able to tell whether good teaching is happening or not. I wish that all classroom evaluators would do the same. It's not just about the scores."

I can't put it any better than that. In the end, assessment isn't about the scores - it's about what we do with the scores.

Next: What the CAAP test can tell us about our General Education program.

-- Pete Ellertsen, editor, Nuts & Bolts

Silver, Debbie. "More to Teaching than Tests Can Tell." Illinois School Board Journal May-June 2004: 37-39.

Nuts & Bolts is an electronic newsletter published by the Assessment Committee of Springfield College in Illinois. Members are: Moses Allen, Bob Blankenberger, Alice Gutierrez, Scott McCullar, Dave Saner, Steve Stowers, Barb Tanzyus and Pete Ellertsen; Kevin Broeckling, dean of students, and Jeff Mueller, dean of the college, serve ex officio.

• link here to back issues of Nuts & Bolts
• link here to SCI's assessment website

If you have information, comments or feedback, please contact any committee member or Nuts & Bolts editor Pete Ellertsen, in Becker L-16A on the SCI campus, 525-1420 ext. 519 or by e-mail at <ellertsen@sci.edu>.