|
|
|
|
|
|
|
|
Last of a 2-part series
Editor's note: SCI students' standardized reading test scores were reported in the April issue of Nuts & Bolts. This month's issue continues the discussion.
Last month we got back results on the CAAP (Collegiate Assessment of Academic Proficiency) reading test we buy from ACT Inc. They show that our second-year students read about as well as second-year students in two-year colleges nationwide. While that may not seem like a remarkable conclusion in and of itself, the CAAP numbers are the first hard evidence we've had to back up that hunch. That in itself is important.
Table 1 (repeated from last month's Nuts & Bolts for the sake of convenience) gives the overall scores for both years, expressed in percentages ranking of the total test-taking population nationwide (n = 24,701 last year and 26,647 this year). These percentages are as follows:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
We also have subscores for questions relating to arts, letters and humanities and to behavioral and physical sciences (which were broken out in last month's newsletter). The scores may not mean much by themselves, but as we look at similar rankings over time - say over the next three to five years - we should be able to tell whether we're getting better, getting worse or holding steady.
There are other data we can purchase, too, from ACT Inc., if we decide they would yield enough information to justify the added expense, including a "Longitudinal or Linkage Report" comparing CAAP scores to such tests as the ACT tests now required of high school students in Illinois or other ACT Inc. tests that we might choose to administer to our own students in the future. More immediately, classroom teachers can look at the CAAP results and use them to plan for the coming school year in our own classrooms.
In setting up the General Education testing program, the Assessment Committee chose to go with the reading test because it measured a skill that is basic to all of our programs and curricula. ACT Inc. says the 40-minute test is made up of four passages, one each in:
The test covers not only reading comprehension but also such higher-order cognitive skills as the ability "to generalize and apply information beyond the immediate context, to draw appropriate conclusions, and to make appropriate comparisons." No other CAAP module, in the committee's judgment, covered as much material across the Gen Ed curriculum.
Convincing students to get serious about assessment measures, which by definition do not go into grade-point averages, is a nationwide problem. At SCI we have elected to work with our students and rely on voluntary compliance. In 2003, we had 66 students out of 85 eligible take the test, for a response rate of 78 percent; in 2004, we had 69 out of 86, for a response rate of 80 percent. So the overall response is encouraging.
But we can't pat ourselves on the back. Another measure of how seriously students take the CAAP test is a "Self-Reported Performance Effort" survey incorporated into the test itself. Last year, 33 students marked the bubble for "Tried My Best," 30 marked "Gave Moderate Effort" and 3 marked "Gave Little Effort." This year, 27 marked "Tried My Best," 35 marked "Gave Moderate Effort," 5 marked "Gave Little Effort" and 2 gave no response to the question. Table 2 gives the performance effort scores for both years, converted to percentages of the number of SCI students taking the test each year for the sake of comparabilty.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Now I've got to admit I'm not sure exactly where - or how - I might draw the line between "tried my best" and "gave moderate effort." Or what the operational difference is between "gave moderate effort" and "gave little effort." And I know it is difficult at the best of times to generalize about standardized test scores for a population as small as ours. But I suspect the numbers here reflect a slight falling off of student effort once we got them in to take the test.
Accordingly, I wonder if we may have inadvertently introduced an intervening variable by giving students a drug-alcohol survey for the Illinois Higher Education Center this year at the same time as the CAAP test. We already know that trying to do too much last month on Assessment Day gave us administrative problems.
"We found that it was too much for some of the students to complete both (the CAAP and the drug-alcohol test) in the test period, and we are reevaluating that for next year," said Student Affairs Dean Kevin Broeckling.
One of our operating philosophies on the Assessment Committee is planning for continuous improvement, evaluating our procedures in the light of experience and revising them as necessary. Out of those deliberations we will get a better sense of where to go next with CAAP testing, and how to refine our procedures for adminstering the test. That's how assessment is supposed to work.
In the meantime, we now have hard data on our students' reading abilities to take into account as we plan our courses and develop classroom and program assessment measures for the coming school year. And we have a good start on acquiring benchmark data for future assessment initiatives in the coming years. It isn't a full-fledged Gen Assessment program yet, but overall it's an encouraging start.
-- Pete Ellertsen, editor, Nuts & Bolts
Nuts & Bolts is an electronic newsletter published by the Assessment Committee of Springfield College in Illinois. Members are: Moses Allen, Bob Blankenberger, Alice Gutierrez, Scott McCullar, Dave Saner, Steve Stowers, Barb Tanzyus and Pete Ellertsen; Kevin Broeckling, dean of students, and Jeff Mueller, dean of the college, serve ex officio.
If you have information, comments or feedback, please contact any committee member or Nuts & Bolts editor Pete Ellertsen, in Becker L-16A on the SCI campus, 525-1420 ext. 519 or by e-mail at <ellertsen@sci.edu>.
