NUTS & BOLTS

An electronic assessment newsletter serving Springfield College in Illinois, a Benedictine University partner

---

Dec. 2006-Jan. 2007 Vol. 7 No. 5

Assessment guru questions ‘value-added’ tests

Banta is IUPUI’s vice chancellor for planning and institutional improvement. She’s an author of Assessment in Practice: Putting Principles to Work on College Campuses (1996), a basic text in the field, and numerous monographs on assessment. In the Jan. 26 edition of Inside Higher Ed, Banta said:

While standardized tests can be helpful in initiating faculty conversations about assessment, our research casts serious doubt on the validity of using standardized tests of general intellectual skills for assessing individual students, then aggregating their scores for the purpose of comparing institutions.

But after re-reading Banta’s column and thinking it over, I think we’re on the right track at SCI … even though we do use standardized tests as one of several measures of what our students are learning. The key to it is what she says about using the tests for “initiating faculty conversations” instead of comparing institutions.

That’s basically what we do at SCI, even though some of our data are comparative. We give the reading and math modules of the Collegiate Assessment of Academic Proficiency to our sophomores, and we purchase data from ACT Inc. comparing those scores to what they made on ACT tests in high school.

Thus we get a rough, norm-referenced measure of “value added,” as we are required to do for accreditation. But on SCI’s assessment committee, we have developed a strong instinct our standardized test data are giving us at best a very imprecise measurement of what SCI students are learning.

So we have been cautious in interpreting the data, and we have opted for multiple measures of student learning including classroom assessment as well as standardized tests. Frankly, I don’t think any of the measures are very precise. But we’re better off if we have two imprecise measures, and we can use them to control for each other.

So I didn’t take Trudy Banta’s column in Inside Higher Ed as an attack on the kind of standardized testing we do at SCI. It was more an explanation of what testing can and can’t do.

It was also a clear warning that a testing regimen currently proposed by U.S. Education Secretary Margaret Spellings’ Commission on the Future of Higher Education would be a step in the wrong direction.

“I appreciate the commission’s focus on student learning and its assessment,” Banta said. “But my experience and my reading and conduct of research on these topics lead me to argue against the use of standardized tests of general intellectual skills to compare the effectiveness of colleges and universities.” Here’s her analysis in a nutshell:

For nearly 50 years measurement scholars have warned against pursuing the blind alley of value added assessment. Our research has demonstrated yet again that the reliability of gain scores and residual scores — the two chief methods of calculating value added — is negligible (i.e., 0.1).

We conclude that standardized tests of generic intellectual skills do not provide valid evidence of institutional differences in the quality of education provided to students. Moreover, we see no virtue in attempting to compare institutions, since by design they are pursuing diverse missions and thus attracting students with different interests, abilities, levels of motivation, and career aspirations.

And, of course, we continue to give the CAAP tests. For one thing, we’re required to.

The CAAP data we receive back from ACT Inc. so far suggest we’re in the ballpark with other two-year colleges. Specifically, when ACT compared our students’ CAAP scores with earlier ACT test scores, our students were within a few percentage points of national averages on such linkages as “made expected progress” (or lower or higher than expected) on ACT products over the years.

That isn’t very precise, but at this point the CAAP tests are the best normative measure we have of how we’re doing in our General Education program overall. And they’re reassuring – if we were way out of line, I think it would show, even on a standardized test..

More importantly, over time we are accumulating important data from the CAAP tests on reading skills like “students' ability to determine meaning from context, to infer main ideas and relationships, to generalize and apply information beyond the immediate context, to draw appropriate conclusions, and to make appropriate comparisons” and an array of mathematical skills as well.

Those data on basic skills, and subsets of the cognitive skills that go into complex activities like reading or mathematical reasoning, are what I’m most interested in as chair of the assessment committee.

In time the numbers from the CAAP math and reading modules can help give us a better understanding of what our students can do with these basic skills, where their strengths are and where they need help, and how we can help them better develop those skills.

That’s far from the world of federal bureaucrats and “value added” testing mandates. But that’s the part that will help us the most over the long haul. Besides, we’re required to give the tests as part of the assessment plan we submitted when we were reaccredited by the North Central Association.

I think Trudy Banta’s article in Inside Higher Ed is best taken not as a warning that we’re doing assessment wrong (especially when we’ve got no choice in the matter) but as a warning to be careful about interpreting the data we’re required to gather and to keep looking for newer and better ways to do assessment. .

ACT Inc. “Reading Test.” 2007. Collegiate Assessment of Academic Proficiency http://www.act.org/caap/tests/reading.html

Banta, Trudy. “A Warning on Measuring Learning Outcomes.” Inside Higher Ed 26 Jan. 2007 http://www.insidehighered.com/views/2007/01/26/banta

“CAAP Scores Look Good; Storm Clouds on Horizon.” Nuts & Bolts April 2006 http://www.sci.edu/comm_arts/assessment/newsletter05/nuts0608april.html

SCI’s assessment website has found a new home online, in a folder on the SCI domain set up for the communication arts program. It can be accessed from the SCI homepage by following the links to Faculty Websites and Assessment Program Goals and Objectives, or at

Thanks so much to “IT guy” Chris Norton and webmaster Tyler Gillette for their help in setting up the new website and their patience in putting up with my nagging during the interim. The responsbility for remaining dead links, missing graphics and errors, of course, is mine.

-- Pete Ellertsen, chair, assessment committee