Assessment Performed
by FacultyA Report to the Assessment Committee
Springfield College in Illinois
December 8, 2000
Assessment Performed
by Faculty
A Report to the Assessment Committee
Springfield College in Illinois
December 8, 2000
At the October meeting of the Assessment Committee, I was asked to report on what student outcomes assessment techniques, other than grading, Springfield College faculty use in their classes. Dave Holland was asked to poll faculty in Science and Mathematics; Bob Blankenberger was asked to poll faculty in the Social and Behavioral Sciences and Humanities and Fine Arts Divisions; and I polled my colleagues in the Language/Literature Division. In all we received responses from 21 of 35 faculty members at SCI, including adjuncts. All but two full-time faculty responded - a very encouraging response rate. In order to provide a context for our survey data, I consulted past members of the Assessment Committee; Susan Full and Lynette Shaw-Smith, who recently attended an assessment conference in Indianapolis, Ind.; Judi Anderson, who has agreed to give a workshop on assessment in December; literature on the subject including Classroom Assessment Techniques (1993) by Thomas A. Angelo and K. Patricia Cross; addresses by Cecilia L. Lopez, associate director of the North Central Accreditation Commission on Institutions of Higher Education, setting forth NCA requirements (1996, 1998); and several sites on the World Wide Web that provide information on student outcomes assessment. Since our survey was informal, our findings are somewhat impressionistic. But in view of time constraints and the small number of people to be surveyed, we believed a statistically rigorous approach would be neither feasible nor appropriate.
In general, we found most faculty make a good-faith effort, independent of grading, to assess student outcomes - roughly defined as a measure of what students learn - and they use the information they gain from the effort to change such matters as teaching methods, textbook selection, syllabuses and/or emphasis on skills or content to be covered in class. On the other hand, we found potential areas of concern in: (1) an apparently low level of use of specific assessment techniques recommended in the literature; and (2) insufficient documentation of how assessment data are factored into planning processes, with particular reference to changes resulting from data analysis. It would appear that the ongoing efforts we found reflect SCI's traditional commitment to close interaction with students more than they do mandated student outcomes assessment. They do, however, give us something to build on.
Assessment, planning and change (background). Student learning outcomes assessment is best understood - and performed - as part of a larger commitment to institutional research and planning. According to the American Association for Higher Education, assessment "is most likely to lead to improvement when it is part of a larger set of conditions that promote change." Multiple measures of learning are preferred, since learning involves too complex a set of behaviors to be adequately measured by any one instrument, and assessment is best understood as a continuous process of gathering information and using that information for further planning. According to a set of principles adopted by the AAHE:
Learning is a complex process. It entails not only what students know but what they can do with what they know; it involves not only knowledge and abilities but values, attitudes, and habits of mind that affect both academic success and performance beyond the classroom. Assessment should reflect these understandings by employing a diverse array of methods, including those that call for actual performance, using them over time so as to reveal change, growth, and increasing degrees of integration. (2000a)
NCA's strong recommendation is that "a successful program to assess student learning must be faculty owned, developed, and implemented." But NCA also suggests assessment must also have the backing of senior administration and student services personnel, and the process is best implemented by "establishing feedback loops that regularly communicate the results of assessment activities" (Lopez, 1998). On the national level, the assessment movement grew out of widespread political agitation during the 1980s for reform and "accountability" at all levels of education. In 1989 NCA adopted a policy calling for "its member institutions to submit plans for assessing the learning of their students at every level of study" (Lopez, 1996). That policy has continued in force, and NCA requires an assessment plan for reaccreditation. Mandated measures of student learning include such tools as pre- and post-tests, student portfolios, national standardized (normed) tests, analysis of transfer results and satisfaction surveys of students, graduates and employers, as well as the use of classroom assessment techniques (CATs) by individual faculty members. A widely accepted definition of the overall process of assessment was formulated in 1995 by a panel of educators headed by Thomas A. Angelo, then the AAHE's assessment forum director. It reads as follows:
Assessment is a means for focusing our collective attention, examining our assumptions, and creating a shared culture dedicated to continuously improving the quality of higher learning. Assessment requires making expectations and standards for quality explicit and public; systematically gathering evidence on how well performance matches those expectations and standards; analyzing and interpreting the evidence; and using the resulting information to document, explain, and improve performance. (AAHE, 2000)
Included under the general heading of assessment, as it is envisioned by AAHE and mandated by NCA and other accrediting bodies, are: (1) measures of institutional effectiveness designed to assess instructional, student services and administrative procedures; (2) program assessment designed to measure the overall effectiveness of departmental majors at four-year universities and the General Education program at two-year colleges like SCI; and (3) classroom assessment, which is designed to provide individual instructors with the tools they may need to double-check and cross-reference the measurements of student learning that go into grading. This report deals only with classroom assessment and does not attempt to address issues relating to Gen Ed or institutional effectiveness.
Like our colleagues nationwide, faculty at SCI are now required to use multiple tools for measuring student learning in our classrooms. Angelo says:
Classroom assessment is a simple method faculty can use to collect feedback, early and often, on how well their students are learning what they are being taught. The purpose of classroom assessment is to provide faculty and students with information and insights needed to improve teaching effectiveness and learning quality. College instructors use feedback gleaned through Classroom Assessment to inform adjustments in their teaching. Faculty also share feedback with students, using it to help them improve their learning strategies and study habits in order to become more independent, successful learners. (qtd. SIU-Edwardsville, 2000).
Because assessment is seen as a means of obtaining and comparing feedback on student learning from different sources, in much the same spirit that a scientist controls an experiment for intervening variables, grades in and of themselves are not counted as an assessment tool. Individual faculty members are urged to use multiple assessment tools, but are given considerable leeway in choosing how they will assess learning in their own classrooms.
Commonly recommended classroom assessment techniques (CATs) include pre- and post-tests given at the beginning and end of content courses; background knowledge questionnaires; a "one- minute" or "muddiest point" paper in which students are asked to comment on what they do and do not understand from a lecture; portfolios and studio/performance critiques in the fine and performing arts; and a variety of qualitative measures of student learning (Angelo and Cross, 1993). In addition, while NCA does not consider grading to be a legitimate assessment technique in and of itself, several procedures for analyzing the components of graded work - such as Primary Trait Analysis or the use of publicized grading rubrics - do pass muster with the accrediting authorities and would appear to be rather close in spirit to what several of our faculty at SCI already have undertaken.
Central to the assessment process is an emphasis on planning and implementation of those plans - if the data gathered by the process are not used to effect change, the process has not accomplished its intended goals. An unusually lucid model of how assessment data can provide feedback for change was presented to SCI's Language/Literature Division in 1998 by Judi Anderson, who learned of it at the 101st annual NCA meeting in 1996. Presented by by Sinclair Community College of Dayton, Ohio, it is called the PDSA model (an acronym for Plan, Do, Study and Act, sometimes cited in the literature as PDCA for Plan, Do, Check and Act). It derives from the industrial management philosophy of W. Edwards Deming and is intended to put into effect a cycle of continuous planning, feedback and improvement (Clark. Cf. Marchese, 1997. Link here to see a graphic version by the Michigan Quality Council). As Judi explains it, classroom teachers can apply PDSA like this:
The key to the PDSA model is planning for continuous improvement. While it derives from quality control engineering, its spirit is close to that of the scientific method with its testing of hypotheses and its emphasis on using data for further planning.
Assessment at SCI. Faculty were surveyed early in November. Rather than use a "one-size-fits-all" questionnaire for faculty in all subject areas, the Assessment Committee allowed division representatives to poll faculty in their divisions as they believed most appropriate. Since it was our belief that classroom teachers at SCI already try to determine what their students are learning, either intuitively or as a result of being exposed to CAT literature (or a combination thereof), we used open-ended questions that allowed faculty members to describe what they do in their own words. While this methodology is admittedly somewhat impressionistic, it allows us to establish a benchmark against which we can measure the effects of further faculty development in the area of assessment.
Of the 21 faculty members who responded, all reported formal or informal efforts to determine what students were learning (other than grades), with the intention of modifying their teaching to ensure that students understand the material they are to cover. In a cover memo accompanying results from the social sciences, humanities and fine arts, Bob Blankenberger suggested SCI's size and commitment to interaction with students may help account for this: "I think that most faculty at SCI do a good job of being accessible to students. If there is a problem, students are generally comfortable in approaching most faculty. This is an opportunity for feedback that may not be as readily available at other types of institutions." In addition, some awareness of mandated assessment and specific CATs recommended in the assessment literature exists among full-time faculty at SCI. That stands to reason: Former members of the Assessment Committee have spoken at faculty meetings about CATs and about such matters as incorporating the College's mission statement in syllabuses, and a number of faculty report using techniques that were discussed at those meetings. SCI faculty in general seem to be aware of the importance of assessment and attempt at least informally to assess student learning in their classrooms already by means other than grades.
On the negative side, we found resistance to what at times is perceived as a time-consuming mandate that does not give faculty useful information. "Personally, I think we should continue to teach our students, each of us assessing in his or her own way, and spend less time and energy trying to quantify how we do it. ... I will make every effort to 'comply,' but I will probably complain while doing it," said an adjunct instructor. "Two faculty members said their experiments with assessment techniques were unsuccessful. One reported:
One semester a few years ago, I gave a pre-test to my [reference to specific course deleted] class at the beginning of the semester and the same test again at the end of the semester as a post-test. ... I don't remember what kind of results I got, but I wasn't convinced it was all that useful a thing to do, because (1) it meant I had two fewer days to actually teach, and cover the material to be covered; (2) it was a problem how to motivate the students to take the test seriously; and (3) quite a few students dropped or disappeared over the course of the semester, so they were not post-tested. Also in some classes ... I don't expect students to necessarily know anything about the subject at the beginning of the semester, so why pre-test?
Another said, "I used to use a [specific reference deleted] standardized test - if students didn't receive credit for the test, they didn't 'try' at all. The data was skewed." That faculty member now analyzes graded work for assessment purposes: "I give a comprehensive final with both objective and subjective components. I compare their lecture test average with their final average." Skepticism also was expressed with pre- and post-tests administered College-wide. "Hard to do with varied majors and few students," said one respondent. "Our low retention and graduation rate would make most entrance and exit tests useless/invalid," said another. "Why aren't instructor grades enough of an assessment?" asks another instructor. "Why are tests inadequate? Should we merely codify our standards? List exactly what an 'A' on a test means and why? What is it that the people putting pressure on us think we're doing wrong and are we truly doing it wrongly?" We found SCI faculty making little use of specific terminology that appears in the CAT literature. Even though the open-ended nature of questioning in our survey did not elicit specific terms and several faculty described measures obviously designed to accomplish the same purposes, this may point to a lack of familiarity with the literature.
Of the 16 full-time faculty members who responded to the survey, more than half (9) use pre- and post-tests to assess student knowledge at the beginning and end of a course. Eight instructors across the disciplines reported informal discussions or, as an instructor in the sciences put it, "use of verbal questions [to] give a qualitative assessment (on an individual basis) of students' understanding on a specific or general concept." The same instructor administers "a written survey that includes information on science and math background" and "tr[ies] to use verbal assessment in early conversations" in all classes. A languages and literature instructor administers a diagnostic test on "basic grammar (including terms)" at the beginning of the semester, and "a short evaluation of the diagnostic test is given the student (not a grade) advising the student whether the student needs help with grammar or coherency or both or neither."
Other assessment measures also are used, although their use is not as widespread. Two reading instructors use a nationally normed standardized test to measure reading levels as the beginning and end of the semester. Both full-time faculty members in the fine and performing arts use performance or studio critiques, and the art instructor uses portfolios.Two instructors reported using the "one-minute essay" recommended in the CAT literature, and others attempt to accomplish much the same purpose more informally. "I ask at the beginning and end of each class if they understood or had questions about the previous night's homework," says one instructor in the science and mathematics division. One instructor in the social science and humanities division reports use of "pre-class discussions of what was not understood from the last class," and two report "student sessions before exams to provide feedback as to what's not getting through." A science instructor says, "I often ask several questions of students after the second lecture test, about the nature of the class, tests, progress, teaching methods, etc." Another, in languages and literature, says his use of a Socratic method of questioning "provides instantaneous feedback" on learning throughout the semester, which he corroborates with the "one-minute" and "fuzziest point" exercises recommended in the CAT literature. "When it is time to look at new editions for textbooks, I have the students review both lab manuals and textbooks for being 'user friendly,'" notes one instructor in the sciences. Several instructors said they use quizzes, or assessment items embedded in quizzes, for diagnostic purposes. "Quizzes on sections are used as a guide and not grades," says a social sciences instructor, who also administers pre- and post-tests.
Use of CAT measures is most consistent among instructors in the Languages and Literature Division, and that division has used assessment data to make decisions on curriculum, most significantly the expansion of developmental writing courses from three to four credit hours that went into effect with the 1999-2000 college catalog. Two language/lit instructors report extensive use of assessment techniques and a rigorous effort to use the information thus derived for planning purposes (a third uses CATs but so far has gained little information from the exercise that he didn't already know).
"I use a pre- and post-test in all of my classes: ENG 095, 098, 099 and 100," reports the College's developmental English instructor. "I have used CATs, muddiest point and minute paper. I talk to my 099 students while or after they are in [ENG] 111 to see what helped and what else they have needed. I have changed 099 every year. At first I focused 099 on grammar and they wrote several single-paragraph papers and one essay. Then I refocused 099 -- now they write only essays. We discuss thesis, unity and development far more than grammar."
Another language/lit. instructor reports considerable use of assessment indicators embedded in graded work, a sophisticated technique that attracts increased attention in the literature of the field. She uses students' first and last in-class essays in her ENG 111 classes as pre- and post-tests for "errors of correctness," and prepares individual grade sheets to evaluate essay content in all of her classes. "Different writing assignments stress different critical thinking strategies and essay content should be judged as such," she says. "Using criteria-specific evaluation of content has improved my assessment of student writing." She also prepares questionnaires "designed to elicit student feedback on their understanding of my specific writing assignments; the design of my assignment; my comments on and evaluation of their written work; their own reasoning and writing abilities; and to evaluate whether or not individual conferences I have with students on their writing have been beneficial." As a result of data gathered from these and other assessment measures, This instructor has used assessment results to reaffirm her use of the SCI Guidelines for the Evaluation of Written Compositions, a list of correctness errors, and makes it a point to "[r]espond one-on-one to student feedback. Depending on their comments, I have changed assignments, clarified and revised essay prompts, etc." She says assessment helps create "a 'community of learners' [as] students benefit from hearing each other's response/analysis" and "become more involved in their study of literature and its application to life and other studies."
Discussion. Results of our survey suggest that SCI's faculty is at least partly on board in terms of attempting to gauge what learning takes place in their classrooms; the survey also suggests, however, that most of us would benefit from a thorough faculty development effort designed to familiarize us with the language and techniques of mandated student outcomes assessment. Several of the objections and concerns voiced in responses to our survey might be resolved if faculty were better grounded in the subject; making expectations public, or telling students "exactly what an 'A' on a test means and why," for example, is an important part of assessment. There is nothing new or startling about instructors finding out what their students have learned. In his classic The Art of Teaching (1950), Gilbert Highet counsels teachers "every day you should find out whether your pupils have improved or not" and suggests ways to ensure they "will begin to see their learning and your teaching as a co-operative endeavor" (124-26). As Highet and others make clear, good teachers have been doing that since the days of the ancient Greek academy. What appears to be lacking at SCI, at least in terms of classroom assessment, is familiarity with specific NCA-approved techniques.
Several faculty members indicated on their questionnaires that they would welcome more information on how to do assessment the NCA way, and some of us would be able to better focus our efforts if we knew more about CATs and where they fit into the mandated requirements. At least among some of us who have tried to learn about learning outcomes assessment on our own, there is a good deal of confusion about when information taken from graded exercises can be used for assessment purposes and when it cannot. We also need to know more about how to document our assessment of student learning and the changes we make as a result of it, and we could benefit from more instruction in how best to factor assessment into a planning cycle for continuous improvement. Several people I talked with said faculty members at SCI appear to be doing assessment, but we don't appear to be doing anything in particular with the data we gather.
In that case, what is - or isn't - happening at SCI reflects a national trend. In 1999 researchers for the National Center for Postsecondary Improvement surveyed 1,393 institutions nationwide and reported that:
State agencies and institutional accrediting bodies may have stimulated the adoption of assessment activities by many postsecondary institutions, but these initiatives appear to have had little impact on how institutions have supported or used student assessment to improve their academic performance. ... Still missing is a sustained commitment to using student-assessment data to make academic decisions, to link goals to educational improvement, and to monitor the impact of assessment - internally and externally - on institutional performance. (56)
In commenting on the NCPI study, Marchese of AAHE suggested (1999) part of the problem is that assessment has been perceived "as an external imposition" rather than something that can help educators do their jobs better (cf. Burke). A good way out of the resulting impasse, Marchese said, rests with faculty:
A big part of the answer surely lies in constant, repeated acts of gathering student work and feedback; reflecting on it; trying new, old, or altered approaches, followed by more feedback, reflection, and trial; and so on - in other words, through assessment, continuously done, in the service of smarter teaching and improved learning. (4. Italics in the original.)
Much needs to be done at SCI, but faculty development would appear to be a first step that would build on what we are already doing and give us more of the resources we need to strengthen our classroom assessment initiatives. This in turn would give more faculty the necessary background to take on the crucial role envisioned by NCA as the College addresses a plan for assessing SCI's Gen Ed program and institutional effectiveness.
-- by Peter Ellertsen, secretary, Assessment Committee
References
Occasional Papers on Assessment, Springfield College in Illinois [link here to return to index]