An analysis of the reported use of classroom assessment techniques during the 2004-2005 academic year shows that faculty reporter greater use of CATs in general, and considerably greater use of multiple measurements to gauge student learning in their classes than in previous years. For the first time, instructors were asked to detail changes in their teaching methods as a result of the feedback they received from the CATs; with greater or lesser specifity, many reported that the use of CATs enabled them to gauge student learning and to give more instruction on matters in which the CAT results indicated a lack of student understanding. Others reported raising the stakes for certain types of student work -- e.g. giving quizzes over chapter readings when assessment measures indicated that students were not in fact doing the assignment readings -- and thus enhancing student performance in subsequent assessment measures. The overall picture that emerges is of instructors making use of a wide variety of CATs as formative assessment measures enabling them to address areas of concern during the semester; of the high number of instructors using pre-tests, post-tests or a combination of the two, several volunteered changes in instructional strategies as a result of pre-/post- testing, suggesting that they use that technique, although it involves testing at the end of the semester and its therefore considered a summative technique, as a formative as opposed to summative assessment technique.

Response rate to the annual CAT questionnaire continued to be an area of concern during the 2004-2005 school year, and the Assessment Committee this year is studying the feasiblity of administering the questionnaire in connection with a classroom assessment workshop at the beginning of the next academic year and incorporating a discussion of the techniques into the workshop in order to give greater reason for faculty to feel that filling out the questionnaire would positively enhance their classroom performance as opposed to fulfilling a bureaucratic mandate solely for the purpose of compiling unused data.

In general, however, the data from 2004-2005 suggest broader acceptance of assessment and a more sophisticated use of assessment techniques on the part of faculty than in past years. This is evidenced not only by the reported use of techiques discussed in the literature on assessment but not elicited on the questionnaire, as was noted in the 2003-2004 annual report on classroom assessment, but also a significantly greater number of instructors reporting multiple use of assessment techniques. Some credit for this increased sophistication concerning assessment may be given to a series of articles on classroom assessment in SCI's assessment newsletter, Nuts & Bolts, beginning in August 2004 and lasting into January 2005. More credit, however, probably is due to division chairs who have been working with instructors on assessment as well as other professional development iniatives on a continuing basis.

Under standing procedure initiated in previous years by the then-Dean of Academic Affairs, all fulltime and part-time faculty in SCI's traditional program and the New Horizons adult accellerated program were asked to complete a questionnaire checking off enumerated CATs they use from a list of nine. Those who checked "Other" were asked to elaborate in a blank space on that sheet. Incorporated into the questionnaire for the first time in 2004-2005 was a form on which for their classes they were asked to list an "Assessment technique used," their "Findings" and "Action Taken." They were also asked to attach documentation. These two-page questionnaires were distributed in faculty mailboxes with follow-up letters mailed to those who did not respond. Faculty members also were contacted by telephone by the Academic Affairs office.

As in past years, completed surveys showed that the faculty who responded to the questionnaire used a wide variety of techniques to assess student learning in their classrooms; to a greater extent than in the past, they reported using multiple measures, and this affected the overall results, with the use of a substantially larger number of individual CATs being reported, even though the number of instructors responding to the survey was about the same as in the previous year. None of the 30 instructors surveyed reported using no CATs in their classes; five reported using one, and 21 reported using multiple techniques. Questionnaire results were tallied to reflect the number of courses in which a given CAT was used. Their incidence was as follows:

Spring 2005 Full Time Faculty Classroom Assessment Technique Results

*Please note: Of the 30 instructors who returned surveys, 21 cited the use of CATs in more than one course.

Among the instructors who marked "Other," a wide variety of techniques was used. Written measurements such as laboratory reports and scientific literature review papers, reflective essays and "lessons learned" journals were reported. An instructor in humanties reported frequent classroom discussions over metacognitive issues arising from material that was covered in class. An English instructor used an online "ratemyprofessor.com" survey to measure his performance at the end of a class period. An instructor in engineering reported the iterative or repeated use of the "RSQ2C" technique (a menomic for Recall, Summarize, Question, Comment and Connect) in which students describe an issue from the day's class and try to connect it to a broader understanding). A psychology instructor used a self-assessment questionnaire given early in the semester in which students evaluate their academic progress and anticipate concerns that might arise in class. One, a behavioral scientist who teaches statistical method, turned in a detailed statistical analysis of student performance on questions embedded in graded student work; the instructor, in communications, who asked his students to write reflective essays embedded those essays in graded work as well.

In comparison with the previous year, the use of classroom assessment techniques increased dramatically. In addition, some of the responses to questionnaires suggest a greater depth of knowledge of assessment on the part of some faculty than in the past; descriptions of assessment techniques volunteered in open-ended questions tended to be less informal and more in line with descriptions of assessment techniques in the literature than in the past, perhaps, although it would be difficult to quantify this impression. In any event, any improvement suggests the need for continued effort in a continuous improvment cycle model. It may be that the newsletter articles cited above had some impact on faculty awareness of assessment techniques, but the newsletter series discussed those assessment techniques that are mentioned on the questionnaire, and citation by faculty of other techniques would indicate rather a pattern of independent reading on the part of the faculty. A Benedictine University instructor in management and organizational behavior, for example, reported on having his students write "lessons learned" journal entries -- a procedure detailed in the literature on management but not in the most widely used sources on classroom assessment for teachers. These were weekly reports in which students reflected on "1. What went well, and why? 2. What did not go so well, and why. 3. Suggestions for improvment." Not only did this technique give students practice in aa technique they will use in their professional lives; it gave him feedback he could use in the classroom. As a result of the journals, he reported, he was able to "customize course deliberables to meet student's needs [and] balanced the concerns."

It is believed that overall the data may be interpreted as a hopeful sign that SCI's organizational culture is slowing becoming more receptive to assessment. For the sake of comparison, last year's reported use of classroom assessment techniques is detailed below:

Spring 2004 Full Time Faculty Classroom Assessment Technique Results

*Please note: Of the 28 instructors surveyed in 2005, 13 cited the use of CATs in more than one course.

Instructors reporting the use of pre- and post- tests to ascertain student learning, along with the allied technique of administering a prior knowledge inventory at the beginning of the semester or before taking up a subject in class nearly doubled from 2004 to 2005. Use of the "minute paper," in which students are asked at the end of a class period to write down the clearest and most difficult point covered during that class, increased from seven reported in 2004 to 30 reported in 2005. Use of other CATs increased as well, but not at as high a rate as the pre-post- tests and the minute paper.

Instructors reporting the use of pre- and post- tests to ascertain student learning, along with the allied technique of administering a prior knowledge inventory at the beginning of the semester or before taking up a subject in class nearly doubled from 2004 to 2005. Due to the questionnaire design, it is not possible to determine how many of the faculty who indicated using the technique used both pre- and post- tests in order to ascertain value added, and how many used a pre- test as a probe of background knowledge. Discussion of the technique in open-ended questions on the second page of the survey would suggest that both were being used in order to measure student learning and vary teaching methods. Use of the "minute paper," in which students are asked at the end of a class period to write down the clearest and most difficult point covered during that class, increased from seven reported in 2004 to 30 reported in 2005. Use of other CATs increased as well, but not at as high a rate as the pre-post- tests and the minute paper. This may reflect the fact they appear first on the questionnaire, or it may reflect a sense of their greater efficacy on the part of faculty -- most likely it is a combination of both.

As in the previous year, the survey suggests that CATs are used by full- and part-time faculty alike. Of the 30 instructors who reported the use of CATs, one used them in six separate courses; two used them in five; four used them in four courses; four used them in three; eight used them in two courses; and nine used them in one course. Since the same people often teach in the SCI tradition and New Horizons programs, as well as the Benedictine University baccalaureate courses now offered on the SCI campus, it is difficult to get an exact number of the instructors falling into each category. It would appear, however, that the large number of people reporting the use of CATs in one or two classes reflects the number of adjunct teachers especially in the New Horizons and Benedictine programs.

For the first time in 2005, the CATs survey asked faculty to describe and document the changes made in thier teaching as a result of using CATs to measure student learning. This was done by including three open-=ended questions in the survey. While the answers to these questions do not lend themselves to coding and cquantification, they suggest an awareness of the importance of using assessment techniques to evameasure student learning and thus evaluate their performance and modify it in accordance with the principles of planning for continuous improvement.

Several of the many instructors who checked that they used pre- and post- tests as an assessment measure indicated an awareness of continuous improvment planning processes in their discussion. several instructors say they used their pre- and post- tests for planning. An adjunct psychology teacher, for example, gave pre-/post- tests on the 13 chapters in her textbook throughout the semester. She found that "students were not reading -- thus pre-tests were bing submitted with blanks." Accordingly, she "discussed [the] issue with students. Then pre-tests were given then as 'take-home' assignments which they brought back the next day. Successful completion supported more discussion, and psot-tests were completed at [a] 90-100% success [rate]. Assignments were completed." In other words, she raised the stakes and got better results. More importantly, she adjusted her teaching to put more emphasis on the reading.

Pre- and post-tests were used in other ways as well. When an ethics instructor had students complete a moral inventory that accompanies the textbook, he said "giving the same inventory at the end of the course showed growth in outlook, helped them think out of their moral box." He planned to repeat the procedure the next semester. An English instructor said pre-/post- testing showed students in English 112 "still do not have a clear understanding of the difrerences between persuasion and argument." She added, "I will spend more time illustrating the differences and assess on this topic more." Another freshman composition teacher said:

<blockquote>In both ENG 111 and 112, I used this traditional form of assessment to gauge what knowledge the students were coming to class with and with what knowledge they were leaving. Both classes reflected positively in both, though there are still areas that need concentration (paraphrasing, unity, transitions). In both classes, I will use more in-class and online exercises to increase knowledge in these areas."</blockquote>Again, the post-test results were "summative," but he was unsing them as a "formative" planning tool. In much the same way, a full-time humanities instructor indicated the use of pre- and post- tests as both a summative and formative assessment tool in a Benedictine University humanities survey covering "current political, economic, cultural and technological challenges." He reported the post-test at the end of the semester showed improvment in non-western topics is weaker than in western one." Accordingly, he said, "greater emphasis will be placed on non-western [cultures] next spring. I will reduce detail and concentrate on more general information to improve cultural awareness."

In the 2004 report on classroom assessment,

<blockquote>Overall ... it must be said that surveying classroom assessment at SCI has up to this point been an exercise in counting items checked off on a questionnaire instead of collecting the kind of data that can directly benefit faculty members or stimulate further inquiry on their part. The Assessment Committee has realized this and continues to take steps to address the problem.</blockquote>The chief innovation in 2005, pursuant to that finding, was to add the second page of open-ended questions to the survey questionnaire eliciting discussion of the use of CAT results to change teaching methods. Responses to the questionnaire accordingly have been compiled, and the open-ended discussion will be used as a basis for future articles in the Nuts & Bolts newsletter. Use of pre- and post-tests as a summative and formative technique, as reported above, can be shared with instructors via the newsletter, as can such added techniques as having students write "lessons learned journals" or reflective essays or embedding assessment measures in graded work. In furtherance of that objective, especially because there is a considerable time lag in the collection and reporting of data, the Assessment Committee is studying the possiblity of incorporating the information gathered in the CAT survey into an assessment workshop before the beginning of the fall semesterr.

-- Peter Ellertsen, chair, Assessment Committee, Jan. 14, 2006