Classroom Assessment Report for Fall 2006 Semester

By Brian Carrigan

This report is based upon the Assessment Committee Report Forms returned by 33 faculty members, and represents 70 individual courses (counting the cross-listed (MGT 251/PSY 251 as a single course). This represents a response rate of approximately 60% of the courses taught. Of these courses:

Classroom Assessment Technique	Courses used	%
Pre/Post Testing	46	66
Minute Paper	21	30
Muddiest Point	10	14
Directed Paraphrasing	16	23
Three Question Survey	29	41
RSQC2	18	26
Assignment Assessments	26	37
Misconception/Preconception Check	17	24
Other	29	41

Some instructors employed what they described as modified versions of instruments listen in the questionnaire, such as the three-question survey. In some cases, these modifications may have been quite minor (in which case it might have been preferable to count them in one of the other categories); in other instances, the modifications were quite severe (e.g., informal opportunities to express questions or confusion regarding the discussions/lectures; application of the day’s discussion to cases), to the point that these appear to be genuinely different instruments. For the sake of consistency, in all cases these were considered “other” instruments. No two instructors used the same “other” instrument; therefore, although these were used in 41% of the courses studied, no significant generalizations about there use may be inferred. Some of these are, no doubt, uniquely suited to a particular subject matter; however, it might be of interest for instructors to share information regarding the instruments they have devised. For example, an application of discussion material to specific cases might be considered a hallmark of a law course – but it could also be used in other courses with practical applications (e.g., physical sciences, nutrition).

It may be seen that Pre/Post testing is the most commonly used single instrument (employed in two-thirds of all courses studied), with three-question surveys, assignment assessments, and “other” instruments all used in about 40% of the courses.

It may be of interest to consider some of these instruments as a group. Minute papers, muddiest point questions, directed paraphrasing, three-question surveys, RSQC2, and misconception/preconception checks each involve a brief written response to the day’s class. If these instruments are binned together, then modified versions of them might also be moved from the “other” category. The remaining “other” instruments may be divided into essentially informal techniques and unique, but formal assessment instruments. If these recategorizations are done, the results are as follows:

Classroom Assessment Technique	Number of Courses used	% of Courses used
Pre/Post Test	46	66
Written Class Assessment	45	64
Assignment Assessment	26	37
Informal Assessment	6	9
Other	15	21

It may be seen that one or more written class assessments is used in about as many classes as employ pre/post tests. These techniques complement one another, in that the written class assessments (and assignment assessments) provide information about the day-to-day activities, while the pre/post tests measure performance improvement over the semester.

All told, some type of formal assessment instrument was used in approximately 93% of the reported courses. This is significant: though informal assessments such as class discussions provide information about student learning, they do not provide a permanent record, and hence cannot be accessed at a later date. If the purpose of assessment is to provide evidence for claimed student outcomes, informal assessments (however valuable to instructors – and they are quite valuable for that purpose) are insufficient.

As stated previously, not all instructors have submitted report forms. It is therefore impossible to state with certainty what percentage of courses are assessed with a given instrument type. Fortunately, a large proportion of courses taught in the Fall 2006 semester (approximately 60%) are represented, so one has some confidence that the sample we can study is broadly representative of all courses. Under the assumption that the results for one instructor are independent of those for another instructor, one may use a binomial distribution to estimate the uncertainties in these percentages. The results are given below:

Classroom Assessment Technique	%	+ %
Pre/Post Testing	66	8
Minute Paper	30	4
Muddiest Point	14	2
Directed Paraphrasing	23	3
Three Question Survey	41	5
RSQC2	26	3
Assignment Assessments	37	5
Misconception/Preconception Check	24	3
Other	41	5

Similar results are obtained for the rebinned data. These estimates are, however, not rigorously warranted (in part because the sample size is comparable to the total number of courses taught), but may serve to illustrate order of magnitude of the uncertainty. The only lesson to be gained from this analysis of uncertainty is caution in stating that one instrument is used slightly more often than another. Thus, although it would appear that three question surveys are used in slightly more courses than are assignment assessments, the difference in use of these instruments is not statistically significant.

A study of the individual reports reveals some weaknesses in the current assessment efforts. The report form includes not only space to list which assessment techniques are used in various classes, but also what information was discovered by use of that technique and what action has been taken (or what action the instructor proposes to take) in light of that information. In all too many cases, this portion of the report has been left blank. This is unfortunate: the purpose of assessment is not to burden students with additional surveys, but to discover how effectively our course objectives are actually being achieved. Once that is known, one would presumably wish to adjust one’s strategy to achieve greater success. Perhaps the Assessment Committee might wish to sponsor a workshop on how to decipher assessment instruments, and another on one or more newer teaching strategies that have application in a broad range of fields.

Another weakness that is apparent on studying reports from individual instructors is the use of what I have called “informal reports”. These cannot be verified by an outside observer. They are, therefore, of little value should a prospective student, the parent of a prospective student, or another stakeholder wish to know the value of an SCI education. They are of little value, again, when outside accreditation teams wish to evaluate our programs. Moreover, some of these informal methods appear to be really the sort of evaluation of course work, essentially equivalent to grading assignments or tests. These are evaluations, of course – but of the individual student, not the course! The fall 2006 survey points op the value of continuing workshops on the nature and purposes of the assessment program.