UW-Madison: Outcomes Assessment

1998-99 & 1999-00 VERBAL ASSESSMENT REPORT

 October 1, 2000

Prepared by

Denise H. Solomon
Chair of the Verbal Assessment Committee and
Associate Professor of Communication Arts
  Leanne K. Knobloch
Verbal Assessment Project Assistant and
Doctoral Candidate in Communication Arts
  in collaboration with members of the
Verbal Assessment Committee
  Michael Bernard-Donals, English
Marion Brown, Agricultural Journalism
Allan Cohen, Testing & Evaluation Services
Michael Cruz, Communication Arts
Brad Hughes, L&S Program for Writing Across the Curriculum
Linda Hunter, African Languages & Literature
Robert Hawkins, Journalism & Mass Communication
Abbie Loomis, Library User Education Office
Stephen Lucas, Communication Arts
Evelyn Malkus, Engineering
Linda Marshall, Child & Family Studies
Ken Mayer, Political Science
Jim Sweet, Wisconsin Survey Center
Nancy Westphal-Johnson, L&S Dean�s Office

 

 

Abstract

The Verbal Assessment Project is designed to provide feedback on the two courses that comprise the general education communication requirements. Although not all facets of the general education program have been evaluated, the results of studies conducted thus far suggest that the requirements are working well. In particular, preliminary results indicate that students completing the Comm-B requirement positively evaluate their Comm-B class, have confidence in their writing, public speaking, and library research skills, and perform well with respect to a variety of writing performance criteria. A second study focused on one of the Comm-A courses (Communication Arts 100) revealed significant decreases in students� communication, writing, and library anxieties over the course of the semester. To provide input into general education instruction, the Verbal Assessment Project has also developed a number of channels for communicating with the university community. Most notably, the Verbal Assessment Bulletin disseminates information about assessment activities in the form of a campus newsletter. The future agenda for the Verbal Assessment Project includes studies of communication instruction outcomes among both graduating seniors and students in the Comm-A course, as well as continued efforts to provide insights and tools related to verbal assessment to constituencies at the University of Wisconsin.

This document provides a description of verbal assessment at the University of Wisconsin-Madison. Following a brief overview of the Verbal Assessment Project, specific assessment activities spanning the 1998-99 and 1999-00 academic years are summarized. Then, the various channels through which information about verbal assessment has been communicated to the university community are reviewed. The concluding sections highlight the agenda for the Verbal Assessment Project in 2001 and directions for future verbal assessment efforts.

 

 

The Verbal Assessment Project: An Overview

Since the fall of 1997, the Verbal Assessment Project has focused on evaluating the general education communication program. Broadly speaking, the objectives of general education instruction in communication address two types of student outcomes. First, students should manifest improved skills in writing, oral communication, and information literacy. In addition, students should develop attitudes about the process of writing, speaking, and accessing information resources that promote the implementation of those skills. Thus, the mission of the Verbal Assessment Project is to provide insight into the impact of general education communication courses on the associated abilities, knowledge, and attitudes of University of Wisconsin students. (Appendix A includes a summary of the context for verbal assessment at UW-Madison submitted as part of the 1997-98 Verbal Assessment Report.)

 

Verbal Assessment Activities: 1998-2000

In the 1997-98 academic year, the Verbal Assessment Committee addressed the need to clarify objectives, review assessment alternatives, and develop a long-range plan for assessment activities. The results of those efforts were formalized in the 1997-98 Verbal Assessment Report (available upon request). Although the timing and implementation of specific verbal assessment activities have been revised along the way, the Verbal Assessment Project has in large measure attended to the principles outlined in the earlier report. The following sections summarize the specific assessment studies conducted in the two years since that report was filed.

Communication B Study

The Communication-B course comprises the second and final campus-wide requirement in the general education communication program. In contrast to the Communication-A course, Comm-B classes are offered in a variety of departments throughout the university. In addition, classes that fulfill this requirement may take one of two forms: (a) classes that provide advanced instruction in written or oral communication, or (b) classes focused on conventions of inquiry particular to specific fields of study (i.e., content courses with substantial writing components). Although the Comm-B course was originally conceptualized as a low-enrollment class, Comm-B classes range from small seminars to large faculty lectures with divided discussion sections. In sum, Comm-B classes are presented in numerous departments, represent diverse disciplines, and take a variety of instructional forms.  

The assessment of the Comm-B course focused on writing performance and self-reported attitudes about writing, speaking, and library use among students completing the Comm-B component of the general education requirements. By focusing on performance and attitudes at the end of a semester of Comm-B instruction, this assessment effort highlights the outcomes associated with both the Comm-B course and the two course general education sequence. In addition, an examination of writing performance and students� beliefs and self-perceptions attends to both skills and attitudes as outcomes relevant to students� subsequent academic performance. 

The study conducted was designed to assess the Comm-B component of the general education communication program, while being sensitive to the great diversity of courses that meet this requirement. These goals demanded a research design that would support conclusions about the Comm-B course, in general, while at the same time integrating information about a student�s particular Comm-B class and academic background. To these ends, we randomly sampled sections of Comm-B classes offered in the spring semester of 1999, surveyed a subset of students enrolled in these classes, surveyed instructors about both their courses and the students in our sample, gathered descriptive profiles of students and courses from campus databases, and evaluated final papers that students submitted as part of their workload in the Comm-B classes.

In drawing the sample of Comm-B classes and students, we were guided to two goals. First, to provide a foundation for generalizable conclusions about the Comm-B course, we sought a large sample of classes representing the diversity of departments involved and capturing various instructional formats. Second, we focused primarily on students subject to the general education requirements who were enrolled in their first Comm-B class. Although many students take more than one Comm-B class, examining outcomes for students at the end of a first Comm-B class assesses the general education requirements as formally specified.

Classes were randomly sampled from departments across campus, and a total of 70 classes were visited to solicit participation and informed consent from students. The primary participants in the study were students enrolled in their first Comm-B class who consented to participate (N = 446). To provide a broader basis for assessing students� perceptions of the Comm-B course, a smaller number of students who had previously taken a Comm-B class (N = 53) also completed a subset of procedures.

At the end of the semester, student participants were asked to complete a web-based survey about their Comm-B course. In addition, a profile of each participant=s academic background was assembled from student record data retrieved from the ISIS system. Instructors were also asked to provide final papers written by students, which were subsequently evaluated by a team of trained raters according to a variety of writing performance criteria. Further information about student participants was gathered from questionnaires completed by instructors. The survey of instructors also solicited information about the class itself, which was supplemented by details retrieved from the timetable database. In total, the Comm-B study yielded a wealth of information from a large and representative sample of courses and students: 369 students completed surveys, papers were collected for 385 students, 70 different sections from 24 departments were represented, and 58 instructors completed surveys. 

Although the student sample was drawn in spring semester of 1999, assembling the components of the data file occupied much of the 1999-2000 academic year. These activities included substantial effort devoted to evaluating the Comm-B student papers and ongoing attempts to obtain student academic profile data.

Evaluating writing performance as evidenced in the Comm-B student papers required developing appropriate criteria, assembling a team of paper readers, and training paper readers to apply the criteria in a consistent fashion. An initial set of writing performance criteria was developed by a subcommittee of the Verbal Assessment Committee comprised primarily of the Comm-A course directors. To evaluate the relevance of these criteria to the Comm-B course, we asked instructors participating in the Comm-B study to rate the extent to which they attended to each performance outcome in their class. Then, seven graduate students who had experience as instructors in the Comm-A course were hired to evaluate the Comm-B student papers. As part of training efforts, the paper readers applied the criteria to a small subset of papers and met to discuss decision rules and to explicate the criteria; over the course of three training sessions, a detailed coding manual was developed. Once a sufficient level of inter-rater reliability was obtained, each paper was evaluated by a subset of four readers. Reliability was assessed periodically to ensure that raters were maintaining equivalent standards for applying the criteria.

A second component of the study addressed in the 1999-2000 academic year focused on accessing student record data for academic profile information. At the direction of Al Cohen of Testing and Evaluation Services, the Division of Information Technology (DoIT) created a program to retrieve information from the ISIS system. The implementation of ISIS and competing campus priorities resulted in substantial delays in this portion of the project. A complete set of student profile data was generated in June of 2000. Verification of the data file has continued to identify errors and omissions; therefore, efforts to secure this component of the Comm-B study are ongoing.

Given the difficulties assembling the student profile data, a full report of the results of this assessment effort is pending; however, descriptive statistics for the measures of student outcomes included in the study provide insight into the overall effectiveness of the Comm-B course. Perceptions of Comm-B classes solicited by the student surveys revealed that students positively evaluated both the writing skills they obtained from their classes and the overall value of the Comm-B course. In addition, students reported relatively low levels of anxiety and substantial confidence associated with their writing, public speaking, and library research skills. Moreover, ratings of writing skills based on papers that students completed at the end of the Comm-B classes indicated that students, on average, performed well with respect to a variety of basic writing performance criteria. Although further analyses are needed to clarify how these outcomes vary as a function of course format, curricular features, and student characteristics, the preliminary findings suggest that the Comm-B course in general is functioning effectively.

A full report on the procedures employed and the measures available as of this writing are provided in Appendix B.

Communication Arts 100 Study

The development of self-report measures to assess attitudes about writing, speaking, and information literacy in the 1998 Senior Survey (see Appendix D of the 1997-98 Verbal Assessment Report) provided a tool for examining attitudinal outcomes associated with particular course experiences. In an application of these measures, the impact of Communication Arts 100: Introduction to Speech Composition on communication, writing, and library anxiety was investigated.

The Comm-A component of the general education communication requirement is intended to promote general skills and attitudes as a foundation for subsequent course work and careers after college. As part of this effort, the Comm-A course incorporates a library training module taught by the Library User Education Office. By focusing on attitudinal outcomes of in the form of communication, writing, and library anxiety, this assessment study examined whether CA100 promotes self-perceptions relevant to students= academic performance beyond the Comm-A course.

The sample for the study included students (N = 419) enrolled in CA100 in fall of 1999. Participants completed a survey of library, communication, and writing anxiety on the first day of class and at the end of the semester. On both occasions, surveys were completed during class time. For measures of communication, writing, and library anxiety, students indicated the extent to which a series of statements applied to them. In total, 11 specific academic anxieties were measured, including anxiety about (a) communication, in general, (b) dyadic communication, (c) group communication, (d) public speaking, (e) writing, in general, (f) having writing evaluated, (g) using the library, in general (h) finding resources in the library, (i) starting projects in the library, (j) relative library ability, and (k) asking librarians for help.

Across all 11 measures of communication, writing, and library anxiety, students reported significantly less anxiety at the end of the semester relative to their scores on the first day of class. Although the magnitude of decreases in some academic anxieties varied as a function of students= demographic characteristics, results indicated reductions in communication, writing, and library anxiety for all subgroups examined.

Predictably, writing and library anxiety at the beginning of the semester were generally greater among freshman students, and declined across the sophomore, junior, and senior ranks. Nonetheless, students in all classes experienced equivalent decreases in communication, writing, and library anxiety over the course of the semester. Thus, although some general improvement in academic anxieties may be likely to occur simply through college experience, this study highlights the added benefits accruing from enrollment in CA100.

An examination of demographic differences in communication, writing, and library anxiety at the beginning of the semester also provides insight into the needs of particular groups of students enrolled in CA100. Specifically, students who did not have a public speaking course in high school and upper class students who have perhaps delayed fulfilling the Comm-A requirement reported significantly higher levels of anxiety about public speaking.

A full report of Communication Arts 100 Study is provided in Appendix C.

Senior Survey  

Spring 2000 marked the eighth semester for students who matriculated in the fall of 1996 - the year that general education requirements went into effect. To evaluate attitudes and beliefs relevant to public speaking, writing, and library use in this population, the Verbal Assessment Project agenda called for a replication of the 1998 Senior Survey. These data could also be compared to the baselines established by the 1998 survey to shed light on the overall impact of the general education communication program on students= attitudes and beliefs about associated skills.

As in 1998, the Senior Survey was comprised of two primary components assessing communication and information literacy. The oral and written communication literacy component of the survey included measures of (a) communication apprehension, (b) the perceived importance of various communication competencies, and (c) confidence in abilities to perform facets of oral and written communication tasks. The information literacy component addressed (a) confidence in abilities to use library resources, (b) library use, (c) the utility of any previous information management instruction, and (d) library anxiety. In a third component of the survey, a number of potential impediments to student learning were included.

The UW Survey Center was commissioned to implement the survey in April of 2000, to coincide with the timing of the previous survey conducted in 1998. Due to problems associated with the implementation of ISIS, the Survey Center was unable to obtain a random sample of seniors meeting the sampling criteria. Thus, the replication of the Senior Survey has been postponed until the spring semester of 2001.

 

Mechanisms for Communicating with the University Community  

The mission of the Verbal Assessment Project includes serving as a source of tools and information related to communication instruction and verbal assessment. Accordingly, verbal assessment activities have included providing information about the Verbal Assessment Project to the university community. These efforts are summarized in the following sections.

 

Newsletters

The Verbal Assessment Committee created a newsletter to provide information about verbal assessment activities. Two issues of the Verbal Assessment Bulletin have been distributed thus far. The first provided information about the long-range verbal assessment plan, introduced readers to the Comm-B study underway at the time of publication, addressed the university=s efforts to meet student demand for seats in Comm-B classes, and summarized Comm-B instruction provided by the UW Library User Education Office. The second issue reviewed the procedures of the Comm-B study, included a detailed summary of the procedures and criteria employed to evaluate writing performance, reported on Comm-B instructor training provided by the L&S Program in Writing Across the Curriculum, and presented the results of the CA100 study of communication, writing, and library anxiety.

The Verbal Assessment Bulletin is circulated to faculty, lecturers, and instructors instrumental in presenting Comm-A and Comm-B courses. In addition, copies of the newsletter are sent to a variety of campus administrators. The Verbal Assessment Bulletin was also shared with representatives of the North Central Association during their visit to the UW campus as part of the 1999 re-accreditation study. Thus, the Verbal Assessment Bulletin has developed as an important and ongoing source of information about verbal assessment efforts.

Information about the Verbal Assessment Project has also been disseminated through articles in Time to Write, the newsletter of the L&S Program for Writing Across the Curriculum. In March 1999, Time to Write carried a lengthy cover story addressing fundamental questions about verbal assessment, reviewing the general goals and specific agenda for the Verbal Assessment Project, and addressing the relationship of verbal assessment activities to the faculty=s ongoing teaching efforts. A follow-up report in the November 1999 issue provided a progress report on the Comm-B study.

In sum, these newsletters have disseminated information about the Verbal Assessment Project to those faculty and instructional staff who are centrally involved in delivering the general education communication courses on this campus.

TA Training

The Verbal Assessment Project has provided direct feedback into the training of teaching assistants in Comm-A and Comm-B courses. Directors of the Comm-A courses in Communication Arts (Professor Stephen Lucas), English (Professor Michael Bernard-Donals), and Agricultural Journalism (Professor Marion Brown) all serve on the Verbal Assessment Committee. Brad Hughes, Director of the L&S Program for Writing Across the Curriculum, is both a member of the Verbal Assessment Committee and responsible for Comm-B Instructor Training Sessions on this campus.

The participation of these individuals on the Verbal Assessment Committee provides a direct avenue for implementing insights gained through verbal assessment efforts. For example, preliminary results from the Comm-B study have been used by Mr. Hughes to supplement training sessions for teaching assistants teaching their first Comm-B class. In particular, comments from the instructor survey portion of that study provide insight into the rewards and challenges of teaching Comm-B that is useful to teachers preparing for their first semester in the course. In addition, Professor Solomon participated in an English 100 staff meeting focused on the development of criteria for evaluating student writing; the procedures and results of the student paper evaluations conducted as part of the Comm-B study were central to this discussion.

Through these channels, the Verbal Assessment Project provides direct and timely feedback into TA training.

Reports to Administrative Committees

Professor Solomon serves on the L&S General Education Committee, the L&S Curriculum Committee, and the University Assessment Council. Thus, she is positioned to provide information derived from the Verbal Assessment Project as it becomes relevant to the activities of these committees. For example, the review of additional communication requirements in the College of Letters and Science (i.e., the �Comm-C2� requirements) conducted by the L&S Curriculum Committee in 1999 was informed by Professor  Solomon=s understanding of enrollment and instructional issues in the Comm-B courses.

As a more direct channel for disseminating information, Professor Solomon presented a summary of the Comm-B assessment efforts to the L&S General Education Committee in fall semester of 1999. Because Professor Sherry Reames, chair of the Communication Implementation Committee, also serves on the L&S General Education Committee, information about the Comm-B study was indirectly channeled to the committee responsible for overseeing the general education communication courses.

Professor Solomon also reported directly to the University Academic Planning Council in December, 2000. This report included a review of the Verbal Assessment Project goals and long-range agenda, as well as a summary of procedures involved in implementing the Comm-B assessment study.

Information about verbal assessment activities focused on information literacy has also been shared with the Library Planning Committee. Abbie Loomis, campus coordinator for Library User Education, serves on the Verbal Assessment Committee. Thus, there are direct lines of communication between the unit primarily responsible for library instruction and the Verbal Assessment Committee. More specifically, Ms. Loomis has shared results from the CA100 study highlighting students= anxieties about using the library with the Library Planning Committee.

In sum, the Verbal Assessment Project has a number of avenues for providing information about general education instruction and outcomes to the relevant policy making committees at the UW-Madison.

Conferences and Colloquia

Professor Solomon has been available for conferences or colloquia focused on more specific aspects of the Verbal Assessment Project. Although the results of the Comm-B study do not allow an evaluation of individual courses or instructors, instructors involved in the study can get information about outcomes in their particular course from Professor Solomon. (For reasons of confidentiality, information about an individual course is not provided to anyone other than the specific instructor who participated in the Comm-B study.) More generally, Professor Solomon has consulted directly with faculty and instructors teaching Comm-B courses to clarify how their courses might better address the objectives highlighted by assessment efforts.

Addressing the challenges inherent in assessment has also been the subject of meetings sponsored by the University Assessment Council. As part of these efforts, Professor Solomon participated in a colloquium on using web-based surveys for assessment purposes. Because student participants in the Verbal Assessment Comm-B study were drawn from a number of courses across campus, the student survey component of that study was implemented using a web-based survey supported by Testing and Evaluation Services. This methodology disseminated the survey quickly to a large number of students, allowed students to complete the survey at their convenience, and directly translated survey responses into a usable data file. Difficulties contacting students and alumni to complete surveys about educational programs are common to all academic units involved in assessment; therefore, the lessons learned from the Verbal Assessment Project are widely applicable.

 

Verbal Assessment Project Agenda: 2001

The long-range plan proposed in the 1997-98 Verbal Assessment Report called for assessment efforts focused on three levels: (a) outcomes associated with the Comm-A course; (b) outcomes associated with the Comm-B course; and (c) outcomes associated with the cumulative undergraduate education. Assessment efforts thus far have most directly targeted outcomes associated with the Comm-B course; therefore, the agenda for the near future involves attending to the other outcomes levels. In addition, the Verbal Assessment Committee will strive to improve the dissemination of verbal assessment results and resources to the university community. In total, the verbal assessment agenda for the next year includes the following four activities.  

Senior Survey

The Senior Survey, originally scheduled for the spring of 2000, will be implemented in the second semester of the 2000-01 academic year. This survey will assess attitudes and beliefs relevant to public speaking, writing, and library use among students who matriculated under the general education communication requirements. Thus, a comparison of these data to the baselines established by the 1998 survey will shed light on the overall impact of the general education communication program on students= attitudes and beliefs about communication skills.

Communication A Study

An assessment study focused on the Comm-A course will be developed during the 2000-01 academic year for implementation in the fall of 2001. The implementation of the Comm-A course has been relatively smooth; therefore, assessment efforts thus far have focused on outcomes associated with the general education requirements overall and the Comm-B course in particular. With studies addressing those outcomes completed, the Verbal Assessment Committee will examine teaching and learning in the context of the Comm-A course.

Verbal Assessment Bulletin

Thus far, two issues of the Verbal Assessment Bulletin have been produced at a rate of one per year. Because this newsletter constitutes a primary means for communicating with faculty, instructors, and administrators involved in the general education communication program, increasing and regularizing production of the Verbal Assessment Bulletin is warranted. At present, two issues of the newsletter are planned for the 2000-01 academic year.  

Verbal Assessment Toolkit

Part of the value of the Verbal Assessment Project is the development of tools that can be used by other programs, departments, or instructors for assessment purposes. For example, measures of library anxiety  that were developed in the 1998 Senior Survey and refined in the CA100 study could easily be used to assess the effectiveness of other programs targeting the development of library research skills. Similarly, the survey developed as part of the Comm-B study to assess students� evaluations of their Comm-B classes could be used by individual instructors in their own courses. Likewise, the writing performance criteria developed by the paper readers in the Comm-B study could be implemented as a pedagogical tool in any class with a substantial writing component. Thus, verbal assessment efforts in the coming year will include assembling assessment instruments into a toolkit available to other units on campus.

 

Directions for Future Efforts

Assessment efforts beyond 2001 will necessarily be informed by the results of the studies previously conducted or planned. In addition, studies that target particular communication instruction objectives (e.g., oral communication skills) are needed to supplement the general picture that should emerge from the research conducted thus far. Ultimately, the Verbal Assessment Project will evolve into a balanced mix of studies that combine to support generalizable conclusions about the implementation of the general education communication program and to provide specific insight into the effective operation of the courses comprising the general education communication requirements.  

  *****

Appendix A - The Context for Verbal Assessment at UW-Madison

Appendix B - Spring 1999 Communication-B Study 

Appendix C - Communication Arts 100 Study