After two years of collecting completed forms, the forms were divided into two groups: 'purchased' software and 'rejected' software. Evaluation forms without comments were not used. This left 43 completed reviews of purchased CAI programs and 34 completed reviews of rejected CAI programs. Comments, consisting of phrases and sentences on the evaluation forms, were placed into two categories: forms for recommending and forms for declining purchase. Within these categories comments were sorted into related groups. Illegible comments and comments with ambiguous meanings were not recorded. This resulted in four comments from the purchased group being eliminated.

Each comment on the evaluation form was read and placed into a group. For example, if a comment was 'good graphics for student use' this was placed into the group about 'good graphics/images/sound.' The recording and tallying of all comments was done for both groups.

There were 107 comments for the 43 programs that were recommended and 41 comments for the 34 programs that were rejected. Comments for all previewed and purchased titles and then all previewed and rejected titles were sorted into related groups, counted and evaluated.

Evaluation of comments about purchased CAI programs
Groups for purchased software consisted of comments about content, image/sound quality, test or quiz features, navigability, appropriateness for course, test or board review preparation, critical thinking skills, speed, balance of images with text, currency of material, search capabilities, references, ability to compliment other course material and level of fun.

Analysis of comments for purchased CAI programs

• 32 (29.9%*) Content quality. Comments on the program's content. These were statements about content quality (ex. 'Very well done case study with rich details.'), accuracy (ex. 'The information appears to be accurate and up-to-date.'), and clarity (ex. 'Clear explanations of complex information with focus on nursing role.'). These statements show that the faculty member thought the program succeeded in effectively conveying the scope of the topic.
• 18 (16.9%*) Image use/quality. Comments on the quality, number, or appropriate use of images/graphics/videos/sound files. (ex. 'Excellent images.').
• 12 (11.2%*) Exam/quiz features. Comments on quizzes, exam questions, or other test features included in the software. (ex. 'Very good exam.').
• 12 (11.2%*) Navigability. Comments on navigability (ex. 'Easy to use.' 'User-friendly.').
• 10 (9.3%*) Appropriate content for course/grade level. Comments on content being appropriate for a specific course (ex. 'Case studies appear to have merit for our current course N5000.' ), grade level (ex.'Will recommend to 2nd year medical students.'), or group of learners.
• 5 (4.7%*) Appropriate content for board/test review. Comments on content being suitable for board review, course review or test preparation. (ex. 'Good review for didactic classroom material.').
• 4 (3.7%*) Promotes learning. Comments on how CAI promotes learning or critical thinking (ex. 'Using this program will be valuable in promoting students' use of critical thinking skills & applying knowledge from the texts.').
• 4 (3.7%*) Balanced content. Comments on good balance of text/images.
• 3 (2.8%*) Speed issues. Comments on speed (ex. 'Program or sections load quickly.').
• 2 (1.9%*) Current awareness. Comments on content containing current material.
• 2 (1.9%*) Search function. Comments on search capabilities (ex. 'Good search capacity.' & 'Search methods only fair.').
• 2 (1.9%*) Lack of references. Comments noting the lack of references within the content.
• 1 (0.9%*) Integrative course material. Comment on how the CAI program compliments other information sources (lectures, textbooks and assignments).
• 1 (0.9%*) Fun. Comment on how program was fun to use.

Table 1. Purchased CAI Program Comments

Number of Comments	Concepts
32	Content Quality
18	Image Quality/Use
12	Exam/Quiz Features
12	Navigability
10	Course/Grade Level
5	Board/Test Review
4	Promotes Learning
4	Balanced Image/Text
3	Speed
2	Current Awareness
2	Search Function
1	Integrative Course Material
1	Fun to Use

• 11 (26.8%*) Content quality. Comments on content's quality, accuracy or clarity. (ex. 'Not consistent with current guidelines.', ex. 'Omissions are numerous.').
• 7 (17.0%*) Interactivity. Comments on levels of interactivity (ex. 'No student interaction.').
• 7 (17.0%*) Image use. Comments on inappropriate number of images/video (6 comments said too few, 1 comment said too many).
• 6 (14.6%*) Inappropriate content. Comments on content being inappropriate for intended audience (ex. 'Inadequate even for undergraduate anatomy.').
• 3 (7.3%*) Navigability. Comments on navigation issues (ex. 'Too difficult to progress, change course and get out of program.').
• 3 (7.3%*) Image quality. Comments on poor-quality images and/or screen difficult to read.
• 2 (4.9%*) Speed. Comments on speed (ex.'Slow.').
• 1 (2.4%*) Search function. Comment on lack of a search function.
• 1 (2.4%*) Instructional value. Comment on program having limited instructional value.

Table 2. Rejected CAI Program Comments

Number of Comments	Concepts
11	Content Quality
7	Interactivity
7	Image Use
6	Inappropriate content
3	Navigability
3	Image Quality
2	Speed
1	Search Function
1	Instructional Value

Hardin & Patrick (1998 p.208) divided software evaluation methods into three areas: practical aspects, content and empirical data. The factors that faculty used to determine a program's instructional value consisted of "content" and "practical aspects."

Content issues concern the scope of a topic and effectiveness of the CAI program in presenting the material to the intended audience. The vast majority (%) of comments for both purchased and rejected CAI programs referred to issues of content. There were 148 total comments between the purchased and rejected evaluation forms. Of these, 118 (79.7%) (91 purchase, 27 reject) referred to the program's content. Thirty comments were on practical aspects (16 purchase, 14 reject).

This data suggests that faculty members consider the content of the program to be the most important aspect when considering the suitability of CAI software for a course. For purchased software, content quality & accuracy was mentioned the most (32 times), followed by use of images/graphics/video/sound (18), quiz/test/exam features (12) and content appropriate for specific course/grade level (10). In contrast, practical aspects generated only 16 total comments, with the most (12) on the CAI program's 'user-friendliness' or navigation issues.

The focus on content as the most important aspect also can be seen when reviewing the rejected CAI comments. Reviewers noted the lack of essential knowledge/information (6), inappropriate content for user (6), lack of good images (6) and found content inconsistent with current standards/practices (4). Navigation issues again made up the majority of practical comments (3), followed by speed (2), lack of a search function (1) and a difficult to read screen display (1).

To conclude, comments made by health sciences faculty at the University of Minnesota indicate that content issues are the most important factor when previewing CAI programs for curriculum use, followed by image quality and then navigability.

Glenn, Jerry. 1996. "A Consumer-Oriented Model for Evaluating Computer-assisted Instructional Materials for Medical Education." Academic Medicine 71:251-255.

Hardin, Laura, and Timothy B. Patrick. 1998. "Content Review of Medical Educational Software Assessments." Medical Teacher 20:207-211.

Huber, Jeffrey T., and Nunzia B. Giuse. 1995. "Educational Software Evaluation Process." Journal of the American Medical Informatics Association 2:295-296.

Miller, Judith and Frederic Wolf. 1996. "Strategies for Integrating Computer-based Activities Into Your Educational Environment: A Practical Guide." Journal of the American Medical Informatics Associtation 3:112-117.

FORM 1

This is the percentage of the specified group (i.e. content) to all comments in the respective (purchased rejected columns).

ISSN 1069-6792
Revised: 10/5/99
URL: http://wings.buffalo.edu/publications/mcjrnl/v6n2/hslcai.html