A "Flashlight" for Evaluating Technology
Andrea Greene
MCLI

Flashlight is a national project involving five institutions of higher education and coordinated through Western Interstate Commission for Higher Education (WICHE) that has as its purpose the development of a "toolkit" of instruments that can be used to evaluate the impact of technology on higher education. The project was originally funded by FIPSE and subsequently funded by Annenberg/CPB Foundation, with in-kind contributions from the participating institutions. When completed, the toolkit will consist of surveys and interview protocols for students, faculty, alumni, and employers. The survey item bank and interview questions are designed to be applicable for a wide range of questions that might be of interest to colleges and universities.

The Maricopa Community Colleges have been participating in the Flashlight Project since its inception in 1992-93, primarily through the involvement of the Maricopa Center for Learning and Instruction (MCLI). In fact, the concept for Flashlight was an outgrowth of discussions between one of Maricopa's Ocotillo committees and Steve Ehrmann (formerly of Annenberg/CPB, now with American Association for Higher Education -- AAHE). The MCLI helped conceptualize the project framework and develop survey items for the item bank.

In Spring 1996, when the student survey item bank was ready for beta testing, MCCCD was one of the sites that field tested items. Two customized surveys were designed using questions from the item bank: one survey for Electronic Forum which is used in English 101 and 102 courses and one survey for graphing calculators which are used in College Algebra courses (Math 154 and 155). Items were selected based upon how well they appeared to match some preliminary questions about technology that had been posed by MCCCD participants. The intent of the field test was to test the items in the item bank rather than to evaluate the two types of technology.

Fifteen faculty from five colleges who teach the appropriate courses volunteered to participate by distributing surveys to their classes as an optional take-home assignment. Approximately 250 of each version of the survey were distributed; response rates were 57% for the English survey and 77% for the math survey. Data were entered into the database template provided by WICHE. Local data were transferred electronically to WICHE for the large scale item analysis and also kept at the MCLI for local analysis. MCLI staff have made presentations about MCCCD's experience using the Flashlight student survey at two national conferences last summer: the AAHE--Teaching Learning Technology Roundtable (TLTR) conference in Scottsdale and the Seminars on Academic Computing conference in Snowmass, Colorado.

What Did We Learn From the Student Survey Beta Test?
The results of the student survey beta test provide an example of the type of information that can be obtained about the use of various technologies in MCCCD. An item-by-item analysis was conducted and these detailed results are available through the MCLI. This article highlights selected findings that illustrate areas that were addressed with the surveys. It is hoped that these results will prompt discussion regarding possible future uses of Flashlight and also issues that may need to be addressed before any large scale surveys are conducted. Below are a few highlights from the surveys.

Highlights from the Mathematics Survey

• The vast majority (88%) of students completing the survey plan to earn a bachelor's degree or beyond.
• 75% of students enter college algebra with more than 2 years experience using a computer; 41% have at least 5 years experience.
• More than half of students responding agreed or strongly agreed that, compared to other courses, the course using graphing calculators makes them:
• better able to apply learning to problem solving at work, home, school (50%)
• more likely to complete assignments on time (56%)
• more confident they can reach academic goals (56%)
• better able to visualize ideas and concepts taught in the course (71%)
• Large majorities of students agreed or strongly agreed with the following statements about the course. The graphing calculator helped them...
• understand and retain more about the course (74%)
• graphing calculator helped them learn to work through a process to solve problems (75%)
• improve their productivity and the quality of their learning (66%)
• 65% of students disagreed or strongly disagreed with the statement that an emphasis on groups hindered them from learning in the way they prefer.
• 30% of students agreed or strongly agreed that their anxiety about using technology hurt their performance in the class.
• 61% of students would recommend the course to others.
• 72% of students were satisfied or very satisfied with the course.

• 79% of students responding are 22 years old or younger; 61% are 19 or younger.
• 86% of students plan to earn a bachelor's degree or higher.
• 50% of students have 5 or more years of experience using a computer.
• 77% of students have their own computer at home; 58% have an IBM/PC
• More than half of the respondents said that, in the EF course, they are more likely or much more likely to:
• discuss subject matter with other students
• work on projects with other students
• receive detailed comments from their instructor
• receive instructor comments more quickly
• The majority of students do not agree that:
• they are spending too much time learning to use the technology (58%)
• they are at a disadvantage because they lack computer skills (62%)
• they spend too much time trying to gain access to a computer (68%)
• they spend too much time trying to log on to the college computer (65%)
• 78% of the students would recommend the course to others; 83% are satisfied or very satisfied with the course using EF
• About one-quarter of students agree that they waste time communicating electronically about topics not related to their course work.
• 21% feel they spend too much time sorting through their messages.
• The majority of respondents agree or strongly agree that the course:
• assignments encouraged them to exercise their creativity (67%)
• helped them manage large tasks (65%)
• helped them work through the problem solving process (59%)