FACILLITATOR(S):
Doug Sawyer (SCC) & Ted Wolter (CGC)

QUESTION 1:
What is the definition of Assessment and Evaluation ?

• Comparison to national statistics on technology issues measuring the effectiveness of technology toward teaching and learning.
• Measuring faculty, staff, and student satisfaction with technology.
• a measure of how effective an institution is in providing a learning environment.
• Currently it means placement testing.
• little agreement, and therefore considerable confusion around the meaning of these terms. The terms appear to be used interchangeable, along with "testing" and "measurement." It has also been used to refer to "institutional effectiveness."
• ability to measure student learning, student satisfaction, effectiveness of academic and student services programs.
• Assessment can provide the opportunity for improvement.
• Technology as it supports student learning
• Technology as it supports staff development
• Processes that establish measurable goals and objectives for student learning, course design and content, faculty, and colleges by identifying a starting point (prior knowledge, etc.)
• Processes that follow a continuous improvement cycle of objectives, benchmarks, data collection and analysis, improvements, feedback, and integration
• Though some institutions use these synonymously. here, it is viewed as assessing programs, courses, etc, and evaluation as evaluating the student, the teacher (i.e. the personal aspect).
• a la Webster's:
  • evaluation: the act of examining carefully; appraisal
  • assessment: the act of appraising or evaluating
• a la Brainstorming session
  • assessment: measurement of
  • evaluation: includes a judgment (against some criteria)
• Use of technology for Assessment and Evaluation, for example, using technology in the Assessment of Student Academic Achievement
  • Electronic portfolio
• Assessment and evaluation of the use of technology to enhance learning
  • Electronic portfolio
  • pre/post tests
  • Impact of different technologies
• Assessment and evaluation of Technology Plan
• We view assessment and evaluation as two different but related activities. Assessment refers to collecting, analyzing, and using data related to the measurement of student learning, in this case, student learning that results from some type of instructional technology. Evaluation refers to examining the broad goals of whole programs or interventions (for example, perhaps, an entire degree offered via web-based delivery), and making judgments about their merit, worth or effectiveness. Therefore, assessment is a critical sub component of evaluation. Assessment results are used in combination with other components (for example, analysis of the context of the program being evaluated, program costs, or people's attitudes toward the program) in conducting an evaluation of the program's effectiveness.
• the act of measuring how well we are doing at the institutional and classroom levels at fulfilling our mission goals as a college. We are involved in both program level review (presently three areas are being evaluated- General Education, Organizational Leadership, and Developmental Education), and on going classroom-level assessment.
• Accessing and evaluating learning effectiveness
• how good are the tools
• learning efficiently, effectively
• understanding of what technology tools are available
• how do we integrate the assessment of technology with other assessmen programs
• technology should be convenient and unobstructive (invisible)

• Who will put together the assessment tools?
• Shared understanding of terms.
• Providing compelling reasons/motivation for changing current practices.
• Developing a coherent understanding of current and new assessment and evaluation practices, i.e. we often assume to know "what is" based exclusively on anecdotal indicators that may be inaccurate and thereby misleading.
• Changing learner, administrator and faculty attitudes towards assessment and evaluation.
• Development of an institutional culture in which dialogue and "learning about learning" is valued, recognized and rewarded.
• How can we measure the effectiveness of an institution in providing a learning environment?
• Assessment should contain more of an educational plan with testing as one of many pieces. Educational goals need to be established before any real evaluation can take place.
• Measuring effectiveness of out-of-class learning experiences.
• Do currently practiced standardized assessment really work? Do test scores tell you anything about what a person has learned? Measurement should not be just quantitative but qualitative as well.
• We have no system in place for student learning and staff/development
• Funding
• Human resources
• Wide disparity of students and facultyÑcultural-expertise- resources and knowledge base
• Entry level assessment of students
• Interest on campus
• How does IT change learning outcomes?
• How to measure the effectiveness of teaching methods?
• What are the best strategies to achieve goals?
• How can we assess when there is not enough research to base assessment on, make comparisons?
• How are classroom research and web folios used for assessment?
• How can we move towards measuring learning?
• How do we determine if IT is effective?
• How to decide upon measurable objectives, how to measure, how to collect data, how to make effective improvements, how to give feedback, how to integrate into the college
• The semantics (Assessment often thought to relate to students and evaluation to program)
• How do make sense of the data we have and improve the data collection process in the future.
• Lack of systematic documentation
• Assessment not defined up front
• Defining and articulating the desired outcomes of IT.
• Developing a plan that can be implemented in the real world.
• Determining which are the most important questions to ask about IT, from a seemingly infinite number of potential questions.
• The pace of change in IT creates an atmosphere of uncertainty about where to invest time in assessment and evaluation that will provide information that is relevant in the long term.
• Maricopa does not have a history of using assessment and evaluation in other arenas, so it is not part of the culture.
• Most critical is what exactly do we measure, how do we measure it, how do we evaluate it, and what do we do with the results? Also, who controls the process; it should be faculty. Additionally, District program descriptions and competencies not being very accurate can make it harder to assess the actual program or class as it exists at a specific college.
• no good models to assess the role technology plays in learning
• too busy doing, we aren't pausing to see how we've done before we move on to the next
• communication don't share what has been tried
• effective dissemination of information
• need to have place where people can go to get it (what's been tried before)
• need clear long-range plan--a technology plan that everyone understands
• we have elements--no comprehensive plan to help with assessment and evaluation
• some things are not in our hands--not everyone is participant of plan
• district imposed factors
• as we set our plan technology may become obsolete
• outside influences

• Have used learning theory to focus my A&E efforts.
• Have engaged learners in dialogue about A&E.
• Have involved learners in expanding A&E roles.
• Have identified key factors concerning A&E (Who, what, when, how)
• Organizational conditions (calendar, class times) often mitigate against good practices related to A&E.
• We have found that none of the traditional forms of assessment measure the effectiveness of institutions in providing a learning environment. They tend to focus on easily quantifiable traits such as number of students who graduate, number of students who go on to a four-year institution, etc. These don't address students who are not after a degree nor do they truly measure the ability of students to learn.
• Getting better, need LCS or equiv.
• Trying to carve out the time to institutionalize a systemic A&E plan.
• It is a tedious, time-consuming task. It helps point out our faults as well as our strengths.
• So/soÑComputer literate student base
• Opportunity for on campus training for staff and students
• Software versions?
• Turn out today? Lack of interest? Another meeting?
• 8 years of Student Achievement Committee, 3rd year of Student Achievement report, Assessment of Prior learning
• One of the workgroups is focused on Assessment and Evaluation; the activities of the group are posted on the website. The Office of Research and Planning is working with a faculty member who is working with the Web development group. This has resulted in development of a set of broad guidelines that incorporate assessment into the course development checklist for Web-based courses. This process involves three levels of assessment and evaluation: assessment of student learning of course competencies; survey of students participating in web-based courses administered upon completion of the course; and ultimately, a comprehensive plan to evaluate a web-based AA program. We are in the beginning stages of most of this activity, although the student survey has been used by one faculty member and others are adapting it for their courses.
• Other workgroups are also building an assessment and evaluation piece into their respective plans. For example, the group working on the Student Technical Assistant program will evaluate its goals for that program.
• Sometimes assessment is written off as one more creation of some accrediting group somewhere that requires lip service at best. Getting full participation is most difficult. Also, sometimes administrators are not fully committed to the concept of assessment being faculty owned and operated. Also finding tools to measure assessment is difficult as each campus is unique and national tests tend to force colleges who use them to adjust their mission to match the test. Everyone tends to have their own ideas about what assessment is, and how to do it, and without strict adherence and collaboration, results can be less than useful.
• lack of feedback
• what is the outcome
• never formally linked assessment and evaluation with on-going processes. Example: no effectiveness in past capital request--link with capital request and technology plan--how do we evaluate
• at an institutional level critical feedback occurs when things are not working well
• classroom--praise, feedback is usually good
• we don't ask students if the level of technology is adequate
• assess and evaluate things we can impact
• there are barriers with technology use especially part time (instructors and staff)
• impossible to fill technology needs
• incremental computing power in the last few years
• must match technology with need-->reality-based

• There is no organized effort to gather information about it here.
• Division chairs, DI and institutional researcher have initiated a pilot assessment process.
• Day-long discussion at a recent campus meeting.
• Inadequate.
• If we are talking about A&E for students, then I feel we are not currently doing a good job of measuring and meeting the students and/or employer's goals. Some of us meet and/or teach the course's competencies but we do not do enough to go beyond those. We should all be trying to do more.
• Not in place. There is zero status. WE are in exploration stages.
• Active committee that is faculty driven, and has an increasing role in assessment and evaluation.
• We are working through the issues stated above. We are looking for methods other institutions. And, we are trying out some things on a small scale.
• Here, most faculty are seriously buying in to the pilot. General Education, Organizational Leadership and Developmental Education are involved presently in program level assessment. Organizational Leadership is using a web based feedback form in their assessment efforts.
• fragmented occurring
• NCA has provided input into a more comprehensive plan (institutional Effectiveness)--components with technology
• need to do more in terms of how technology relates to learning
• NCA-- important to get it out and disseminate
• is institutional effectiveness evaluation/assessment of technology integrated?

• GPA, average class scores, learner evaluation of A&E components of course experience.
• Data is hard to get to, hard to evaluate, focused on FTSE, and is usually unusable. -
• We need LCS or equivalent.
• We have begun with the Noel-Levits student satisfaction survey and are also in the process of developing measures for each of the student life areas.
• We could collect data on whether the course met the student's goals and objectives; whether they enjoyed the format, or does it need to be changed.
• Instructional grantsÑevaluation tools
• SurveysÑfaculty and students in classrooms
• Evaluation of new facultyÑtechnology in the classroom
• Make people accountable
• Currently track head count by month
• Overall usage data a) order software b) Media center c) reservations in lab
• Currently have a tracking on server with regards to applications start up (ex. Netscape or TELNET)
• BenchmarkingÑother colleges and high schools
• We collect 2 kinds of data; direct indicators of student learning based on desired competencies related to a college education and on the Governing Board's ends, pre/post test, etc. and indirect indicators (completion rates satisfaction rates, etc.)
• Suggest a matrix for each item considered:
• Training costs and time (people costs)
• Cash/$$$ spend
• Ease of access
• Amount and kind of support needed Hardware Software Continuing Design Content
• Frequency of use
• How many benefit
• Chart for Assessing/Evaluating Technology enhancing learning

           Reasonable costs
           (in terms of time & dollars)
  
          if yes                 if no   --->  stop
            	|
            	|
     individual pilot
          	|
          effective??             If no --->  stop
              |
          	|                   ---> how determined
  	        |                   ---> develop criteria
         if yes                          |
          	|                   test criteria
          	|____________<---------|
                   ____|___          |
                     Broaden
                                     |
                         	      _____|_____
  		                    Status Quo
  

• Classroom level data can be collected daily, weekly, monthly or by semester. Campus assessments should be done only yearly. We can learn how effective our teaching is of a particular days lesson or a multi-week unit of learning or measure the semesters learning. We can measure how well students have learned as they prepare to leave us. Also, in occupational programs, check student career goals attainment and learning against what the faculty want a course to do can be effective.
• staff, faculty, everyone changes in use of technology, needs analysis, knowledge of software. Example: work with faculty--to evaluate classroom use--campus-wide disseminate information
• from students--what kinds of computer, software, etc., technology do students use at home?
• measurement of increase technology use since coming to campus
• demographic information regarding students