Spring 1999
Vol 7 Issue 2
IN THIS ISSUE...
Student Academic Achievement Plan (SAAP)
SEE ALSO...
The Forum
Maricopa Center for Learning and Instruction
The Student Academic Achievement Plan (SAAP)
Kathleen Iudicello and Alex M. Joncas, EMCC
Introduction
Morey and Kitano argue that "Assessment procedures should include methods that accommodate students' strongest strategies for expression of accumulated knowledge and skills"(p. 16). To be fully comprehensive, academic assessment must acknowledge and address the diversity of the students being assessed.
At Estrella Mountain, the Student Academic Achievement Plan (SAAP) identifies three academic program areas for assessment: general education/transfer education, developmental education, and workforce development.
The purpose of SAAP is to promote continued excellence in teaching and learning by improving and enhancing student abilities and success, determining achievement of student abilities, measuring the effectiveness of student abilities, and using assessment as a tool for feedback and learning. Both the assessment and implementation of abilities must constantly consider from where students are coming and aspects they bring with them as significant parts of their identities. "Diverse students' academic achievement is a necessary part of their empowerment . . ." (Sleeter qtd. in Morey and Kitano, p. 14). All these issues have a profound impact upon whether the assessment is appropriate or the implementation strategies are productive as effective ways of measuring and teaching abilities.
Faculty and staff comprise the Student Academic Achievement Committee (SAAC). The committee's purpose is to determine the number of abilities and clarify, define, and promote the abilities, which facilitate communication between programs regarding assessment efforts. SAAC seeks student commitment to the assessment effort and disseminates and limits the use of assessment data based on sub-committee recommendations. Finally, SAAC responds to and ratifies recommendations of the SAAC Steering Team.
At the present time the abilities of communication and critical thinking are assessed at the academic program level. Faculty have defined the abilities of critical thinking and communication in objective and measurable terms.
The SAAP is linked to the first four education related goals of the college and is designed to measure learning in the three primary academic program although only the GrEAT pilot assessment will be detailed here.
The GrEAT Assessment
Each member of the GrEAT [General Education Assessment Team] ranked assessment tools and selected the Educational Testing Services' Tasks In Critical Thinking as the top choice because it appeared this tool measured the abilities of critical thinking and communication as defined by faculty. Their efforts were aided as the result of a partnership between the Maricopa Center for Learning and Instruction (MCLI) and Estrella Mountain faculty and staff. An instructional designer was assigned to work one-on-one and in small groups with assessment teams two days each week at EMCC.
The test used categories of social sciences, humanities, and natural sciences. This made the Tasks interesting and relevant to students in the General Education/Transfer Academic Programs. The ETS instrument was also beneficial because it was performance-based, it resembled what students are required to do in the classroom and the world of work, and the instrument could be scored by trained faculty or returned to ETS for scoring.
This tool required students to read, compile and analyze data and to write a response. Using the disclosed version of the test, the team then matched the Estrella Mountain abilities to be measured to the tool itself.
In collaboration with the Office of Institutional Planning, the cohort wasidentified as students who had completed 15-18 hours in the general education core plus 3 distribution hours. This cohort comprised approximately 130 to 150 individuals.
Faculty met with students to learn what it would take to involve them in assessment efforts. The students were excited to be asked; they saw value in the idea of assessment and provided valuable input as to incentives and the need to schedule assessment for their convenience. In addition, students stressed that the assessment should not be connected to their grades in courses, or to their ability to graduate, transfer, or receive a certificate. It should be anonymous. Additional suggestions on the way to get student involvement were sought from faculty at the spring semester faculty orientation meetings.
GrEAT, other designated faculty, and support staff then developed a marketing approach to alert students to assessment. The approach included the writing of key words on all classroom boards, the ordering and wearing of assessment buttons for faculty and staff which read, "Put Your Abilities on the Line", and placing information about abilities on campus student computer screens as screen savers and on fliers hung in every classroom.
The Testing Center was selected as the site for the testing. Letters, signed by all general education faculty, were mailed to the 137 students invited to the assessment. Follow-up phone calls were made a few days later to answer student questions and to encourage participation. The assessment was given during Abilities Assessment Week, April 6-10, 1998. $20.00 stipends for participating students and a scholarship drawing were arranged.
Forty-two students (30.6% of the 137 identified students) took the assessment. Most spent about 90 to 120 minutes taking the test.
The Analysis
Faculty scorers were trained by ETS. A faculty member did an in-house analysis of the results. Faculty received hourly compensation for their participation.
A faculty scorer noted that the scoring process confirmed that she and her colleagues across disciplines were in fact evaluating students at the same level and with the same high standards. Faculty reported that the scoring process also affirmed that the abilities needed to complete the tasks actually reflected students' ability to think critically and to communicate effectively.
Faculty agreed that the Tasks were not a trivial assessment; they all liked the performance component of the instrument, and they all believed that the assessment was a good reflection of the types of abilities they have tried to help their students develop. From the responses generated by the students, both in terms of depth and breadth, the faculty felt that the students took the assessment seriously.
Analysis of the demographic data collected indicated that those students who chose to take the assessment were representative of the identified cohort.
Preliminary data analysis suggested that the Tasks are appropriate for the student population and that the Tasks discriminate the degree of development for critical thinking and communication of the students tested. Based on the data analysis, faculty concluded that the Tasks in Critical Thinking provided them with the information they were seeking in terms of assessing their students' abilities.
Each Task in the assessment tool had a variable number of questions. During scoring, each question was assigned to one of three categories for analysis: inquiry, analysis, or communication. Inquiry and analysis represent the categories reflecting the critical thinking ability. Each graph compares the percent of answers (not students) at each score. That is, answers are individually assessed rather than individual students. This methodology gives a better reflection of how the student population is doing in each category.
Scoring rubrics were assigned by ETS. The core score represents an answer containing all the basic requirements based on ETS criteria. Answers above the core score included additional correct information. Answers below the core score included less than acceptable amounts of information. Faculty and staff decided to protect assessment data and avoided making sweeping generalizations about individual student's academic achievement and his/her test results.
All three curves or categories show a bell curve with the highest number of answers clustered near the core score of 4 which is a minimal answer showing full proficiency. In inquiry, 50% of the answers scored at or above the core score. In Analysis, 52% of the answers scored at or above the core score. In communication, 46% of the answers scored at or above the core score.
This wide bell curve distribution suggests that as additional cohort assessments are done in the future and compared to previous cohort answers that there is an opportunity to see improvement or deterioration of scores; therefore, it can be said that the assessment tool is appropriate for our population and can reflect a broad range of abilities. We have learned that there is "room for improvement" in the teaching and learning of inquiry, analysis, and communication.
Overall Conclusions
Faculty and staff view the assessment initiative at Estrella Mountain as becoming on-going implementation. The assessment effort is faculty owned and driven. Senior Administration actively support the assessment program and recognize and reward faculty's efforts to implement it. Results from measuring general education/transfer, developmental education, and workforce development academic programs are being collected and interpreted. Information about assessment is being disseminated to various constituencies. Evidence of specific strategies faculty are using to improve student learning is being gathered and documented. Evidence of specific strategies divisions are using to assist faculty in improving student learning is being gathered and documented. Decisions about changing or refining methodology or measures are being made and multiple measures are being researched. Clear feedback loops are being developed to ensure that the results of assessment are used to improve student learning and teaching.
In addition, the relationship of strategic planning, institutional effectiveness and assessment of student academic achievement has been clarified. A model for Program Review that incorporates components of institutional effectiveness outcomes, program specific outcomes, and student learning outcomes has been designed.
Where appropriate, yearly assessments will be done of critical thinking and communication in General Education/Transfer, Developmental Education, and Workforce Development. Program Reviews will be developed, piloted and conducted as appropriate. Faculty and staff will determine the process and model for feedback loops and regular assessment progress reports and annual assessment reports as well as for a yearly assessment report card.
As a result of all of these efforts, the entire assessment program and process will be regularly evaluated and refined. Faculty and staff will work to incorporate the Maricopa Community College Governing Board End Statements into the assessment model. This will provide valuable input to the district's efforts to define and assess these outcomes.
The next formidable task will be to conduct faculty workshops in partnership with the Estrella Mountain Center for Teaching and Learning on how to implement abilities in the classroom with a full appreciation of the issues of diversity that exist with that classroom. The challenges that such discussions will bring to both the assessment plan and the implementation process will contribute to the continuous improvement of diversity awareness and teaching and learning at Estrella Mountain.