The Resource Guide walks you through our assessment framework and each step of the assessment cycle. It includes a glossary and several helpful resources.
The purpose of the Assessment Resource Guide is to assist Student Affairs units in creating a shared process of assessing student learning and building on the many ways we are already doing this work. It will assist units in identifying their functional areas, developing associated learning and operational outcomes, articulating related strategies, and aligning them with broader learning goals.
This Resource Guide is designed to help each unit in Student Affairs develop a common framework for assessment in the division. The Guide is organized by the elements of the assessment cycle:
- Mission and Purpose
- Outcomes
- Strategies and Measures
- Gather Data
- Interpret Evidence
- Make Decisions to Improve
Each element of the cycle includes detailed descriptions and resources to help you through the process. A glossary is included to ensure we are all using the same definitions of assessment terms and a list of resources and other references expands the basic information included in the guide.
Guiding Principles
As we begin this work, it is important to clarify some underlying assumptions about assessment in Student Affairs. First,
Students learn outside the classroom and Student Affairs staff are educators
Our core values articulate, "[USD] is committed to the dignity and fullest development of the whole person." This underscores the fact that students are learning all the time and as Student Affairs professionals, we are responsible for nuturing that learning in intentional ways. Second,
It's all about the students and what they are learning
Assessment can have negative connotation for many reasons. At its best, it helps us to keep our eye on the ball – our students and what they are learning through their experience with our programs and services. Being learning-centered allows us to shed our skepticism, subtext, and ties to strategies that may or may not be in the best interest of our students. It broadens the concept of accountability away from “performance” toward student learning. Finally,
Assessment is an investment
Assessment takes time and effort just like any good investment. If we can view it in this way, that the time and effort we invest in this process will lead to deeper learning for our students, we can justify that time and effort in support of our mission.
Definition of Assessment
“Assessment is any effort to gather, analyze, and interpret evidence which describes institutional, divisional, or agency effectiveness” (Schuh & Upcraft, 2001, p. 3-4).
This definition of assessment captures a broad perspective. There are many types of assessment that together present the whole picture of what students are learning along with the effectiveness of our programs and services. Types of assessment include tracking usage, needs assessment, satisfaction, climate/culture assessment, outcomes assessment, resource effectiveness studies, benchmarking, program review, strategic planning. While these different types vary in the level of complexity and integration it is important to understand what each type can tell us and decipher which type is appropriate to use in differing situations. The Assessment Continuum below along with the definitions of the types of assessment demonstrate this.
While there are many types of assessment, we will mainly focus on outcomes assessment for the purpose of this document and our work within the division. The following definition of outcomes assessment captures the ideals of assessment where the express intent is to improve our programs and services for the sole purpose of enhancing student learning.
A Definition of Outcomes Assessment
the ongoing process of establishing clear, measurable expected outcomes of student learning; ensuring that students have sufficient opportunities to achieve those outcomes; systematically gathering, analyzing, and interpreting evidence to determine how well student learning matches our expectations; using the resulting information to understand and improve student learning. (Suskie, 2009, p.4)
| Types of Assessment | Definition |
|---|---|
| Tracking usage | This is keeping track of programs, projects, students served. Examples include tracking the number of students going through a dining hall at a particular point in time or tracking how many social programs are done in the halls in a semester. |
| Needs assessment | This is trying to understand what needs are. Examples include determining the social support needs students of color may have on a predominantly white campus or understanding the academic support first generation college students require. |
| Satisfaction studies | These are studies that simply seek to measure satisfaction. Examples include understanding how satisfied students are with their residence hall or how satisfied parents were with parent orientation programs. |
| Climate/culture assessment | These are assessments done to understand the culture and its impact. Examples include understanding the impact on a campus for women who have limited space to meet or the impact on a campus of the installation of gender-neutral housing options. |
| Outcomes assessment | This is the assessment of some type of intended outcome. Oftentimes the outcome is a learning or development outcome, but it can also be a program outcome. Examples include evaluating leadership skills acquired by a leadership course or alcohol use after a new alcohol policy on campus has been implemented. |
| Resource effectiveness studies | These are studies that assess resources used for a variety of programs and services. Examples include reviewing the student fees budget for campus concerts or understanding if there is enough usage of the library at early morning hours to keep it open. |
| Benchmarking | These are studies that involve comparisons such as comparisons to other institutions, professional standards, or the institution itself over time. Examples include the Educational Benchmarking Institute study of Residence Life or comparison of the number of student activities staff per students for institutions of comparable size or review of career services using the National Association of Colleges and Employers standards. |
| Program review | This is a review of an entire unit, not just a one-time program. Examples include a comprehensive review of a residential life office or review of a women’s center. It will likely utilize many of the assessment types above. |
| Strategic planning | These include opportunities to assess the current state of affairs and determine strategic priorities for the near future. Examples include determine 5 priorities for the next 3-4 years. |
Gavin Henning, Dartmouth College, 11/27/09
The mission and purpose are the center of the assessment cycle, it is the core from which student learning outcomes and our work emanates. With each cycle, it is critical to revisit the mission and purpose and ensure alignment among outcomes and the larger mission and purpose.
Statement of Purpose
A statement of purpose is different than a vision or mission statement. Consider a traveling metaphor, a vision is the final destination, a mission describes the path you take to your final destination, and the statement of purpose articulates why you started the journey to begin with. In our work, the university vision defines our final destination and our divisional mission defines the path we take in our division to get there.
University of San Diego Vision:
Strengthened by the Catholic intellectual tradition, we confront humanity’s challenges by fostering peace, working for justice and leading with love.
Based on the unit statement of purpose and functional areas, the unit should articulate a few enduring, broadly stated goals that reflect the work of the unit and align with the co-curricular learning outcomes and/or student affairs strategic plan. While unit goals are intended to be broader and more long-term and outcomes are intended to be more specific than a goal, both should be articulated using an outcome orientation.
Rather than thinking about what we will do (facilitator focused), thinking about what we want to happen because of what we do (participant focused). In terms of a learning outcome or goal, it’s a shift from I am teaching to they are learning and articulating what that learning will be.
Tip: It may be helpful to begin with the specific learning or operational outcomes and then derive the key broader goals that capture all of the specific outcomes. Units should have no more than five unit goals.
Unit goals are defined as the broad outcomes that articulate the general student learning or other outcomes you intend through student engagement with your programs or services. Unit goals can be more learning or operational in focus.
Example focused on learning:
Students will develop an understanding and a disposition across the curriculum and co-curriculum, from making simple connections among ideas and experiences to synthesizing and transferring learning to new, complex situations within and beyond the campus.
Example focused on operational:
The program will achieve higher levels of effectiveness and efficiency regarding program implementation.
Each unit goal will have one or more associated learning and/or operational outcomes. The number of outcomes developed will depend on how broad the unit goal is and how complex the intended learning is. It's ok to hav ea lot of outcomes, you don't have to assess them all. Make sure to focus on what's most important.
An outcome is "the desired effect of a program, service, or intervention but is more specific than a goal. It is result focused and participant centered" (Henning, 2010).
A learning outcome focuses on "how students will be different because of a learning experience, the knowledge, skills, attitudes, and habits of mind that students take with them from a learning experience" (Suskie, 2009, p. 117).
Example: Students will be more engaged in their preceptorial course compared with other courses.
Aside from learning opportunities, there are many things we do to support the student experience like providing a safe place for them to live, providing healthy options for dining, delivering services effectively and efficiently, etc. The services we offer that attend to the base of Maslow’s hierarchy (physiological needs, safety, etc.) are considered operational outcomes rather than learning outcomes. Operational outcomes describe the intended effect that a service has instead of what a student learns by participating in a program, service or other intervention.
Example: Students will receive one of their top three preferences for an LLC theme.
Why write learning outcomes?
Writing learning outcomes helps us to be more intentional about the learning opportunities we offer students and creates the opportunity to measure the learning that occurs. This means that we can:
- improve our ability to tell students what we hope they will learn through their participation in a program, leadership opportunity or service we offer;
- be more precise about the strategies we offer to meet the articulated outcomes;
- consider how we will assess learning at the start of the process.
All of this leads to more effective learning opportunities for our students.
Some questions to keep in mind as you write your learning outcomes and consider related assessment and program improvement:
- What do we want our students to learn?
- How will we know if they learn it?
- What will we do if they don't learn it?
As a division, we will use the following formula for writing learning outcomes. Writing outcomes with at least 1-2 other people often produces the most effective learning outcomes by incorporating multiple perspectives and engaging multiple people in the process.
Formula for Writing Learning Outcomes
Condition + SWiBAT + Bloom Word + What
As a result of participating in an LLC, students will be able to make simple connections among ideas and experiences.
Condition - the condition describes the program, leadership opportunity, or service students will participate in or experience where the learning is intended to occur.
SWiBAT - an acronym for students will be able to... Using this format helps you write the learning outcome in a way that is focused on students and what they will be able to know, feel, or do after participating.
Bloom Word - using a “Bloom word” in your learning outcome helps you to calibrate the type of learning you expect to the levels of Bloom’s Taxonomy which demonstrates a progression in the levels of learning participants experience. For example, the learning outcome describing what students will learn through participating in a one-shot service project will use a Bloom word toward the bottom of the taxonomy rather than toward the top for a learning outcome for students participating in a recurring service project. Using a Bloom word also helps to ensure that your learning outcome will be measurable.
What - the “what” describes the actual thing you expect students to know, feel or do after participating. In the example provided above, the “what” is some content that students received during Torero Days about the University mission statement.
The following acronym describes the characteristics of a well-written learning outcome. Once you feel that you have a good learning outcome written, check it against these characteristics to see if you can improve it in any way.
SMART (Drucker, 1954)
Specific - clear and definite terms describing expected abilities, knowledge, values, attitudes, and performance
Measurable - it is feasible to get the data, data are accurate and reliable, issue can be assessed more than one way
Aggressive but Attainable - consider stretch targets to improve program
Results-oriented - describe what standards are expected for students
Time-bound - describe where you would like to be within a specified period of time
Bloom's Taxonomy and Other Taxonomies for Learning
Bloom’s taxonomy attempts to clarify the nebulous terms that can be associated with learning. The taxonomy is comprised of 3 learning domains: Cognitive, Affective, and Psychomotor. The cognitive domain focuses on the learning process associated with knowledge, the affective with attitudes, feelings, emotions, and the psychomotor with skills. Within each domain there are 5-6 categories, ranging from simple to more complex behaviors, each building on its predecessor. The concept of the taxonomy and associated learning domains helps educators to be more precise in the development of learning outcomes, associated strategies, and measures.
Please see Taxonomy for Learning for a table that links action verbs to the various levels within each docmain to help you write more effective learning outcomes.
Strategies are the specific efforts you'll employ to achieve the outcomes. Mostly, these will be the specific programs or services offered but strategies may also include things like the way your staffing and/or physical space is structured, the hours your office is open, policies and procedures, etc. One strategy may align with multiple outcomes and each outcome may require several strategies to achieve it. Think creatively and be specific.
The next step is to determine how you will measure your learning and operational outcomes. An acronym that can help remind us of the criteria for developing sound measures is:
MATURE Measures (UCF Operational Excellence and Assessment Support, 2005)Matches - measure directly matches to the outcome it is trying to measure
Appropriate Methods - uses appropriate direct and indirect methods
Targets - indicates desired level of performance
Useful - measures help identify what to improve
Reliable - based on tested, known methods
Effective & Efficient - characterize the outcome concisely
To determine how to best measure student learning or operational outcomes, consider how you will know the extent to which the outcome was met and what data you would need to determine this. Measures can be direct or indirect.
Direct Measures
Direct methods provide direct evidence of what the student has or has not learned. Whenever possible we should try to gather direct evidence when measuring learning outcomes. Observation, pre-and post-testing, and evaluation of projects/portfolios are examples of direct methods. However, typical indirect methods such as surveys, interviews, open-ended questionnaires, and focus groups can provide direct evidence depending on the question. A question that asks students to list two resources that can help them with academic concerns provides direct evidence of learning. However, a question that asks students to tell you if they know of resources that can help them with academic concerns provides indirect evidence of learning.
Indirect Measures
Indirect methods measure opinions or thoughts about a student's own knowledge, skills, attitudes, learning experiences, perceptions of services received, etc. They do not measure the learning directly but can provide rich information. Typical examples of indirect methods are: survey, interview, focus group, etc. However, depending on the phrasing of the question, you may be able to gather direct evidence from typically indirect methods.
It’s important to think creatively about different ways to assess learning. Often we can slightly tweak something we are already doing in the learning process to include data collection and even analysis. Asking the question, “how will we know the learning occurred?” can help open up the options for measure.
Tip: The first three elements of the assessment cycle, mission and purpose, outcomes, and strategies and measures, are interrelated. It is important to consider all three as you write your outcomes and remember that these three steps are iterative. As you write an outcome and consider the questions
- how will I know the extent to which the outcome was met? (your measure)
- how will I expose students to the outcome? (your strategy)
Once you've identified your strategies, and measures, you will be ready to gather data. The first thing to consider is whether or not the data you need is already collected. For example, the university collects demographic data and tracks a variety of data through Banner and other means.
Qualitative and Quantitative Methodology
Each methodology is defined differently and is based on distinctive underlying assumptions.
Qualitative methodology is the detailed description of the situations, events, people, interactions, and observed behaviors, the use of direct quotations from people about their experiences, attitudes, beliefs, and thoughts; and the analysis of excerpts or entire passages from documents, correspondence, records, and case histories (Schuh and Upcraft, 2001, p.27).
Whereas quantitative methodologies "use structured, predetermined response options that can be summarized into meaningful numbers and analyzed statistically. Test scores, rubric scores, survey ratings, and performance indicators are all examples of quantitative evidence” (Suskie, 2009, p. 32).
The table below summarizes some of the differences inherent in each perspective.
| Assumption | Question | Qualitative Characteristics | Quantitative Characteristics |
|---|---|---|---|
| Epistemological | What is the relationship between the researcher and participants? | Attempt to reduce distance between researcher and participant(s) | Researcher is independent from what is being researched |
| Axiological | What is the role of values? | Research is value-laden and biases are present | Value free and unbiased, values are emotive and therefore outside the scientific inquiry |
| Rhetorical | What is the language of research? | Informal, personal voice, qualitative terms, limited definitions | Formal, based on a set of definitions, impersonal voice |
| Methodological | What is the process of research? | Inductive, study topic within context, use emerging design | To explain and predict |
Based on (Creswell, 2007, p. 17)
It is important to consider your outcomes, strategies and measures to decide if you need to collect qualitative or quantitative data and which will provide the type of evidence you need to determine the extent to which you have met your learning outcome.
Institutional Review Board (IRB) Approval
Depending on the type of data you need to collect, your data collection efforts may require IRB approval.
After you've collected evidence, it is still just data.
To understand the extent to which learning occurred or the operational outcome was met, you need to interpret the evidence and transform it to information that can be actionable. You may have multiple measures and data points that you’ll need to consider together and interpret their meaning.
When analyzing qualitative data we begin with large quantities of data and the goal is to move toward a concise description. Coding is a common form of qualitative analysis and it’s helpful to think of the process like a funnel. Some key steps in the process include (1) reframing and revisiting the original purpose of the assessment, (2) reading or listening to your data without coding, this step helps you to see the whole of what the data includes, (3) reading or listening with the intent of identifying preliminary themes in the data, (4) a third read or listen helps to verify the preliminary themes, (5) and finally, once you are satisfied that you have identified the major themes in the data, you can summarize (Bresciani, Zelna, & Anderson, 2004, p. 66-68).
Reframe--revisit the purpose of the assessment
Read/listen without coding
Preliminary themes
Verify themes
Summarize
When analyzing quantitative data we differentiate between descriptive and inferential statistics. Descriptive statistics describe the characteristics of a given set of data and inferential statistics use a sample to make inferences about the full population (Urdan, 2005, p. 2). Common descriptive statistics that measure central tendency include mean, median, and mode. Descriptive measures of variability include range, variance, and standard deviation and describe the dispersion of the scores within a data set.
Inferential statistics is concerned with making inferences about the population based on the data collected from a smaller sample (Urdan, 2005, p. 57-58). Inferential statistics relies on statistical significance to determine whether or not statistical differences exist between a sample and population or between different subsets of data in sample. For example, we would use a test of difference to help us understand whether or not statistically significant differences exist between the mean SAT score for first generation students and the mean SAT score for continuing generation students entering a certain university. If we learned that there was a statistically significant difference between the two means, we would still need to determine if the difference was practically significant, the effect size would describe how large the difference is without respect to sample size (Urdan, 2005, p. 57-73).
Based on your interpretation of the information, identify any changes needed to improve student learning or operational outcomes. Because you've articulated the strategies you employ to expose students to the outcomes and aligned these with specific outcomes, it should be easier to make changes to specific strategies to continue to improve.
Once you've identified areas to improve and the actions you will take to do so, you've closed the loop on the assessment cycle. This brings you back to the beginning of the cycle, mission and purpose, where you should begin again. Consider whether the outcome should continue to be measured or if it should be inactivated. Consider whether the outcome needs to be revised in light of mission and purpose. Consider whether a new outcome needs to be developed.
Assessment terms can be defined in multiple ways. The following glossary will ensure that we are using consistent language as we define our learning outcomes, strategies and measures as a division.
Assessment - “any effort to gather, analyze, and interpret evidence which describes institutional, departmental, divisional, or agency effectiveness.” (Upcraft & Schuh, 1996)
Assessment Plan – a document that outlines a unit’s plan for assessing various learning outcomes and operational outcomes over a certain period of time.
Benchmark (standard, criterion) – a standard against which to compare assessment results to interpret the meaning of the result. (Suskie, 2009, p. 233-234)
Descriptive Statistics – statistics used to describe the characteristics of a distribution of scores.
- Frequency – the number of times a given score occurs in a set of data.
- Mean – the arithmetic average of a distribution of scores.
- Median – the score in the distribution that marks the 50th percentile. It is the score at which 50% of the distribution falls below and 50% falls above.
- Mode – the score in the distribution that occurs most frequently.
- Range – the difference between the largest score and the smallest score of a distribution.
- Standard Deviation – the average deviation between the individual scores in the distribution and the mean for the distribution.
(Urdan, 2005, p. 2-6)
Direct Measures – measures student learning outcomes directly. “Direct evidence of student learning is tangible, visible, self-explanatory, and compelling evidence of exactly what students have and have not learned” (Suskie, 2009, p.20)
Functional Area - the distinct functions that the unit engages in to provide services and learning opportunities for students.
Goal - the desired effect of a program, service, or intervention but is more broad than an outcome or objective. It can be participant focused or facilitator focused. (Henning, 2010)
Indirect Measures – measures opinions or thoughts about students’ own knowledge, skills, attitudes, learning experiences, perceptions of services received, etc. Indirect methods “consist of proxy signs that students are probably learning. Indirect evidence is less clear and less convincing than direct evidence” (Suskie, 2009, p. 20).
Inferential Statistics – Statistics, derived from sample data, that are used to make inferences about the population from which the sample was drawn.
- Population – the collection of cases that comprise the entire set of cases with the specified characteristics (e.g., all living adult males in the United States).
- Sample – a collection of cases selected from a larger population.
- Statistical significance – the likelihood, or probability, that a statistic derived from a sample represents some genuine phenomenon in the population from which the sample was selected.
(Urdan, 2005, p. 57-73)
Inputs – the various resources available to you to implement your strategy such as budget, time, materials, etc.
Objective – “the intended effect of a program, service or intervention, but is more specific than a goal. It is facilitator focused” (Henning, 2010).
Outcome – “the desired effect of a program, service, or intervention but is more specific than a goal. It is results focused and participant centered” (Henning, 2010).
- Learning Outcome – focuses on “how students will be different because of a learning experience, the knowledge, skills attitudes and habits of mind that students take with them from a learning experience” (Suskie, 2009, p.117).
- Operational Outcome – focuses on the desired effect of the services Student Affairs units provide that target the base level of Maslow’s hierarchy of needs such as physiological needs and safety. Operational outcomes can also focus on the efficiency and effectiveness of the services we deliver. Some examples are providing healthy food options, safe and comfortable living options on campus, and timely housing assignments, etc.
Outcomes Assessment – measures the effect on student learning and development or other intended outcomes such as operational.
Output – a tangible product that marks progression toward an outcome such as the number of programs offered, attendance, etc.
Qualitative Assessment – detailed description of the situations, events, people, interactions, and observed behaviors, the use of direct quotations from people about their experiences, attitudes, beliefs, and thoughts; and the analysis of excerpts or entire passages from documents, correspondence, records, and case histories (Schuh and Upcraft, 2001, p.27).
Quantitative Assessment – “use structured, predetermined response options that can be summarized into meaningful numbers and analyzed statistically. Test scores, rubric scores, survey ratings, and performance indicators are all examples of quantitative evidence” (Suskie, 2009, p. 32).
Satisfaction – “seeks to determine if student or client experiences with programs, services, or learning opportunities are of high quality and consistent with the institution’s mission.” (Shuh and Upcraft, 2001, p.142)
Stakeholders – any party who has a vested interest in the issue, program or results of your assessment.
Strategy – “a means to achieve an outcome or goal” (Henning, 2010).
Bresciani, M. J., Zelna, C. L., & Anderson, J. A. (2004). Assessing student learning and development. A handbook for practitioners. United States: NASPA.
Creswell, J. W., (2007). Qualitative inquiry and research design: Choosing among five approaches. Thousand Oaks, CA: Sage.
Drucker, P. F. (1954). The practice of management: A study of the most important function in American society. Harper & Brothers.
Henning, G. (2009). Assessment continuum. Retrieved from http://www.dartmouth.edu/~oir/pdfs/assessment_continuum.pdf
Henning, G. (2010). Outcomes development. Retrieved from http://www.dartmouth.edu/~oir/pdfs/outcomedev.pdf
Schuh, J. H., & Upcraft, M. L. (2001). Assessment practice in student affairs: an applications manual. San Francisco, CA: Jossey-Bass.
Suskie, L. (2009). Assessing student learning: a common sense guide (2nd Edition ed.). San Francisco, CA: Jossey-Bass.
UCF Operational Excellence & Administrative Support. (2005). UCF Administrative Assessment Handbook.
Urdan, T. C. (2005). Statistics in plain English. Mahway, New Jersey, Lawrence Erlbaum Associates.