Mirror, Mirror on the Wall
This is the eighth in a series of “food for thought” pieces from CIEE. The themes vary but all deal with study abroad for U.S. undergraduates. We present our ideas not as the only viable ones but rather to stimulate discourse in furtherance of the study abroad enterprise. Previous topics include:
– How Are We Doing?
– Standards
– A CIEE Eye for the Study Abroad Guy...or Girl
– Parents, Pills, & Pandering
– A Research Agenda for Study Abroad
– What’s It All About?
– Numbers
We welcome your comments and requests for additional copies at any time to editor@ciee.org.
A little more than a decade ago we asked a simple question:
“Mirror, mirror on the wall, is this the greatest program of all?”
Well, not exactly that question but something like it. More specifically, we asked ourselves “How do we know this program is any good?” Of course, there are the usual answers, “We just know” or “We get feedback” or “We have a Ph.D. so how could we not know if a program is any good?” While there is some truth in this sort of perspective, the fact is there is no such thing as “I know” when it comes to quality assurance unless you have independent, empirical evidence to support the statement. Everything else is conjecture, perhaps educated and informed conjecture, but nonetheless conjecture very subject to subversion of the truth. It is easy in the absence of hard evidence to convince ourselves we’re doing a great job. We’ve all done it. It’s human nature.
Of course, none of us really know how we look in the mirror and none of us really know if our programs, projects, and work are really, really good. We have a sense, an idea of what’s working and what’s not. But we can easily hide from the truth. It’s a lot easier to look skinny in winter when you hide underneath layers of clothing than in the summer when you’re wearing a polo shirt or bathing suit. The best measures of performance are those that are most objective when applied. There is no replacement for high-quality, objectively-executed, and transparently-reported program evaluation. It tells you how you are doing. It provides a roadmap for change and improvement. It keeps staff on their toes in terms of fulfilling program goals. And it helps assure full value for program participants and stakeholders.
Over a decade of hard work has led us to make many improvements in the process we use in evaluating our programs abroad. Other organizations have adopted similar processes and/or are considering such adoption. This is good. The better the quality of the overall provider world, including colleges and universities where program evaluation is often politically tricky, the better off over time we’ll all be. The purpose of this piece is not to focus on what CIEE does. Rather, it is to identify what we see as the key characteristics of an evaluation process and share with you some of what we’ve learned over time. You might see it differently. It’s why they make chocolate and vanilla — so that you can make a choice. We hope these ideas help you make some of those choices. What we can say with certainty is that having all stakeholders involved in designing and implementing the process, from the very beginning, has made an enormous difference for us. Collaboration and cooperation has led to a better process, more meaningful outputs, and stronger ongoing quality.
Goals
“When you know where you are going, any road will get you there.” In any evaluation process, you should have explicit goals as to what you are trying to accomplish. For each program, we created specific goals and outcomes for participants. Therefore, the goal of evaluation is to assess when a program is meeting its goals. It’s not about marketing, or happiness, or success; it’s about performance. Set goals, evaluate performance. It’s the essence of all evaluation, to determine how you are doing. Goals, by the way, come in many types. To our way of thinking, there are two types of goal-setting for programs. The first type is cognitive goals, i.e., what students learn; language, math, etc. The second type of goal is affective; i.e., how people should feel at the end of the experience. These are often harder goals to set and measure. But, they are important nonetheless. A good deal of study abroad learning is affective. It’s about understanding other people and cultures, not just how to conjugate verbs.
Terms of Evaluation
When you evaluate anything, you have to decide what to look at in the process. So, for example, we decided long ago not to evaluate overall host institution quality. We are not an accrediting institution for overseas partner universities. Rather, our focus is on the particular services that a host institution provides. Good institutions can have lousy classes and facilities and weak institutions can similarly have moments of great brilliance. We’re less concerned with the overall quality and reputation of partners than we are with what a partner provides. Housing, meals, library, teaching, class options, facilities, and support services are often part and parcel of the topics to be addressed in any evaluation. However, it varies widely by site and program goals. Therefore, our experience says no one recipe works for all purposes. Rather, evaluation topics for each program need to be developed to assure that the right issues get addressed and a quality evaluation takes place.
Inputs
Before we buy a new car, we often get on the web and do some research. From Hemmings to Consumer Reports, from Car and Driver to Mechanics Illustrated, there is a world of information upon which to evaluate any car for driving comfort, style, performance, and value. If we’re going to evaluate a program, the same holds true. We need to find the reference material to do so. We’ve found that a combination of materials should be available to those performing the evaluation; internal reports, management overviews, student evaluation and feedback both on program and courses, and institutional feedback from users and those that deliver the program. This collection of information, or “briefing book,” should help anyone charged with an evaluation have a clear picture of a program and its operational successes and failures, at least to the extent this can ever be done on paper. And, of course, access to prior evaluations for continuity is equally critical. Good briefing books are big and thick and they require effort to develop and master, but they pay off by providing a reference point to make the evaluation process of the highest quality.
Teams
We long ago opted for three-person teams made up of those familiar with study abroad and our processes paired with qualified faculty, those with subject matter expertise and destination knowledge. Finding the right team and helping them get up to speed via briefing calls and support materials are critical. Each team needs a leader, usually the study abroad professional who is involved in evaluations and has experience in our process. Having opted for three-person teams, it took us a long time to realize that sometimes two people can do the job just as well, and sometimes it pays to take a quick look or perform a mini-evaluation on a program with just a single person. Teams over time have come to mean a troika, but in many cases, for programmatic and/or timing considerations, a threesome is the wrong approach. It’s been another important lesson, that just because the pattern works most places doesn’t mean it works everyplace. If we’ve learned anything, it’s to be willing to change the approach when the situation warrants, but only after careful consideration.
Transparency
Every one of our evaluations generates a report and that report, just like student feedback, is on our website at www.ciee.org for all to see. Transparency is the secret to evaluation. It’s easy to kid yourself about how you’re doing. Why does that mirror always makes me look a bit fuller than I am? The reality is that knowing something is going to be published raises the standards of everyone in the process. Do mistakes and problems get aired? Of course they do, and they should. Yes, there are personnel and other sensitive issues that have to be guarded carefully, but by and large we’ve found very few things that really deserve secret status. Transparency is the guardian of truth.
Follow-Through
Strategic plans are often described by management as fat books that take tons of time to write, are not too interesting to read, and often sit on shelves ignored until it’s time to update them. Evaluations should not be static documents. They should be read and digested and somebody should take action on recommendations. This doesn’t mean that every idea that comes out in an evaluation must be acted upon, but every idea should be reviewed, responded to, and closure achieved. In this way, each idea is further evaluated before action takes place. But action does take place. When evaluations are repeated every three to five years, the place to start is the last one. Evaluation should be seen as a process that encourages introspection, change, and improvement in programming. To some extent, changing attitudes towards feedback to get people to embrace the value is a greater challenge than doing the evaluation itself.
Eternal Loops
An eternal loop is one that never ends and evaluation is clearly an eternal loop. We do many things on an ongoing basis to assure program quality including external monitoring, data collection from various users, and site visits. These ongoing processes are linked and related to the formal evaluation process in a variety of ways. The key is that evaluation is not a one-time cyclical experience but rather a process. The level of detail, the amount of resources committed, and the depth of evaluation varies from program to program and year to year. It should. Yet, overall, the process should add up to a collective and collaborative way to assess, monitor, and improve program quality and it should be happening all the time, not just every few years like accreditation.
There is a lot more to evaluation than these steps, but these are the critical ones to think about. We’ve learned that all of this takes time, money, patience, and dedicated staff and volunteers at every level to make the process work. It requires careful preparation, solid planning and execution, and a good deal of flexibility in dealing with the many twists and turns one is bound to experience. We’ve also learned that getting faculty, staff, and everyone else involved to embrace the process — to see it as a value-added proposition rather than an intrusive requirement — takes a good deal of leadership on the part of many. It doesn’t happen overnight. If a Resident Director learns that program evaluation causes problems, rather than creates opportunities, the process will be subverted.
We encourage everyone to look in the mirror. What’s the saying, “you can’t be too rich or too thin?” Well, you probably can be, and you can probably also be too introspective, but it’s pretty hard to err on the side of too much evaluation. All experience indicates that the marginal propensity to avoid evaluation is greater than the desire to pursue feedback. Study abroad stands to gain a great deal by answering the “Mirror, mirror on the wall…” question. Even when we don’t like the answer, at least we know where we stand and can do something about it. In the end, this is the core of what all evaluation is about…figuring out how we’re really doing and doing something to make sure we do it better as we move forward. While systems and procedures for doing this are certainly important, the most important thing is getting started, doing it regularly, and embracing the value of the process.