Evaluation — an Illusive Goal
Outcomes – Impact – Output – Measures – PROOF???
Funders ask for measurable results. Some nonprofits even want to evaluate impact for themselves, seeking to improve what they do. Regardless of motivation, many nonprofits, even those that have taken the steps to evaluate their work, struggle with evaluation. Even nonprofits with capacity can have this challenge.
For example, the United Way of Anchorage has been dedicated to this goal since Michele Brown took over as CEO. They have truly moved the dial and its service partners’ ability to articulate and measure results. My opinion is they are farther along than most United Ways. They have established buy-in for their process not only from many of our local funders but also their nonprofit partners. But truth be told, they probably wish that with all the effort they have expended to date, they were farther along.
The Foraker Group also became very intentional about “walking the talk” a few years ago. While we understand the theory and even had additional resources from The Murdock Charitable Trust to implement our process, we have found progress very slow. But we know it is the right thing to do for us, our Partners, and for our donors, so we persist.
My hope is that this article better defines the dilemma we face when measuring impact. Few would argue that the ability to prove that our actions have made a difference is important. So what are the obstacles to success? I propose a few factors that make evaluation hard.
The first significant problem is that nonprofits try to evaluate outcomes that are too hard to measure – many choose outcomes that require development of new data, or they choose outcomes where they must wait for years to see the result. Most nonprofits are too busy with their day-to-day activities to add data collection to their already full plate, or to wait years to prove their impact. These obstacles stop many nonprofits from taking evaluation seriously.
But maybe the most limiting factor is that nonprofits are under-resourced to measure impact. While funders ask nonprofits to measure impact, most don’t provide the extra funds required to get the measures they seek. Other funders impose outcomes they want rather than fully understanding how nonprofits see their most important outcomes, therefore creating a disconnect with the real reason for evaluation. That alone is hard to overcome – are we doing this for the funder, or for ourselves? And in defense of funders, nonprofits should not wait for a funder to measure outcomes. They should budget evaluation like any other expense, and not wait on special funding to engage in the process.
These are not new dilemmas. In order to set the stage for what I hope is an ongoing conversation with our Partners (via our blog), I offer my version of history on the subject and try to shine some light on the difficulty we face when it comes to evaluation.
Thirty years ago, the United Way of America hosted sessions at its annual leadership conference on program evaluation and outcomes. I know, I was there. Bill Aramony the national CEO, (later made a villain for his ethical lapses), was leading the nonprofit sector to a new level of professionalism. For example, before Aramony, nonprofit employees had little respect as professionals. They were not paid a living wage nor did many even have benefits. Despite his later indiscretions, he had a positive impact on the sector (that is before he almost destroyed it). However, when he was still a respected leader, he recruited many outstanding thinkers for United Way of America like Russy Sumariwalla who defined social services through a system called UWASIS (United Way of America Services Identification System). It was also when United Way began a conversation on how it could do a better job in measuring the impact of its funding.
Other leaders both in the national office and local communities worked to put that theory to practice. One of those big thinkers involved from the beginning at the national organization was Dirk van den Bosch who still serves as a Foraker consultant. And on the community level Gary Ostroske, the CEO of United of New Orleans and our keynote speaker at the 2008 Leadership Summit, was one of the early leaders on impact. Russy, Dirk, Gary, and others became fluent in the theory of impact and worked to educate those of us eager to learn. Thirty years later, unfortunately little progress has been made and the system has at best inched toward identifying what it hopes to accomplish, thankfully with United Way of Anchorage leading the way!
Other parts of the social sector also worked to measure impact. Education leaders began their quest to measure impact years before the United Way system. They had to report to elected officials that wanted “facts” to justify allocating tax dollars. They were the ones that first defined many of the terms we use today like impact, outcomes, etc. Around the same time, leaders in the health care industry, also eager to measure results, started looking for the facts.
While we can claim to be farther along than we were 30 years ago, most, if honest, would say they are not satisfied with their progress on measuring outcomes. Measuring the impact of the social sector is not easy. If it were, we would be doing it.
Even with more capacity, large institutions like universities, hospitals, and school systems, struggle with measuring their impact. It’s little wonder that most nonprofits, with far less capacity, still only monitor their outputs – like how many people they serve. It’s much harder to measure what difference serving all those people has made.
In past articles we have provided definitions for logic models/impact terminology we hope were useful. Here they are re-stated:
- Outcome – a change in attitude, behavior, knowledge
- Output – what we do and how often, like clients served, number of events, etc.
- Resources – what is needed to produce outcomes (people, money, etc.)
- Impact – what changed
- Indicators – the measure used to evaluate performance
The first problem in doing effective evaluation is clarity on what you what to measure – doing that is much harder than most think. At Foraker, when we started our process, we quickly determined that we needed to stop long enough (about four years!!!) while we developed our sustainability model. Then we had to create a theory of change on how that model moved us toward accomplishing our mission. That took another year or so. Then we engaged the Foraker boards in identifying the indicators they felt would best provide the information they needed to monitor and then evaluate performance.
We got there!!! We had all the parts in place and even started to publish our process and progress on a dashboard, the tool we use to monitor performance. All of this took tremendous effort with support from our colleagues in the region in the Murdock capacity building cohort and the national consultants TCC Group.
We made progress on measuring our impact for about 18 months. We received positive feedback from many in and out of state. We became a model on how to do outcomes right!!! Then at its last meeting, the Governance Board felt that we could do even better – so now we are in the process of improving what we monitor. And that is good news.
We have confirmed that measuring impact is not a short-term venture with a clear destination. Like many things, it is a journey and while we may always get closer to the goal, we may never arrive. Knowing this may help others that are stalled waiting to start this work. Most hesitate because they seek perfection. Perfection, or maybe believing in it, is perhaps the biggest obstacle with evaluation.
The first question to be answered in evaluation is “what outcome do you want to measure?”
For example, is a soup kitchen there to end hunger? Or is it there to make sure no one starves?
Does a shelter for battered women exist to end domestic violence. Or to keep women in those situations safe?
And while there may be a long-term benefit to the arts, is the outcome of the arts sometime in the future, or is it in the moment – the inspiration we feel after seeing great art or listening to an amazing performance?
Sometimes when it looks like a fish, swims like a fish, and tastes like a fish – it’s a fish. Does the fish really need to grow legs and walk on land? While this may seem a provocative comparison, most nonprofits were created to provide immediate impact. Sometimes nonprofits try to change why they were founded or are encouraged to do so to get funding. This is a bad idea regardless.
If your nonprofit was founded to meet short-term outcomes, my advice is to stop sweating evaluation. It is easy to see an audience’s response. We can easily monitor many short-term impacts for social and health services. If your organization is working for short-term impact, you have no excuse in not learning about, then using the terminology, and evaluating your outcomes. You will impress your donors. More important, over time you can evaluate if your services are still needed and if they have impact.
But other outcomes are more elusive. Most substance abuse programs want to help people become healthy and live productive lives. Is successful completion of the program enough evidence of impact? Maybe not – so in that case, looking down the road may be necessary. Successful completion of the program is just the start. So how does that nonprofit evaluate? And what resources does it need to better evaluate the intervals of success that can be measured in order to justify ongoing funding?
Again United Way of Anchorage’s journey may be a good example. One of their big goals is “90 by 2020,” short for the outcome of a 90% high school graduation rate by 2020. They have aligned partners both in their funded nonprofits and other community organizations. All have adopted the vision and all have specific outputs they will produce over time to accomplish the goal. Since 2020 is seven years away, they will monitor the graduation rate year-to-year to see if they are making progress. They have identified ways to determine what strategies work and what others may need adjustment. We will have to wait to see if they achieve the goal. But the point is, they have begun what we all should do – prove to ourselves that what we do matters. That reality will also help us secure funding. Some of us can measure impact easily with a standing ovation, others will need to be more patient and strategic.
The outcomes mentioned earlier are social outcomes – the hardest to measure. In our sector there are two bottom lines.
So there is another outcome that can be easier to monitor. For-profit execs serving on our boards are quick to let us know how well they measure their performance. And many of them do. But the fact is they have one bottom line that really matters, and that is to achieve a “return on investment,” or ROI – a profit. While having a positive outcome – making a profit – can be elusive, the measure is easy.
We have worked to express that the term “nonprofit” is not a strategy for sustainability. We need to make a profit in order to thrive and we should measure our profit just like a business. Did we make more than we spent? Have we built our reserves? Are we prepared to expand services?
Nonprofits can make profit in two ways. First is to raise money from individuals. In our sector, most individual donations are like “profit” by providing unrestricted cash. We can spend those donations the way we want. And if we practice development the right way, those donations grow over time and provide a competitive advantage. Of course, charitable donations are only available for 501(c)(3)’s. But there is one more way to generate profit. All nonprofits including charities can earn revenue, and as long as it’s mission-related we pay no tax on the “profit.” We can reinvest that profit into our mission, building our capacity to do more, better.
Making a profit for a nonprofit is as hard as it is in the business world. You must have a good product or service. You must know your customers and meet their needs. You must determine value and prove you are worth it.
And guess what? When you do a good job and when you are relevant in the marketplace, you can make a profit. Nationally earned revenue comprises about 50% of all charitable nonprofit income – non-charitable nonprofits generate over 90% of their total from earned income.
In summary, there are few points to make on evaluation:
- Measuring impact is a worthwhile activity for all high performing nonprofits. We owe it to our funders. More important we owe it to the people we serve.
- In order to measure impact we have to know who we are and make sure what we measure reflects that understanding.
- We need to budget for ongoing evaluation.
- Funders interested in their grantee’s outcomes need to fund ongoing evaluation over the time period they want analyzed.
- Nonprofits have two bottom lines – ROI and “return on social investment,” or ROSI, the impact we have as a result of our mission. ROSI is harder to measure, especially when outcomes are not measurable for years. And ROI, while easier to measure has never been a priority for nonprofits, especially those that feel they must spend every penny they get. Nonprofits need a profit in order to survive.
We know that the debate on outcomes is ongoing. We appreciate your comments and insight. Please put your thoughts on our blog to encourage ongoing insight into this huge issue.