Monitoring, evaluation and impact

 

Useful resources
Measure impact on young people’s well-being
– NPC’s Well-being Measure
New Philanthropy Capital has launched a product to help charities, schools and youth organisations measure their impact on young people’s well-being. It’s an evaluation tool designed for surveying across a group or cohort of young people aged 11 to 16. Each subscribers has their own online account where they can create surveys and track how well-being changes in eight areas, including self-esteem, emotional well-being, resilience, and friendships. It’s flexible so you can choose from this set of validated measures and add your own questions if you wish. Once complete, you can download a report of your results. NPC’s Well-being Measure is supported by the Esmée Fairbairn Foundation, the Paul Hamlyn Foundation, and the Private Equity Foundation. For more information visit www.well-beingmeasure.com

Project Oracle: new resources: the Mayor of London’s programme for “understanding and sharing what really works” in preventing youth crime and in improving the lives of children and young people in London. The website invites funders, providers and advisers to register on the site. Effort has been made to keep the functionality simple with the potential to develop further.

The development of Project Oracle
Nesta and Mayor of London, 2012
This paper examines why "many services are unable or unwilling to measure the improvements they make in outcomes for young people” and explains how Project Oracle is remedying this in London by linking youth programmes with academically rigorous and internationally recognised standards of evidence to improve consistency and quality in understanding what does and does not work. Small and large projects are signing up to make use of Project Oracle’s resources. This paper is a summary of the need and the context for Project Oracle, and its experience so far.

Center for Evaluation Innovation: The Center's publications database can be searched for resources on evaluating advocacy. 
 
External Publications
Malin Arvidson, Fergus Lyon, Stephen McKay and Domenico Moro, Third Sector Research Centre, 2010
The benefits and the challenges of using SROI, including the high costs of using the SROI tool and its limitations. The paper looks at the origins and use of SROI before identifying some emerging challenges. The authors draw out implications for both those using impact tools to demonstrate the value of their work, and those interpreting the results of SROI exercises which do not easily lend themselves to comparison across organisations and projects.
 
Charities Evaluation Services, 2010
A handbook of practical ways to identify and collect information on the outcome of work and the impact it has made. It provides help in designing a monitoring tool or choosing one already available.
 

Breakthroughs in shared measurement and social impact
FSG Social Impact Advisors, 2009
Reports on nonprofit organisations measuring their performance on common indicators and shared evaluation platforms using web-based systems. Breakthroughs include shared measurement platforms (an agreed-upon set of benchmarks developed by funding organisations and their grantees), comparative performance systems (ways to compare results between different grantee organisations) and adaptive learning systems (using the leverage of both of those systems to develop strategies and coordinate resources between multiple foundations and grantees). 

Can ICT help your monitoring and evaluation?
Charities Evaluation Services
Guidance for voluntary organisations on the selection of a suitable IT system or database to support their monitoring, as a response to evidence that this was a key barrier to such work – what is required from a system and which off-the-shelf examples are cost-effective. 

Community investment by social housing organisations: measuring the impact
Survey Report for HACT, 2012; researchers: Vanessa Wilkes and Professor David Mullins, Third Sector Research Centre
This picture of the measurement tools being used by over 30 social landlords shows that while there is general recognition of the importance of measuring impact, there are also concerns about cost, approach and potential duplication and wide variation in the approaches used. This proves to be a complex area with no easy choices. Organisations that have developed in-house tools are generally less satisfied than those using external tools but only one external tool has been specifically designed for the sector. There is a strong interest in measuring joint outcomes, for example where housing associations join with other agencies to invest in neighbourhood based initiatives, but very little existing practice. Full research reportSummary report by HACT.

Does Your Money Make a Difference?
Charities Evaluation Services, 2010
Fully revised second edition of CES’ good practice guide on monitoring and evaluation for funders. It covers the basics well and tackles proportionate monitoring, intelligent funding and “funder plus” activities. It encourages funders to improve their monitoring and evaluation, but flexibly in relation to their circumstances and their key purposes. The text is illustrated with examples, links to further reading and a good practice self-assessment model. London Funders’ Research and Evaluation project group participated in the updating and its Chair, Andrew Cooper of The Diana, Princess of Wales Memorial Fund contributes the foreword in which he says that the guide, “is a welcome and helpful encouragement to us, as funders, to challenge assumptions, improve our monitoring and evaluation and embrace a flexible culture of learning to make our funding as efficient and effective as possible.”

Evaluation in philanthropy: perspectives from the field
Grantmakers for Effective Organizations, 2009
A stimulating six-page executive summary from this US donor association on how funders, and their funded organisations, can use evaluation as a learning tool, not simply for accountability. Access to the executive summary is free but GEO membership is required for the full report. All examples illustrated are US ones but the material is accessibly written and includes pithy examples around specific themes such as:

  • How evaluation can achieve continuing improvement and not just evidence of outcomes: 

  • How it can help explain change, beyond describing success and failure

  • How evaluation can involve a range of actors and share learning widely

  • What the funder can learn, not just from one grant but a cluster or a whole programme

  • How failure can be worked hard to provide lessons

Evidence for social policy and practice: perspectives on how research and evidence can influence decision making in public services
NESTA, 2011
Six essays from UK and US organisations using different methodology and approaches to generate evidence and influence policy and practice in a number of service areas. It challenges the limited rigour with which much UK social policy is evaluated and argues that at a time in the UK of public service reform and a push for decentralised decision-making, the need for accessible and reliable evidence is more important than ever. Includes a clear summary of the GLA’s Project Oracle.

Grassroots arts 
Third Sector Research Centre, Universities of Exeter and Glamorgan, Voluntary Arts, 2012
A report finds that arts activities help improve the well-being of both individuals and communities, increasing feelings of self-confidence and self-worth. The report notes the significant impact activity can have for minority groups such as disabled people, reducing isolation, increasing social networks and enhancing quality of life. The research partners intend to develop a toolkit for assessing the outcomes of amateur community arts activities. 

Harmonising reporting
Scotland Funders’ Forum, 2009
This practical working group report builds on the work of Turning the Tables which was a project commissioned by the Scotland Funders’ Forum and funded by BIG Scotland. The Forum followed it up with funders prepared to test out the issues more, and this report shows how to make reports more useful for funders and make reporting more useful and less burdensome for funded organisations.

Impact measurement 
A report on a discussion between funders, led by Big Lottery Fund and New Philanthropy Capital: looking at the different stages of grantmaking and considering what role funders can play in requiring, supporting, signposting or funding impact measurement – particularly for smaller organisations struggling to know what to do.

Increasing impact, enhancing value: a practitioner's guide to leading corporate philanthropy
Council on Foundation, 2012
The product of a year-long field research project, this guide offers a five-point framework for redefining corporate philanthropy. The points are: create a new narrative around corporate philanthropy as an investment in society; develop an inclusive "operating system" for philanthropic investment; improve collaboration, communication, and knowledge sharing; mobilise field leadership behind the agenda; and professionalise the field.

Monitoring and evaluation of family intervention projects and services
Department for Education, 2011
An evaluation report on the experience and effectiveness of family intervention services from 2007-11. Valuable background to the coalition government’s new programme to create a national network of Troubled Family Trouble-Shooters for radical transformation in the lives of “the country's most troubled families” (see CLG press release here).

Monitoring and evaluation on a shoestring
Charities Evaluation Services, 2010
This National Performance Programme practical guide recognises the limited budgets available to many organisations for their monitoring and evaluation.

Outcome and outcome indicator banks: availability and use
Avan Wadia and Diana Parkinson, Charities Evaluation Services, 2010
This research investigates outcome indicator banks - the bringing together of a range of methods already used to demonstrate outcomes. It identifies existing banks, how they are being used, their strengths and weaknesses as a resource; and gives recommendations for further development. 

Pathfinder: a practical guide to advocacy evaluation
Innovation Network Inc.
An introduction to advocacy evaluation from an advocate or funder's perspective. It recommends learning-focused advocacy evaluation, which yields the type of information funders and advocates need to understand their progress. Click here for the Funder version or the Advocate version.

Principles into practice: how charities and social enterprises communicate impact
Charity Finance Group, ACEVO and New Philanthropy Capital, 2012
A report addressing the challenges of impact measurement, setting out six general principles that define how charities should communicate their impact: clarity, accessibility, transparency, accountability, verifiability and proportionality. Each principle is based on making clear links between a charity’s activities and its aim and mission. The principles come to life in nine compact case studies. London Funders attended the launch of the report to hear from charities involved in the research and the challenges impact measurements poses for funders. Read our comments here.

Reframing the conversation: expanding the impact of grantees
Grantmakers for Effective Organizations, 2011
This report, sub-titled How do we build the capacity of nonprofits to evaluate, learn and improve?, outlines ways to boost VCS capacity to systematically collect, analyse, and learn from data, including helping link programme goals to evaluation questions and underwriting the costs of new technology and evaluation skills training.

SROI for funders
Lucy Heady, New Philanthropy Capital, 2010
Social Return on Investment is a type of economic analysis that focuses on listening to stakeholders, identifying outcomes and giving these a financial value. This report is written specifically to help funders understand its role and how funded organisations might be helped to use it as a reporting tool. 

Tools to support public policy grantmaking
Martha Campbell and Julia Coffman, The Foundation Review, 2009
This journal article, especially recommended for funders, offers guidance on how foundations can frame, focus and advance efforts to achieve public policy reform. It covers five essential steps for developing public policy strategy, and offers two tools to support foundations during the strategy development process.

Universal set of principles for impact reporting
New Philanthropy Capital, with six voluntary sector partners, 2011
Principles out for consultation in late 2011 and early 2012 and intended to inspire better and widespread impact reporting. 

User's guide to advocacy evaluation planning
Julia Coffman, Harvard Family Research Project, 2009
Four basic steps that generate the core elements of an advocacy evaluation incl including what will be measured and how. 

Value of charity analysis: how reviewing your organisation can help you achieve more
Esther Paterson and Iona Joy, New Philanthropy Capital, 2011
How organisational reviews can help charities be more effective and achieve more with their resources. Three main benefits of analysis emerge: a tool for internal reflection and assessment, helping inform future strategy; a prompt for change and improvement to a charity’s work; and the creation of opportunities such as funding, communications and collaboration.

What's Different About Evaluating Advocacy and Policy Change?
Julia Coffman, Harvard Family Research Project, 2011
An article covering four ways that evaluating advocacy is different from traditional programme evaluation.