Monitoring, evaluation and impact

London Funders' publications
It's child's play
(2010) Summary of a conference organised by London Borough of Tower Hamlets and London Play, looking at monitoring and evaluation expectations and practice, especially in relation to the funding of children’s play.

Useful resources

Impact measurement practice in the UK third sector 
Jenny Harlock, Third Sector Research Centre, University of Birmingham, 2013
A research review that argues there is a long way to go before impact is measured and understood consistently. A wish to demonstrate impact is not new to the VCS, but pressures such as the shift to outcomes-based commissioning mean an increased proportion of VCS resources is being devoted to impact measurement activities. The review notes that impact measurement has been largely focused on individual organisations but that the development of relevant shared measurement and tools remains a challenge.

NB London Funders’ meeting on 26 September, Youth Work Works?, includes a look at Project Oracle, the youth evidence hub. See here for more information and to book.

Intelligence to transform local infrastructure 
LVSC offers an interesting exploration of how new technology and open data can help the VCS provide evidence of need and impact. They held a “data jam” recently and some of the results can be found here:  ‘how to use research’policy evidence and new ways to look at impact. For more information, email policy@lvsc.org.uk.

Measure impact on young people’s well-being – NPC’s Well-being Measure
New Philanthropy Capital has launched a product to help charities, schools and youth organisations measure their impact on young people’s well-being. It’s an evaluation tool designed for surveying across a group or cohort of young people aged 11 to 16. Each subscribers has their own online account where they can create surveys and track how well-being changes in eight areas, including self-esteem, emotional well-being, resilience, and friendships. It’s flexible so you can choose from this set of validated measures and add your own questions if you wish. Once complete, you can download a report of your results. NPC’s Well-being Measure is supported by the Esmée Fairbairn Foundation, the Paul Hamlyn Foundation, and the Private Equity Foundation. For more information visit www.well-beingmeasure.com

Evaluating Well-being: Learning from the Big Lottery Fund's five year evaluation of the Well-being Programme
The final results of the Big Lottery Fund’s five-year national wellbeing programme have been published and confirm that positive improvement can be achieved with well-targeted action. The programme funded projects throughout England to encourage healthy eating; increase physical activity; and promote good mental health. The evaluation teases out factors that influenced different levels of improvement. Peabody’s Activate London and MIND’s national project Time to Change were both contributors to a lot of learning about what works.

SROI Network
Tools and resources on this website on social return on investment including a demystifying paper here.

Project Oracle: new resources: the Mayor of London’s programme for “understanding and sharing what really works” in preventing youth crime and in improving the lives of children and young people in London. The website invites funders, providers and advisers to register on the site. Effort has been made to keep the functionality simple with the potential to develop further.

The development of Project Oracle
Nesta and Mayor of London, 2012
This paper examines why "many services are unable or unwilling to measure the improvements they make in outcomes for young people” and explains how Project Oracle is remedying this in London by linking youth programmes with academically rigorous and internationally recognised standards of evidence to improve consistency and quality in understanding what does and does not work. Small and large projects are signing up to make use of Project Oracle’s resources. This paper is a summary of the need and the context for Project Oracle, and its experience so far.

Center for Evaluation Innovation: The Center's publications database can be searched for resources on evaluating advocacy. 

 

External Publications

The ambitions and challenges of SROI
Malin Arvidson, Fergus Lyon, Stephen McKay and Domenico Moro, Third Sector Research Centre, 2010
The benefits and the challenges of using SROI, including the high costs of using the SROI tool and its limitations. The paper looks at the origins and use of SROI before identifying some emerging challenges. The authors draw out implications for both those using impact tools to demonstrate the value of their work, and those interpreting the results of SROI exercises which do not easily lend themselves to comparison across organisations and projects.

Assessing Change
Charities Evaluation Services, 2010
A handbook of practical ways to identify and collect information on the outcome of work and the impact it has made. It provides help in designing a monitoring tool or choosing one already available.

Annual Review 2015
Big Society Capital (May 2016)
The review focuses on the unemployed, vulnerable young people and children, people with mental health and children services.

The Big Society Audit 2013
Caroline Slocock (December 2013)
The Big Society Audit 2013 looks beyond the rhetoric of this controversial policy to find out what’s been happening in practice.  There are some positives but also a large gap between actions and words, particularly in relation to public services. The Big Society Audit focuses on three key strands identified by the Government as key to its delivery when it was launched: community empowerment, opening up public services and social action.

Beneficiary involvement framework
Earlier in 2013, the Institute for Voluntary Action Research (IVAR) undertook a study for BIG Lottery Fund exploring the way BIG involves beneficiaries in their funding processes. In addition to a short findings report, a practical framework was also produced for thinking about beneficiary involvement. The framework poses questions that BIG might consider when deciding whether and how to involve beneficiaries in specific programmes or other funding processes and is available now for other funders and those interested in grantmaking practices.

Breakthroughs in shared measurement and social impact
FSG Social Impact Advisors, 2009

Reports on nonprofit organisations measuring their performance on common indicators and shared evaluation platforms using web-based systems. Breakthroughs include shared measurement platforms (an agreed-upon set of benchmarks developed by funding organisations and their grantees), comparative performance systems (ways to compare results between different grantee organisations) and adaptive learning systems (using the leverage of both of those systems to develop strategies and coordinate resources between multiple foundations and grantees). 

Can ICT help your monitoring and evaluation?
Charities Evaluation Services
Guidance for voluntary organisations on the selection of a suitable IT system or database to support their monitoring, as a response to evidence that this was a key barrier to such work – what is required from a system and which off-the-shelf examples are cost-effective. 

Centre for Market and Public Organisation research: impact of Lottery grants
Helen Simpson and Sarah Smith (2013)
The Centre for Market and Public Organisation at Bristol University analysed a sample of more than 5,000 grant applications made by the Community Fund and concluded that receiving a grant made it more likely that a charity would survive and that their incomes would be higher, with a positive effect for up to 4 years in smaller charities. Unlike the more targeted US state lotteries, National Lottery grants were spread widely across many areas, which may account for the positive impact. The results were published in the autumn issue of Research in Public Policy.

Charity Commission: revised public benefit guidance
Charity Commission (2013)
The revised guidance has been published and is available in online and hard copy formats. It is split into three short, high level guides: Public Benefit: The Public Benefit Requirement, Public Benefit: Running a Charity and Public Benefit: Reporting.

A chief digital officer for all Londoners
Centre for London (June 2016)
Following the commitment made by Mayor of London, Sadiq Khan, to appoint a Chief Digital Officer (CDO), this report, produced in conjunction with London First, sets out the main requirements of the role.

Community Action and Social Media, TSRC (January 2017)

The literature on the social uses of social technologies relates predominantly to its political impact, much of it arguing for the transformative power of social media. The third sector literature is dominated by material relating to marketing and fundraising for charities, with little research on community organisations’ use of social media.

This working paper - Working Paper 139: Community Action and Social Media - a review of the literature - reviews over 400 items and outlines key themes including: lack of evidence that social media use contributes to lasting empowerment, equalities or social justice; and that networked individuals may now carry out community action roles more efficiently than organisations.

Download here

 

 

Community capital – the value of connected communities
RSA (October 2015)
The vision of ‘Connected Communities’ is one in which people are embedded within local networks of social support; social isolation is reduced and people experience greater wellbeing and other benefits from the better understanding, mobilisation and growth of ‘community capital’ in their neighbourhoods.

The Connected Communities programme explored this vision by surveying residents in ward-sized localities, analysing this data for insight into local social networks and wellbeing, and then working with local people to build projects that support social connections.

Community investment by social housing organisations: measuring the impact
Survey Report for HACT, 2012; researchers: Vanessa Wilkes and Professor David Mullins, Third Sector Research Centre
This picture of the measurement tools being used by over 30 social landlords shows that while there is general recognition of the importance of measuring impact, there are also concerns about cost, approach and potential duplication and wide variation in the approaches used. This proves to be a complex area with no easy choices. Organisations that have developed in-house tools are generally less satisfied than those using external tools but only one external tool has been specifically designed for the sector. There is a strong interest in measuring joint outcomes, for example where housing associations join with other agencies to invest in neighbourhood based initiatives, but very little existing practice. Full research reportSummary report by HACT.

Carnegie UK Trust, September 2016
Addressing the digital divide is one of the great social challenges of our age. Digital Participation and Social Justice in Scotland examines the link between being offline and other forms of social deprivation. Drawing on detailed statistical analysis by Ipsos MORI, tells us who is offline, why and what we can do about it.

Does Your Money Make a Difference?
Revised and updated second edition of CES’ good practice guide on monitoring and evaluation for funders originally developed from a study for the National Audit Office. It covers issues such as proportionate monitoring, intelligent funding and “funder plus” activities, with examples, links to further reading and a good practice self-assessment model. London Funders’ Research and Evaluation project group assisted with this edition.

Evaluating social innovation
FSG, 2012
FSG is a US-based non-profit consultancy firm. This research report, collaboratively written with the Center for Evaluation Innovation, challenges grantmakers to explore the use of developmental evaluation when evaluating complex, dynamic, and emergent initiatives. It includes several concise case studies. See also London Funder's meeting report, In the eye of the beholder? Funding Innovation

Evaluation in philanthropy: perspectives from the field
Grantmakers for Effective Organizations, 2009
A stimulating six-page executive summary from this US donor association on how funders, and their funded organisations, can use evaluation as a learning tool, not simply for accountability. Access to the executive summary is free but GEO membership is required for the full report. All examples illustrated are US ones but the material is accessibly written and includes pithy examples around specific themes such as:

·         How evaluation can achieve continuing improvement and not just evidence of outcomes: 

·         How it can help explain change, beyond describing success and failure

·         How evaluation can involve a range of actors and share learning widely

·         What the funder can learn, not just from one grant but a cluster or a whole programme

·         How failure can be worked hard to provide lessons

Evidence for social policy and practice: perspectives on how research and evidence can influence decision making in public services
NESTA, 2011
Six essays from UK and US organisations using different methodology and approaches to generate evidence and influence policy and practice in a number of service areas. It challenges the limited rigour with which much UK social policy is evaluated and argues that at a time in the UK of public service reform and a push for decentralised decision-making, the need for accessible and reliable evidence is more important than ever. Includes a clear summary of the GLA’s Project Oracle.

Four essentials for evaluation
Grantmakers for Effective Organizations, 2012
Outlines how to create a culture that values evaluation for learning; plan a framework to ensure all stakeholders evaluate with a purpose; organise needed infrastructure and technology; and share lessons with grant recipients and partners.

A funder conundrum: choices that funders face in bringing about positive social change
DP Evaluation, The Diana, Princess of Wales Memorial Fund, 2012
Fund-wide evaluation in preparation for the planned closure of the Fund. Staff and trustees wanted to learn and share as much as possible from their experience and in particular on the choices funders face in their work. The evaluation began from three questions: how can funders create change in policy and practice, does “funder plus” add value, and does working in collaboration make funders more effective?

The Good Investor
Investing for Good, 2013
A guide (commissioned by Big Society Capital in partnership with Deutsche Bank) offering a standard framework to support rational and effective decision-making for impact investment.

Grassroots arts 
Third Sector Research Centre, Universities of Exeter and Glamorgan, Voluntary Arts, 2012
A report finds that arts activities help improve the well-being of both individuals and communities, increasing feelings of self-confidence and self-worth. The report notes the significant impact activity can have for minority groups such as disabled people, reducing isolation, increasing social networks and enhancing quality of life. The research partners intend to develop a toolkit for assessing the outcomes of amateur community arts activities. 

Harmonising reporting
Scotland Funders’ Forum, 2009
This practical working group report builds on the work of Turning the Tables which was a project commissioned by the Scotland Funders’ Forum and funded by BIG Scotland. The Forum followed it up with funders prepared to test out the issues more, and this report shows how to make reports more useful for funders and make reporting more useful and less burdensome for funded organisations.

Impact measurement  
A report on a discussion between funders, led by Big Lottery Fund and New Philanthropy Capital: looking at the different stages of grantmaking and considering what role funders can play in requiring, supporting, signposting or funding impact measurement – particularly for smaller organisations struggling to know what to do.

Increasing impact, enhancing value: a practitioner's guide to leading corporate philanthropy
Council on Foundation, 2012
The product of a year-long field research project, this guide offers a five-point framework for redefining corporate philanthropy. The points are: create a new narrative around corporate philanthropy as an investment in society; develop an inclusive "operating system" for philanthropic investment; improve collaboration, communication, and knowledge sharing; mobilise field leadership behind the agenda; and professionalise the field.

Investing for impact: practical tools, lessons and results
NPC (November 2015)
Investing for impact shares lessons learnt and the tools used to measure the social returns achieved through the Foundation’s investments. Among these is a new framework, NPC’s Impact Assurance Classification, which is a starting point for comparing the quality of impact practice across asset classes and sectors.

Legacy Trust UK: evaluation report
Legacy Trust, 2013
This final evaluation report on the Trust’s support of arts, sports and education projects across the UK as part of the cultural celebrations of the Olympic and Paralympic Games 2012 shows a strong impact in engaging over one million children and young people in new, creative activity, and significant numbers of volunteers and emerging artists, and leveraging new funding.

London 2012: have we inspired a generation?
Legacy Trust, 2013
A short report on the Trust’s strategic grantmaking. 

Monitoring and evaluation of family intervention projects and services
Department for Education, 2011
An evaluation report on the experience and effectiveness of family intervention services from 2007-11. Valuable background to the coalition government’s new programme to create a national network of Troubled Family Trouble-Shooters for radical transformation in the lives of “the country's most troubled families” (see CLG press release here).

Monitoring and evaluation on a shoestring
Charities Evaluation Services, 2010
This National Performance Programme practical guide recognises the limited budgets available to many organisations for their monitoring and evaluation.

Measuring Impact: How can TSO’s make sense of a rapidly expanding marketplace of tools? Working paper
Matcalf, October 2013
This TSRC research takes a broad look at the range of impact measurement tools available to this sector organisations, and provides guidance on selecting and using these based on analysis of some of their main features.

Outcome and outcome indicator banks: availability and use
Avan Wadia and Diana Parkinson, Charities Evaluation Services, 2010
This research investigates outcome indicator banks - the bringing together of a range of methods already used to demonstrate outcomes. It identifies existing banks, how they are being used, their strengths and weaknesses as a resource; and gives recommendations for further development. 

Pathfinder: a practical guide to advocacy evaluation
Innovation Network Inc.
An introduction to advocacy evaluation from an advocate or funder's perspective. It recommends learning-focused advocacy evaluation, which yields the type of information funders and advocates need to understand their progress. Click here for the Funder version or the Advocate version.

Payment by results contracts: a legal analysis of terms and process
David Hunter and Ruth Breidenbach-Roe, (2013)
This report from NCVO and BWB gives an analysis of a sample of payment by results (PbR) contracts held by voluntary sector providers.

Positive Change in Challenging Times: How Spice Time Credits are creating system change
Spice, July 2016
Spice Time Credits are a window into a possible future. The model is practical, not theoretical. It’s shown its usefulness in many places. And it has now reached a point where it deserves to be much more widely known

A practical guide to measuring and managing impact
European Venture Philanthropy Association, 2013
A step by step, quite technical guide through the process from the perspective of venture philanthropy. 

Principles into practice: how charities and social enterprises communicate impact
Charity Finance Group, ACEVO and New Philanthropy Capital, 2012
A report addressing the challenges of impact measurement, setting out six general principles that define how charities should communicate their impact: clarity, accessibility, transparency, accountability, verifiability and proportionality. Each principle is based on making clear links between a charity’s activities and its aim and mission. The principles come to life in nine compact case studies. London Funders attended the launch of the report to hear from charities involved in the research and the challenges impact measurements poses for funders. Read our comments here.

Putting evaluations to use: from measuring to endorsing social value, working paper
Arvidson and Kara, (2013)
This TSCR research reports on the role of impact measurement in helping organisations and commissioners to identify the social value produced by organisations delivering services. The context of the recent Social Value Act is considered and the role of the leading impact measurement frameworks is assessed.

Reframing the conversation: expanding the impact of grantees
Grantmakers for Effective Organizations, 2011
This report, sub-titled How do we build the capacity of nonprofits to evaluate, learn and improve?, outlines ways to boost VCS capacity to systematically collect, analyse, and learn from data, including helping link programme goals to evaluation questions and underwriting the costs of new technology and evaluation skills training.

Refreshing perspectives: exploring the application of peer research with populations facing severe and multiple disadvantage
Revolving Doors (May 2016)
The paper focuses on how peer research has been carried out with groups including prisoners, homeless populations, people on probation and drug and alcohol users. Peer research is the active involvement of people with lived experience in designing or delivering research.

Non-Profit Quarterly, October 2016
Research and evaluation have long influenced activities in the nonprofit sector. Nonprofit organizations are increasingly establishing or being asked to establish metrics or to conduct evaluations for a variety of reasons. Indeed, research, evaluation, and data regularly inform philanthropic and policy decisions, and vice versa. We live in a data-driven society that has furthermore become obsessed with big data, a term that describes the ability to collect and analyze data on every participant or even every transaction. Thus, instead of periodically surveying or interviewing a sample of a nonprofit’s members or employees, organizations can now create and use databases that measure every contact, activity, donation, or half hour of an employee’s time.

Scaling social impact: a literature toolkit for funders
Social Impact Exchange, 2012
A literature review on taking projects and services to scale. 

Scaling Social Impact in the UK: Insights from Scale Accelerator
International Centre for Social Franchising, September 2016

In 2015 ICSF partnered with nine of the UK’s largest Trusts and Foundations to deliver the pilot Scale Accelerator. The aim of Scale Accelerator is to support promising social enterprises and charities to replicate and scale the proven impact of their programmes.

The Scaling Social Impact in the UK report shares 8 key insights from Scale Accelerator. ICSF hopes the insights will be of benefit to social purpose organisations aiming to scale their impact and those who wish to support them to do so.

Secret of success: how funders use and share evidence in practice
Charlotte Ravenscroft, Alliance for Useful Evidence, 2013
Insights from interviews with funders from eight foundations on how they use and share evidence. It unearths many of the dilemmas about how to differentiate good evidence from good outcomes and how evidence can be collected and analysed more effectively (and how smaller organisations can be supported in collecting evidence rather than a “whole bunch of data”). It also tackles questions around funders sharing evidence with each other and with practitioners and policy-makers. 

Shared measurement: Greater than the sum of its parts
Inspiring Impact (February 2016)
This report discusses the benefits and challenges associated with shared measurement. Through analysis of 20 approaches, it examines how it is developed and draws lessons for future initiatives.

SROI for funders
Lucy Heady, New Philanthropy Capital, 2010
Social Return on Investment is a type of economic analysis that focuses on listening to stakeholders, identifying outcomes and giving these a financial value. This report is written specifically to help funders understand its role and how funded organisations might be helped to use it as a reporting tool.

Strategic co-funding: an approach for expanded impact
GEO (Grantmakers for Effective Organizations) and REDF (a US venture philanthropy network), 2012
This is the latest publication from a multi-year initiative, Scaling what works, and while written for an American audience and using US examples, is a practical guide to the reasons for and methods of jointly financing projects and programmes.

Sustainable Livelihoods Resource Book, August 2016
The Volunteer Training Company and Manchester Metropolitan Universty, August 2016

The resource book was designed to support community groups and VCS organisations to critically evaluate their assets and vulnerabilities.

The resource book will empower small VCS groups to conduct an assets based organisational evaluation. In identifying assets, small voluntary and community groups can then focus on their strengths. Groups can identify their organisational assets and are then able to build a strategy, which will enable them as a group.

Tools to support public policy grantmaking
Martha Campbell and Julia Coffman, The Foundation Review, 2009
This journal article, especially recommended for funders, offers guidance on how foundations can frame, focus and advance efforts to achieve public policy reform. It covers five essential steps for developing public policy strategy, and offers two tools to support foundations during the strategy development process.

Understanding the whole person: a series of literature reviews to support the research network
Revolving Doors (November 2015)

This series will explore aspects of severe and multiple disadvantage, or multiple and complex needs. The reviews support the goals of the research network by drawing on different disciplines and sectors, and building the evidence base. The first one examines common concepts across research into recovery from mental illness, desistance from crime and recovery from substance misuse. Future reviews may address causes of severe and multiple disadvantage, routes out, interventions and 'what works', methodological trends in research, and more. 

Universal set of principles for impact reporting
New Philanthropy Capital, with six voluntary sector partners, 2011
Principles out for consultation in late 2011 and early 2012 and intended to inspire better and widespread impact reporting. 

User's guide to advocacy evaluation planning
Julia Coffman, Harvard Family Research Project, 2009
Four basic steps that generate the core elements of an advocacy evaluation incl including what will be measured and how. 

Using research evidence
Nesta (January 2016)
A practical guide to help organisations use evidence to improve their work and make better decisions.

Value of charity analysis: how reviewing your organisation can help you achieve more
Esther Paterson and Iona Joy, New Philanthropy Capital, 2011
How organisational reviews can help charities be more effective and achieve more with their resources. Three main benefits of analysis emerge: a tool for internal reflection and assessment, helping inform future strategy; a prompt for change and improvement to a charity’s work; and the creation of opportunities such as funding, communications and collaboration.

NPC, September 2016
There are over 10,000 charitable foundations in the UK. Between them, they generate a vast amount of data. But this data has not traditionally been seen as a resource in the same way that money has been, and this is a missed opportunity. In this report NPC outlines how grant-makers can use data at both an individual and a collective level to improve their funding practice.

What's Different About Evaluating Advocacy and Policy Change?
Julia Coffman, Harvard Family Research Project, 2011
An article covering four ways that evaluating advocacy is different from traditional programme evaluation.