What was the issue you were addressing or working on?
Stakeholders in North and South Lanarkshire Partnerships agreed a theory of change for Reshaping Care for Older People in April 2012, represented in logic model format. The outcomes they want to achieve provided a framework for the evaluation. The problem to be addressed was how to determine the success of a programme comprised of a wide-ranging set of initiatives. Some may be more successful than others, some may encounter unexpected blocks that thwart progress for a time, whilst others may simply turn out to be unworkable in practice. A means of bringing the different initiatives together under a common framework was required.
What you did?
A wide-ranging set of initiatives has been developed in the partnerships of North and South Lanarkshire, and this presents a considerable challenge for determining success of the RCOP Programme. A Contribution Analysis (CA) approach attempts to determine individual project influence on an agreed set of outcomes and enables us to be explicit about the strength of evidence gathered from each project and the impact on outcomes. A total of 39 functional Change Fund initiatives are underway in Lanarkshire, although a number of enablers (e.g. Communications Officer, Programme Managers) are also being supported. Some required only pump-prime funding, while others will receive support over a longer period of time. The evaluation is focusing on these functional projects including those exclusive to the North or South Lanarkshire Partnerships and those jointly funded by both.
All funded initiatives have agreed which logic model outcomes they will influence and are providing evidence to demonstrate this. The evidence is being assimilated into a Contribution Story, which will enable credible causal claims to be made about the influence of the different initiatives on the observed results. The Contribution Story will be updated at six monthly intervals and completed by March 2015. Initial evidence suggests that the theory of change is supported i.e. many initiatives are generating evidence of outcomes achieved.
What were the outcomes - benefits or otherwise?
The 3rd Contribution Story was published in March 2014
This report summarises how robust the evidence has become as a result of ongoing discussion/support with individual initiatives. This is put in the context of where each contribution will impact i.e. in local communities versus hospitals, and in terms of the scope to influence community-based supports/alternatives to hospital admission. Given that RCOP has been running for almost three years now, it was felt that most initiatives should have gone beyond achieving their agreed short-term outcomes and the report focuses on the medium-term outcomes. The quality of evidence to demonstrate influence on the outcomes was set at a higher bar than for the second Contribution Story, because questions were now being layered in about whether or not the changes observed could have happened in the absence of the initiative under scrutiny, or be attributable to other influences unconnected with that initiative.
The majority of changes in the rating of evidence submitted were more robust rather than less so. The majority of initiatives had submitted reasonably robust evidence of achieving their short-term RCOP outcomes. Some initiatives already had reasonably robust evidence of medium-term outcome achievement and their evidence was sufficiently robust at that time to inform discussions around future developments. For those initiatives that had limited/weak evidence or no evidence of medium-term outcome achievement, further evaluation data is needed to determine if this can be strengthened. The detail of the Contribution Story is presented in four parts:
- An overview of RCOP initiatives relating them to where the investment was targeted
- The robustness of evidence against each RCOP outcome now (March 2014) and at the time of the previous contribution story (September 2013)
- A more detailed view of short- and medium-term outcome achievement
- The initiatives demonstrating robust evidence of achievement against their medium-term outcomes mapped onto a diagram to indicate where the maximum benefit in future may be achieved in relation to reshaping care for older people
The report is available here.
The main benefit of this work is that it provides a structure to consider the success or otherwise of the wide variety of initiatives that were awarded Change Funding. The Contribution Story will continue to be refined, both to explore alternative explanations for the evidence appearing to support the outcomes for each initiative, and to fill any gaps that still exist in the evidence needed to make credible causal claims about how they have influenced the observed results. The Story will be updated at six monthly intervals and will be complete by March 2015.