We can’t measure all that matters!

In simple terms impact analysis attempts to provide answers to the question 'Are we making a difference?' However, there was animated discussion about how that might be judged. It was recognised that there are a number of drivers that influence perceptions on what might be regarded as a 'positive impact', and different views expressed on the relative merits of the potentially conflicting, perspectives.

Drivers could take a number of forms:

Political – Does Impact Analysis reflect that the service provides value for money?

In a context in which successive governments represent the main funding source for adult guidance, it is inevitable that for policy makers, impact analysis might be expected to reflect how career guidance links to government agendas. There may be pressure to demonstrate achievement of pre-determined outcomes such as entry into employment or raised skills levels among particular client groups. Furthermore, this impact may need to be achieved within relatively short timescales. Such expectations could shift analysis of impact from holistic investigation and enquiry, to the potentially more suspect arena of meeting the requirement to demonstrate success against project criteria. That is, e.g. has a particular project met agreed outcomes that reflect the desire of government to pursue specific targets related to participation in learning, and does the investment of resources therefore represent value for money?

Professional. Does Impact Analysis help me judge if what I do is meaningful and of value to my users?

For the guidance professional, impact measurement may be more subtle. Practitioners may view the soft outcomes of their work as absolutely fundamental. Can the practitioner gauge the extent to which what they do matters and makes a difference to their clients? A careers adviser may sense that soft outcomes such as increased motivation, increased self confidence or greater self awareness are beneficial to individuals, but how can that perception be translated into a more robust measurement? If it is possible to find a mechanism for impact analysis that relates to professional experience, then it could be a basis for improving practice. Subjective anecdote needs somehow to metamorphose into a more robust evidence base, and from this perspective impact analysis needs to find a mechanism by which to capture those impressions. Professional satisfaction – and therefore performance, may relate to impact analysis, but how can this be judged?

Employer led. Does Impact Analysis suggest that practitioners are achieving organisational goals?

Employing organisations inevitably dictate how practitioners spend their time. Impact Analysis could take a different form, it might link to broader aims such as partnership development, maintenance of a service, productivity judged in terms of numbers of clients seen with effectiveness of individual interactions placed in a broader context of efficiency. A cynic might argue that sustainability of project funding may at times seem to link more to correctly completed paperwork than the quality of an individual client’s experience. A properly recorded National Insurance number may be more important to an employer than the practitioners claim to have ‘moved a client on’ in some significant, but intangible way.

Personal. Does Impact Analysis demonstrate that using guidance services will help me in some way?

For the individual on the receiving end of guidance impact analysis may be even harder to pin down. It may link to the extent to which a client’s expectations have been met, but given that some research suggests expectations of careers guidance are at times pitifully low, that may be an insufficient guide. Equally a client may initially perceive their encounter with guidance negatively because it was challenging in some way, but over time recognise this as having value in helping them to face up to issues that needed to be addressed. In isolation, the client perspective, though pivotal, may not be enough. Equally, it may be possible to show that an intervention has made a difference, but it does not follow that the difference is one that is valued positively by the client. (E.g. if they are placed in employment, but it is not sustainable, or linked to their particular talents or interests.) There remains a debate about the extent to which individuals – or policy makers – understand what guidance is, and therefore its potential.

The collective experience of the group suggested that the drivers that determine success can conflict with one another. For example, a unit providing impartial advice to individuals enquiring about college courses may be judged as performing poorly if the time spent on dispensing advice does not translate into the expected number of enrolments. However, if the service was to be judged in terms of the retention rate of students who had enrolled on courses following their advice sessions it might be found to score very highly. When conflict occurs, it may be that the dominant driver becomes the economic imperative, irrespective of the front line experiences of guidance practitioners. This creates a dilemma. In relation to Impact Analysis, there is a need for 'joined-up thinking' to use the cliché, but nevertheless some studies of impact are too narrow in focus to take account of the broader context of the work they attempt to evaluate.

Impact Analysis needs to address these conflicting drivers, and yet evolve a mechanism for communicating the effectiveness or practice against a wide range of possible outcomes that is sufficiently robust to be meaningful. There was some discussion around the theme of ‘objectivity’. Given that guidance is often more than an art than a science, this raises questions about the extent to which it is possible to illustrate success. There is an ever present danger that what is measured is that which is easy to measure, rather than that which is most relevant.

Much guidance is linked to outcomes that may only emerge over time. These outcomes are particularly hard to measure, and it may be hard to link progression to any single intervention where a client may be seeking support from a wide range of different sources.

How can Impact be measured?

The discussion around drivers led to consideration of how impact might be measured. It was agreed that to demonstrate impact, it was necessary to know the starting point of the client, in order to judge how any intervention had impacted on them. Impact Analysis therefore necessitates finding a means by which to capture that shift. It was agreed there is no single obvious way of achieving this, though a number of different approaches might be valid, appropriate and helpful.

'Distance travelled' might involve a comprehensive survey of clients’ pre and post interview. However, this approach could be intimidating and burdensome for a client, and not conducive to building rapport as an opening for effective guidance.

It could be argued that the effective practitioner will as part of good practice gain this information through a structured guidance interview. It was accepted that most careers guidance workers have been trained to begin by 'hearing the client’s story' and that involves understanding the starting point of the individual, in order to move the client on in some relevant and appropriate way during the course of an interview. Progression might then be reflected in any action points arising, or in asking directly of the client what they feel they have gained and/or will now do as a result of the interview, as part of the closing dialogue. However, it is recognised that for many practitioners the individual vocational guidance interview with time for reflection is now an unimaginable luxury, and it would be naïve to imagine everyone involved in giving IAG is consistent in their adherence to good practice when practical considerations rather than idealised theory become the main influences on day to day practice.

Prof Sampson, in Florida State University, has been looking at developing a system of output measurement, linked to assessing the extent to which individuals have moved in relation to skills, knowledge and attitudes, perhaps this approach may be found to have resonance in Impact Analysis in the UK.

Tol Bedford may provide a basis for analysing what goes on in a guidance interview with his FIRST framework. The mnemonic suggesting effective guidance according to him, should demonstrate movement along the areas of focus, information, realism, scope and tactic. How relevant do these goals seem in today’s context?

There is an issue around proxy indicators, which are meaningless without context, e.g. the individual who drops out of a course (a negative outcome) but enrols on and completes a different more appropriate course of action or study (a positive outcome). Around 50 have been identified that could be termed in this way.

It is increasingly recognised that different clients have different needs, and therefore different outcomes may be appropriate – this should be taken into account in any Impact Analysis enquiry.

What about forms of interaction that are based on e.g. group work, information provision or brief staff-assisted queries. Where the contact is more minimal, how does this influence the process of Impact Analysis?

What relevance do Impact Analysis studies have for guidance practitioners?

The immediate response of the group was that although relevant and effective Impact Analysis could be of fundamental relevance for guidance practitioners, in reality they were probably largely perceived as at best irrelevant and at worst actually threatening.

It was felt that Impact Analysis was sometimes linked to a sense of being judged. This observation led to discussion around the theme of peer review. Practitioners may welcome input from others who understand the issues they face as a device for improving practice, yet the experience of being watched and evaluated by a manager perceived to be operating from a different set of objectives might be viewed very differently. How Impact Analysis studies are viewed, may link to who is doing the studying, against what criteria and the audience to which any Impact Analysis study is to be addressed.

Impact Analysis is not unproblematic, yet the group were unanimous in agreeing that these obstacles need to be overcome. Whilst recognising the limitations of particular approaches, and the influence of the different drivers, it was felt that Impact Analysis is crucial to building an effective evidence base that could improve practice. If practitioners could be involved in Impact Analysis then they may be better placed to inform the various debates around guidance. By capturing 'soft outcomes' through documented case study or longitudinal follow up, practitioners might then be better placed to lobby policy makers and demonstrate the professional value of their work in terms other than those imposed by financial imperative. Impact Analysis studies might then cease to be threatening to practitioners and perceived as focusing on what is irrelevant or unreasonable, but instead regarded as a useful tool for collecting qualitative data that could be used to champion and disseminate good practice. For example, it may be that somebody 'dropping out' from a course following guidance, would be judged by policy makers as inherently negative and a 'failing', yet case study research might indicate that for the individual concerned this was a positive outcome, leading to an alternative that was more appropriate for the individual concerned.

The group felt that there is common ground to be found between policy makers and practitioners operating from a client centred approach. Outcomes are not necessarily mutually exclusive. However, at present there is a lack of shared language and dialogue, between the different groups. Practitioners may feel isolated and far removed from the decision makers who dictate how they work. Perhaps the development of this strand of discussion may help to bridge that gap. By providing a platform for individual practitioners to share concerns and views the process of knowledge creation will develop arguments that can be presented in a coherent and persuasive way so practitioners are able to influence the future direction of career guidance work.

How might Impact Analysis discussions develop?

At present there is a mass of evidence in existence, but it is often fragmented or collected and stored rather than put to any particular purpose. It was hoped that this strand of discussion might represent an opportunity to bring together some of this data so links can be made and findings synthesised to begin to accumulate an evidence base that informs practise, rather than existing in a vacuum.

It was felt important that the language used should be inclusive and recognise the different starting points of users of the forum. Contributions continue to be needed in a whole range of areas, but specific suggestions were made. One idea was that a straight forward argument could be put forward in terms aimed at practitioners to address the question 'Why bother with Impact Analysis' in order to engage and stimulate their involvement in the forum. Practitioners need to 'own' Impact Analysis studies, rather than feel victims of them. Understanding and influencing such studies could be to make them tools of empowerment, rather than tools for monitoring. An outline of some of the many proxy indicators that may distract from rather than illuminate what is going on in practice.

Research and academic contributions whilst highly relevant, and pivotal to this project, should be contextualised for practitioners, and where possible accompanied by abstracts and comment that draw out key points using accessible language that is jargon free. Further debate is required about why policy makers value hard outcomes and how can impact analysis meet that requirement. A celebration of soft outcomes might also remind readers that qualitative data is (or should be) especially pertinent to any study of impact.

In conclusion the group had a lively discussion that it is hoped will stimulate further contributions on the theme of Impact Analysis. It was agreed Impact Analysis studies not only have value, but are crucial to improving practice, however, there is much to debate about how to ensure such studies are relevant, meaningful and used to inform the future rather than just reflect on the past.