Guidance Community Discussion Space

Regarding the strap line "We can`t measure all that matters!" This is paraphrases the famous quote from Albert Einstein. I found it on Google under famous quotes:” Not everything that can be counted counts, and not everything that counts can be counted."

I think you will agree that this is really neat and this would be an even better motto for our search for evidence of impact.

Coming at this from the angle of practitioner I'm very conscious that working with clients from time to time you are aware that the guidance intervention has 'had an impact' but the difficulty is in demonstrating this to e.g. funders who are looking for something more tangible than my opinion. Perhaps subjective anecdote needs somehow to metamorphose into a more robust evidence base, and from this perspective impact analysis needs to find a mechanism by which to capture those impressions. Professional satisfaction, and therefore performance, may relate to impact analysis, but how can this be judged?

This is written from a research perspective on the Challenges for research into guidance to provide an evidence base to inform policy and practice

One important issue is whether the knowledge claims of research into guidance can be justified beyond the particular context in which the research occurs. This is a particular challenge where the research is complex, affected by many variables and in many cases is interventionist in that it is trying to improve practice. Additionally, many studies rely on narrative accounts to communicate and justify their findings. The narratives though are written from a particular perspective and may be open to very different interpretations.

One key question here is therefore what is our basis of understanding guidance encounters as outlined in research studies? How can we demonstrate a basis of our knowledge claims to others? Can we build evidence-based knowledge to help answer complex questions about the nature and outcomes of guidance, and is our idea of what constitutes an appropriate evidence base different from the conceptualisation of evidence-based practice of policy makers (and paymasters at the Treasury)?

We are not confident that we can fully answer such testing questions, but within this space for discussion of and reflection on research we believe that we can make a number of claims. We will seek to discuss (ideally theory-informed) research that focuses upon practice, work collegially with practitioners, policy-makers and researchers, in the co-construction of knowledge about guidance. We will acknowledge the range of issues facing practitioners and the variety of contexts within which guidance takes place and how guidance is adapted to these different contexts, in ways that recognise the limits of application of particular theories. However, by highlighting the specifics of practice it should be possible in an iterative fashion to adapt and sharpen theories and explanations of guidance in context.

It is inevitable that for policy makers, impact analysis might be expected to reflect how career guidance links to government agendas. Such expectations could shift analysis of impact from holistic investigation and enquiry, to the potentially more suspect arena of meeting the requirement to demonstrate success against project criteria. A careers adviser may sense that soft outcomes such as increased motivation, increased self confidence or greater self awareness are beneficial to individuals, but how can that perception be translated into a more robust measurement? If it is possible to find a mechanism for impact analysis that relates to professional experience, then it could be a basis for improving practice. Subjective anecdote needs somehow to metamorphose into a more robust evidence base, and from this perspective impact analysis needs to find a mechanism by which to capture those impressions. Employing organisations inevitably dictate how practitioners spend their time.

Impact Analysis could take a different form, it might link to broader aims such as partnership development, maintenance of a service, productivity judged in terms of numbers of clients seen with effectiveness of individual interactions placed in a broader context of efficiency. A cynic might argue that sustainability of project funding may at times seem to link more to correctly completed paperwork than the quality of an individual client’s experience. For the individual on the receiving end of guidance impact analysis may be even harder to pin down. It may link to the extent to which a client’s expectations have been met, but given that some research suggests expectations of careers guidance are at times pitifully low, that may be an insufficient guide. Equally a client may initially perceive their encounter with guidance negatively because it was challenging in some way, but over time recognise this as having value in helping them to face up to issues that needed to be addressed. In isolation, the client perspective, though pivotal, may not be enough.

The indicators that determine success can conflict with one another. For example, a unit providing impartial advice to individuals enquiring about college courses may be judged as performing poorly if the time spent on dispensing advice does not translate into the expected number of enrolments. However, if the service was to be judged in terms of the retention rate of students who had enrolled on courses following their advice sessions it might be found to score very highly. Much guidance is linked to outcomes that may only emerge over time. These outcomes are particularly hard to measure, and it may be hard to link progression to any single intervention where a client may be seeking support from a wide range of different sources. It may be that somebody ‘dropping out’ from a course following guidance, would be judged by policy makers as inherently negative and a ‘failing’, yet case study research might indicate that for the individual concerned this was a positive outcome, leading to an alternative that was more appropriate for the individual concerned.

I recall the following contribution being made, but I can't remember who by, but the observation seems pertinent...

When a patient "gets better" following treatment, how do we know he/she wouldn't have recovered any way, by a process of self-healing; and if "getting better" really was linked to the intervention, how do we know that it was the direct effect of the treatment itself rather than the psychology of the placebo effect?

If the research complexities facing disciplines involved in human interaction are similar, then so too are many of the methodological solutions, or attempted solutions.

Are comparisons valid and could they help us arrive at new and more effective impact methodologies, or at least a better understanding of the issues involved? Or could it be a bit of a wild goose chase?

There is much to be learnt from our colleagues in the Museums and Libraries sector. Inspiring Learning for All, which asked museums, archives and libraries to look beyond the formal learning context and to assess the impact they have on their informal learners and visitors.

This is a contribution made by an participant in discussions during the development of the website

The challenges and complexities facing the guidance community in trying to measure and evaluate the effectiveness of guidance are not unique. Other disciplines involving human interaction, such as medicine and social welfare, face similar challenges and difficulties in evaluating the effectiveness of practice. For example, when a patient "gets better" following treatment, how do we know he/she wouldn't have recovered any way, by a process of self-healing; and if "getting better" really was linked to the intervention, how do we know that it was the direct effect of the treatment itself rather than the psychology of the placebo effect? Similarly, if an unemployed adult successfully returns to work following guidance, how do we know that this outcome would not have happened any way; how do we know that such an individual didn't already have a level of motivation that both inspired him/her to seek out support in the first place, and led the individual to proactively seek employment?

If the research complexities facing disciplines involved in human interaction are similar, then so too are many of the methodological solutions, or attempted solutions. For example, we know that a feature of much medical research, and of some guidance research, involves the use of an "experimental group" (patients/subjects/clients who get the drug/treatment/intervention) who are then compared with a "control group" (those who are similar to the experimental group in most respects except that they don`t get the drug/treatment/intervention). Unfortunately, for both disciplines there are often ethical and practical difficulties with the use of control and experimental groups. As a result, other techniques have to be used such as making comparisons between large-scale population samples, the so-called "control by calculation" technique. For example, it is tempting to speculate that the large scale collection of data relating to the outcomes of "enhanced services", recently implemented by the LSC, will be used in this way.

In order to explore the similarities between impact research within the medical and guidance fields, and to assess what, if anything, the guidance community can learn from this, the School of Health, Education and Science at Derby University has been approached to identify one of their colleagues to join us in our on-line discussions.

In the meantime though, do any of our guidance colleagues have any comments on what we might learn from other research disciplines? Are the comparisons valid and could they help us arrive at new and more effective impact methodologies, or at least a better understanding of the issues involved? Or could it be a bit of a wild goose chase, or a methodological cul de sac? What do you think?

This was an observation made by a participant in a discussion during the developmental stage of this website.

I decided to venture outside of guidance to a sector where there is many a common thread and shared ambition, and a shared experience of restructure and new funding models. Having stepped out of the comfort zone-the guidance community, I was pleasantly surprised by what I found. Indeed I was inspired by a community where the value for money debate is raging but there is very commitment to developing a toolkit, with adverse range of tools to provide clear evidence of benefits. There is much to be learnt from our colleagues in the Museums and Libraries sector. Inspiring Learning for All, however, also asks museums, archives and libraries to look beyond the formal learning context and to assess the impact they have on their informal learners and visitors. In many cases these comprise the majority of their users. Inspiring Learning for All endeavours to embrace all the requirements of these diverse institutions and practitioners. However it is not assumed that every action and evidence statement will be core to all museums, archives and libraries. It is designed to encourage every museum, archive and library to question what they do, why they do it and to assess their effectiveness.

"Inspiring Learning for All" asks museums, archives and libraries to look beyond the formal learning context and to assess the impact they have on their informal learners and visitors. In many cases these comprise the majority of their users. Inspiring Learning for All endeavours to embrace all the requirements of these diverse institutions and practitioners. However it is not assumed that every action and evidence statement will be core to all museums, archives and libraries. It is designed to encourage every museum, archive and library to question what they do, why they do it and to assess their effectiveness against what is widely agreed to be best practice. "If this has whet your appetite have a look at the Resource website. http://www.resource.gov.uk

Wider discussion is invited on the author’s observations that:

there is suspicion of why data on impact is being collected

evidence is seen as something to be manipulated

such imposed activities take time and resources away from the real work which needs to be done.

The findings from all the TLRP workplace learning project will become available from the TLRP website, and they could be pointed to from here. It might be possible to start some discussions about particular papers or articles using either the comment or discussion facilities.

See, for example, the journal article 'Learning as Apprentices in the Contemporary UK Workplace: creating and managing expansive and restrictive participation' by Alison Fuller and Lorna Unwin

Publication Date: Dec-2003

Publication: Journal of Education and Work

full contents viewable from: http://www.tlrp.org/dspace/handle/123456789/39

I'm suprised at the lack of a catagory on scientific research in the LMI trends - can anyone explain its absence? I'm not sure that it fits into any of those already posted.

Users of this part of the site might also be interested in this website which was developed by the Careers and Student Employment section of the University of Westminster. I think its of particular interest because it includes along with the 'usual suspects' women, mature students, students with disabilities and so on, a section on class issues and students with previous convictions. It is aimed at students themselves, rather than a practitioner resource, but I think there is much to be learned from it. Have any other users come across similarly innovative websites dealing with equal opportunities that we could share here?

http://www.wmin.ac.uk/main.asp?page=886