DfES report on the Impact of Advice and Guidance
03-June-2005
permalink comments (2) forum (2) email this2 Replies (comments)
1 Key Findings from DfES report on IAG
- The survey was successful in identifying a group of Advice/Guidance (A/G) recipients and a control group of those receiving Information (I) only, whose characteristics were broadly similar. Any slight differences were removed using propensity score matching. This is important in attributing any differences in the outcomes of the two groups to the impact of the A/G intervention.
- The distinction between the two groups was validated by their different patterns of service use, with A/G users having had more contact with a larger number of providers on average, having sought help with their careers or planning their future in greater proportions than I-only recipients who were more likely to be seeking help with learning.
- The A/G users were more positive than the I-only group about their current/previous work and learning achievements and their current labour market position. This may reflect some form of impact of the higher-level intervention they have received, but without a true baseline measure the fact that the A/G group are simply more positive about everything cannot be discounted.
- There are clear differences in the work and learning outcomes and in changes to the levels of confidence, motivation and opportunity awareness between the two groups. In all cases, the A/G group are significantly more likely to report having undergone changes since their intervention as a result of the help they have received. This study, therefore, provides evidence that the intermediate outcomes of Advice/Guidance recipients are greater than individuals receiving Information only.
Details of the research
The sample of respondents was randomly drawn with the help of three suppliers of IAG services: learndirect, Jobcentre Plus and 24 Information, Advice and Guidance Partnerships (since re-branded as Nextsteps providers). These providers drew the contact details from their records of individuals using IAG services between October 2003 and March 2004. Individuals were interviewed by telephone for an average of 30 minutes. The achieved sample of 4,361 interviews includes 2,979 individuals referred by learndirect, 986 by IAG Partnerships and 396 by Jobcentre Plus.
Individuals were split into two groups for analysis, dependent on the nature of the help they had received. Those receiving just Information (the I-only group), and individuals who felt that they had received more than this (the A/G group).
Respondent details
- Just over 60 per cent of the sample were female and a similar proportion were aged 35 years or younger.
- Around 20 per cent were minority ethnic respondents, and 15 per cent either had a health problem or disability and just over half had dependent children.
- 17% of the sample were single parents and just over 40 per cent were not active in the labour market.
- Half the sample held a level 1 qualification, a quarter held level 2 qualifications and the same proportion held level 0 or no qualifications.
2 Response to Deirdre`s request for comments on the DfES report
This ambitious and long-awaited report was a bit of a disappointment for me. I think it raises as many questions as it provides answers. In particular, the use of a "control group" (Information only recipients) to compare with the "treatment group" (Advice and Guidance recipients) was a methodology that promised much but delivered little, though to be fair to the authors there were many unavoidable constraints and this is only the first phase with hopefully more to follow.
The authors say that members of the A/G group were more likely than those in the I-only group to have made changes in relation to a whole range of attitudinal, learning and work outcomes. This is one of their main conclusions. This sounds very promising, especially when we read that the two groups were matched using "propensity score matching" to try and rule out factors other than the intervention having an impact. Unfortunately, because of a number of factors including lack of co-ordinated MIS data prior to the study, timing of the study etc, the only way the researchers could establish baseline information was after the intervention. The significance of this is that the attitudinal differences noted between the two groups could have been there before the intervention. In other words, the fact that the A/G group were generally more positive about a whole range of things could be because that`s the kind of people they are. Since participation in service is voluntary, this might also explain why some (the A/G group) were more motivated to actively seek out the advice and guidance support than others were (the I-only group).
To be fair, the authors do recognise this potential limitation and they indicate that further follow-up work (subject to funding) could help make the results more robust. This may be the case; however, for me the study as a whole does reinforce the intrinsic methodological difficulties in controlling the many varied and overlapping factors involved in advice and guidance interventions and their possible outcomes. It may be that the only true, controlled test is where (as in some evaluations of "Welfare to Work" programmes in the USA) clients are randomly assigned to an "intervention recipient group" and an "intervention non-recipient group".
What do others think?
Geoff Gration
CeGS Associate
Linking and trackbacks
When linking to this weblog entry, please use the 'permalink', which is http://www.guidance-research.org/collaborate/guidance/entries/4481160190
