Editorial Comment:

Focus

The policy of the National Life/Work centre (which manages the development of The Real Game) is to reach national agreements with other countries wishing to promote and distribute the game. In the UK the first step in this process was the piloting of a 10 hour version of the Real Game in the Kent career Services area in 1997/1998. An evaluation of the first stage was carried out by NICEC and helped to inform the adaptation of The Real Game for the UK market. The results were sufficiently encouraging for Kent Careers Services to launch a phase-two pilot in which an 18-hour version was made available to Kent Schools during 1998/1999. The evaluation of the second phase was also carried out by NICEC.

At the same time as the second-year piloting of The Real Game in Kent, the DfEE awarded a contract to Kent Careers Service to manage a national pilot scheme. It is the evaluation of this national pilot that is the subject of the present report.

The study was done in the specific institutions because they were already piloting the programme with part of the year cohort so allowing us to compare results between experimental and control groups.

Aims

The aim of the study was to ascertain the extent to which the UK version of The Real Game achieved its intended learning objectives and to identify critical factors associated with its successful delivery and management.

The Real Game

The Real Game is an 18-stage career exploration programme for 12-13 year olds covering simulating the world of work. For part of the game pupils take on the role of worker and take on the vagaries of chance and change in their working lives. The class becomes a simulated society made up of small local communities. Pupils take on financial planning, choosing a holiday, and coping with enforced job change. Pupils do quizzes and work sheets on topics such as career relevance of school subjects, gender equality, time management, occupational awareness and the changing world of work. Depending on choices, they may be given opportunities to meet a range of adult workers - for example, if a career day is held. Pupils complete a questionnaire at the beginning and the end of the programme to find out how their career knowledge and understanding have improved. Five key messages run through the programme: change is constant; learning is ongoing; focus on the journey; follow your heart; and access your allies.

In the U.K. context distinctive features include: intermittent role-taking over a sustained period; linking careers education to personal finance education; an extensive learning programme to provide not just an element of PSHE or careers ed programme, but a framework for the whole programme for the particular year group.

Study Design

Quasi-experimental - with before (T1) and after (T2) testing. 16 out of 37 schools participating in the Real Game pilot agreed to take part in the quantitative survey of pupils. Schools trialed the Real Game in part of year groups and comparisons were made between Real Game participants and non-participants. 'World of work' questionnaire were administered in schools (with instructions and script from research team) for use with experimental and control groups - at the beginning and the end of the Real Game programme.

Additional information obtained from schools about pupils' age, gender, SATS scores, and entitlement to free school meals, missed sessions, was gathered.

Data Collection

Qualitative Evaluation: a member of the evaluation team visited twelve schools. It was expected that the evaluators would meet as many relevant staff as possible, with the intention of gathering feedback from internal co-ordinators responsible for managing The Real Game, teachers involved in classroom delivery, head teachers or other senior managers, careers advisers, observing lessons in progress.

Also each of the 37 schools in the pilot scheme was sent copies of a questionnaire inviting all teachers delivering the programme to identify the actual potential learning outcomes from the game, including those, which related to career learning, PSHE, Citizenship and Key Skills.

The 15 careers services 'attached' to Real Game schools were also sent a questionnaire. In addition, regular feedback from schools was obtained through the general process of coordinating the evaluation. (Telephone/correspondence).

Part of the quasi-experimental evidence with regards to learning outcomes was gathered by using the World of Work Questionnaire which pupils are expected to complete at the beginning and the end of the programme. The questionnaire was written by the developers and corresponds closely to the programme's learning objectives. It includes both knowledge and opinion items. Many of the items were relevant to our evaluation criteria. The knowledge items can be considered to constitute a 'knowledge test' linked to the objectives of the Real Game.

The other part was a supplementary questionnaire constructed by the evaluation team incorporating additional self-report measures. The basic approach was to concentrate on evaluative criteria, which were not already assessed through the World of Work Questionnaire. The scales which were developed for this part of the study fell into four groups:

1. beliefs in the utility of job exploration, in the utility of self-awareness/self-exploration in relation to jobs and in the utility of career planning. In each case, the utility of constituent activities etc. was rated on agreement-disagreement scales.
2. self-efficacy for the performance of age-appropriate job exploration activities, self-efficacy for self-awareness and self-efficacy for career planning. Items were rated on 'confidence' scales. Within Career beliefs items and within self-efficacy, the order of items was randomised. Career beliefs items were both positively and negatively worded.
3. assessed 'employability'. This was addressed exclusively in terms of 'employability' beliefs.
4. age appropriate items.

Teachers Questionnaires - alongside the quantitative measurement of the outcomes achieved by pupils, teachers involved in teaching the Real Game were invited to complete a Teachers Questionnaire indicating their perceptions of the learning outcomes achieved by pupils. The questionnaires were based on:

1. a topic-by-topic list of intended outcomes drawn from the tutor's handbook and written by the developers
2. a list of career-related learning outcomes devised by the authors of this evaluation at the request of the DfEE
3. a list of learning outcomes taken from the relevant sections of the draft non-statutory framework for PSHE
4. a list of Key Skills outcomes based on guidance from QCA
5. a list of skills and aptitudes related to Citizenship taken from a report by QCA

Careers Service Questionnaires - Each careers service associated with a pilot school was invited to contribute their views to the evaluation.

Visits to schools - feedback from interviews with: Internal co-ordinators responsible for managing The Real Game; Teachers involved in classroom delivery; Head teachers or other senior managers; Careers adviser; Observing lessons in progress

Findings

PARTICIPANT OPINIONS:

Pupils held broadly positive views of what The Real Game had taught them. Half or more of them said that they had learned 'quite a lot' or 'a lot' about each of the learning objectives on which they were questioned. Pupils appeared to believe that they learned more about 'present-day' career-related matters relevant to their current personal situations. Large numbers also believed that they had learned about working in groups and financial budgets. But they did not feel that they had learned quite so much about what might be (for them) more remote career issues such as recruitment criteria and the future shape of work.

Perceived learning was sufficiently independent of measures ability, sex and age (within the age group considered) for us to conclude that The Real Game appeared to be suitable for the generality of young people at this stage of their education.

THE WORLD OF WORK QUESTIONNAIRE:

There were robust results from this questionnaire. The Real Game sample showed a significant gain, relative to the comparison sample. This was reasonably consistent across schools, and the result remained significant when the effects of school were controlled: that is to say, the results for the sample as a whole were not a product of differences in sample sizes, not in the proportions of Real Game to comparison subjects, from school to school. In addition, when sex, age, and ability (SAT scores) were controlled (in the smaller sub-sample for which this information was available) the Real Game effect remained significant. Although not all items showed gains in the Real Game sample, a high proportion of them did, and these included not only 'soft' opinion items but also items forming part of the 'knowledge test'; indeed, the biggest gains were made, in the main, on knowledge items. Thus a test of relevant knowledge and opinions seems to have been influenced positively by participation in The Real Game.

EFFECTS ON THE NEED FOR INFORMATION, EMPLOYABILTY BELIEFS AND SELF-EFFICIACY:

No effects were found on need for information or knowledge of information sources. Employability beliefs were also unaffected.

For technical reasons, the other career beliefs of concern (in the utility of job exploration, of self-awareness/self-exploration and of career planning) were considered in total and at the individual item level. That is to say, the a priori scales were not considered independently of one another. In total, there was a very small gain in the Real Game sample from t1 to t2 and a small decline in the comparison sample. The gap between samples became statistically significant at t2, but this was attributable to the opposing directions of change in the samples. There was some consistency between schools in the sense that, in all except one of those for which data were available, The Real Game sample experiences either less decline, or a larger gain, than its corresponding comparison sample. However, as this implies, career beliefs scores did indeed decline in the Real Game samples in some schools, whereas a consistent pattern of increase would be more compelling. A similar pattern was revealed at the level of individual items: most showed either a greater gain or less decline in the Real Game sample, although this was not true of all of them. Thus, the aggregate outcome owes at least as much to less decline in some Real Game samples, as it does to gains relative to their own starting positions in others.

Subject to this caveat, the effect of Real Game participation remained when school and the somewhat different effect of the Real Game from school to school were taken into account statistically. It also remained true when sex, age-within-year and ability (SAT scores) were taken into account in the sub-sample for which this information was available.

A small gain was made in self-efficacy for job exploration in the Real Game sample; self-efficacy for self-awareness declined to a trivial degree in both samples; and a small gap opened between the samples in self-efficacy for planning, due in large measure to comparison sample decline. None of these changes were statistically significant in two-tailed tests.

Aggregate change was, however, merely the resultant of sharp differences between what happened in schools. For example, for the combined self-efficacy scale, only one school demonstrated the anticipated pattern of stability in the comparison group coupled with a gain in the Real Game sample. Elsewhere, modest improvements in Teal Game scores where accompanied by larger declines in the comparison sample, both samples declined, or Real Game losses were matched by comparison sample gains. All of the combinations of self-efficacy items alluded to above examined in analyses, which took school and the interaction of school with sample into account.

There were no significant Real Game 'main' effects.