3.1 Introduction

Once you have decided on a research topic, you will be able to decide how to collect the evidence you require. This section will be concerned with general issues associated with the selection of methods and appropriate procedures for the project.

3.2 Objectives: stage 3

By the end of this stage of your research project, you will be able to:

• identify appropriate criteria by which to judge the validity of a guidance project;
• approximate and estimate time involved, necessary materials and the quantities in which they are required, costs of the research, etc.;
• identify design principles relevant to the chosen research area;
• identify the major issues or problems in the specific research area;
• demonstrate the ability to re-formulate research design in response to unexpected circumstances;
• synthesise material, evidence and arguments to select appropriate methods and procedures for the project.

3.3 Validity, Reliability and Relevance

When judging the quality of any research, reliability, validity and relevance are important.

Validity

The validity of research refers to the credibility of the results. Has the research actually done the things it claims to do? Does an item measure or describe what it is supposed to measure or describe? In considering these issues, it needs to be acknowledged that discussions about validity provoke controversy amongst researchers. Some have rejected the concept of validity as inapplicable since it implies the possession of knowledge that is absolutely certain and, in this sense, knowledge can never be certain. McLeod (1999) argues that the concepts of validity and reliability that have been developed for use in quantitative research can’t be applied in the same way in qualitative studies. Nevertheless, in deciding the value of our own and other research, we need to be able to make reasoned judgements about it as a new contribution to knowledge.

Measuring the extent of validity can become extremely involved and there are different levels at which the question of validity can be considered. Put simply, if your research is valid, then you are observing, identifying or measuring what you say you are. A rough guide would be to ask the question: ‘would another researcher using my research instrument get the same responses?’ Additionally, ask other people whether the questions or items you have devised are likely to achieve what you want. Three different aspects of validity are discussed below:

Plausibility:

How plausible is this claim: that is, do we judge it to be likely to true, given our existing knowledge? Some claims may be so plausible that we can reasonably accept them at face value without needing to know anything else.

Credibility:

Does it seems likely that the researcher's judgement of matters relating to the claim is accurate, given the nature of the phenomena concerned, the circumstances of the research, the characteristics of the researcher, etc.?

Evidence:

Where a claim is neither sufficiently plausible, nor sufficiently credible, then we require evidence. When examining the evidence, we need to employ much the same means to assess its validity as we applied to the claim itself (plausibility and credibility) - and may require even more evidence!

McLeod (1999:101) outlines nine criteria suitable for evaluating the validity of qualitative research, which include:

Sufficient contextualisation of the study: Since qualitative research is more concerned with developing knowledge that is relevant and useful at particular times and places, it is necessary to contextualise the study in its historical, social and cultural location.

Credibility of the researcher (reflexivity):A reflexive account of internal processes of research, covering, for example, how contact was made with informants, issues of trust and rapport, how mistakes and misconceptions were dealt with, whether there was agreement over goals and tasks.

Catalytic validity: Defined as the degree to which the research process reorients, focuses and energies participants. Implicit in this is the idea that research should empower those who take part in it.

McNiff (1992) discusses the issue of validity in relation to action research, and distinguishes three types: 1) self validation, 2) peer validation and 3) learner validation. Action research is an approach that often has particular appeal to researchers working on issues related to professional practice. Because much of what McNiff (1992) discusses is applicable to a careers guidance and counselling context, a summary of some of the points she makes on validity follows.

Self-validation: If you are researching within the broad area of guidance and counselling, you are likely to be involved in interpreting your own practice and in making decisions about improving it. The implication of self-validation in this context refers to the potential of these interpretations of your own practice being recognised. McNiff (1992:133) discusses how certain criteria justify an individual's claim to knowledge. They include:

• Practice as a realisation of values: where guidance and counselling research starts with a declaration (spoken or written) of values – for example, `my clients have a right to self-determination'. Often the research inquiry is stimulated because those values are being denied in practice. If clients are being denied the service you think they deserve, this is where a cycle of imagined solutions, implementation, observation, evaluation, re-planning is enacted.
• Intentional critical reflection: where research inquiry occurs as a result of critical reflection, a desire to explore an intuitive understanding of practice and communicate it to others.
• Disciplined enquiry: where a researcher demonstrates publicly that s/he has followed a system of disciplined enquiry in arriving at a hypothesis.
• Personal interpretation as a basis for dialogue: where individuals recognise the potential in their interpretations of their own practice.

Peer validation: This relates to the notion that a claim to knowledge or expertise derived from practice must be validated externally. It is the process whereby your findings are made available to and scrutinised by others who can agree that these findings are of interest and could be useful to their own practice. This external validation can come from a number of sources, including colleagues, manager or supervisor, the wider guidance and counselling community, other students, etc. Peer validation can be effective in moving your ideas forward. Questions may be asked which provide new insights and ways of thinking about your research, and invariably you will gain more confidence to progress your research from the process of peer validation.

Client validation: It is particularly useful to get the reactions of the clients themselves. This may be presented in short, written statements, recordings (tape or video), reports, etc. For example, data about the use of labour market information by guidance practitioners could be collected during focus group discussions. This could be written up into a report that was circulated to the practitioners who were asked to validate the accuracy of the reports. The findings could then be circulated more widely to the guidance community who are asked to comment on the relevance of these findings to their own experiences. This represents a powerful chain of validation.

Overall, validation of research findings in a guidance and counselling or related context is likely to involve the development of self-knowledge followed by a genuine attempt to share that knowledge with others.

Reliability

Whatever procedure is used for collecting data for research, it should always be examined critically to assess to what extent it is likely to be reliable, as well as valid. Reliability is the extent to which a test or procedure produces similar results under constant conditions on all occasions. Bell (1993:65) suggests two questions to ask yourself when checking items on a questionnaire or interview schedule you may devise to collect data. They are:

• would two interviewers using the schedule or procedure get a similar results?
• would an interviewer obtain a similar picture using the procedures on different occasions?

There are a number of devices for checking reliability in scales and tests. For example:

• test-retest method: administering the same test some time after the first;
• alternate forms method: where equivalent versions of the same items in the test are given and results correlated;
• split-half method: where the items in the test are split into two matched halves and scores then correlated.

These methods are not always feasible or necessary, and there are disadvantages and problems associated with all three. Such mechanisms are not usually necessary unless you are attempting to produce a test or scale. The check for reliability will come at the stage of wording questions and piloting your research instrument(s).

Relevance

A third criterion for judging research in the area of guidance and counselling is relevance. Since the purpose of any research inquiry is to extend knowledge and understanding, it follows that this must be communicated to a wider audience. When we communicate with people, they assume that we are telling them something that is likely to be of significance to them. It follows that what is communicated should be relevant in some way to the chosen audience. Who are the appropriate audiences for your research and what sort of relevance should your research have for them? Audiences for research reports vary. If you are undertaking a research project within your employing organisation, its primary audience is likely to be members of that organisation. Additionally, your research findings should have some relevance to other audiences. These may include other researchers, a particular practitioner audience or even a more general audience. Whatever audience(s) you select as your target(s), you will need to consider two aspects of the relevance of your research:

• importance of the topic: must relate to an issue of importance to the intended audience;
• contribution to existing: must add something to our knowledge of the issue to knowledge which they relate. Research that merely confirms what is already beyond reasonable doubt makes no contribution to the existing knowledge base.

3.4 Access

No researcher can demand access to an institution, an organisation or to materials. People will be doing you a favour if they agree to help, and will need to know exactly what they will be asked to do, how much time they will be expected to give and what use will be made of the information they provide. They will have to be convinced of your integrity and of the value of your research before they decide whether or not to cooperate. Bell (1993:58) provides a useful checklist of points to consider when negotiating access, including:

• Clear official channels by formally requesting permission to carry out your research as soon as you have an agreed project outline.
• Speak to the people who will be asked to co-operate.
• Maintain strict ethical standards at all times.
• Submit the project outline to the principal, senior staff member.
• Decide what you mean by anonymity and confidentiality.
• Decide who will receive a copy of the report and/or see drafts of interview transcripts.
• Inform participants what is to be done with the information they provide.
• Prepare an outline of intentions and conditions under which the study will be carried out to hand to participants.
• Be honest about the purpose of the study and about the conditions of the research.
• Remember that people who agreed to help are doing you a favour.

Even when strict protocol is adhered to, things can go wrong. Blaxter et al identify the following strategies to consider if access is denied (1996:144):

• approach other individuals. If one person refuses to be interviewed or answer a questionnaire, try approaching another person in a similar position or sharing similar characteristics;
• approach another institution;
• approach another individual within the same institution (more risky because of possible communication with the institution);
• try again later, when people are less busy. Attitudes may have changed, people may have moved on, and you may have more to show to demonstrate the value of your research;
• change your research strategy. This is probably something you should be prepared to do, and plan for, throughout the research process. It may involve using other, perhaps less sensitive, methods for collecting data, or focusing on a slightly different set of issues, or studying alternative groups or organisations.

3.5 Managing your research project

Constraints operate on any research process. This section discusses time and costs, as well as identifying some others which may well be relevant for the type of research you are planning.

Time

Inevitably, the nature and extent of your data collection will be constrained by your access to various resources – in particular time. Whatever the competing demands for your time and attention, it is important to think about what strategies you need to develop to manage the new demands of your research project in parallel with all the established demands.

A key strategy for managing your time effectively is to be realistic, initially, about the methods you are going to use to collect data and the amount of data needed. One common problem with new researchers is over-ambition. Since it’s likely that you will have a limited timescale to complete and write up your research, it follows that the methods you select will have to be informed by these (and other) considerations. For example, time available will limit the amount of any cross-checking you can undertake, and the size of your research sample.

Costs

The costs of research can mount up, so it is advisable to undertake a rough costing of the methods of data collection and analysis that you have considered to make sure they are affordable. The process of costing research activities will be very useful if you are, at any stage of your professional career, ever likely to apply for research funding. For applications for external funding, a detailed costing has to be submitted, and if successful, it is likely that you will have to adhere fairly closely to the original costing submitted.

The costs of your research project could include:

• travel costs to your research sites and/or libraries;
• costs of consumables, such as paper, tapes, batteries, etc.;
• equipment purchase or hire costs (e.g. word processor, tape recorder, software);
• book, report and journal purchases;
• photocopying, printing and binding costs;
• postage and email/telephone costs.

Other constraints

In addition to time and costs, there are bound to be other constraints you need to take into account when designing your research: for example, the willingness of people to be interviewed or observed. If you need to observe meetings or training sessions, you will be limited by the schedule of meetings or training events that will take place during the data-collection phase of your project. If you need to research some aspect of guidance and counselling that requires involvement with schools, colleges or universities, you are likely to be constrained by examination timetables and vacations.

To manage your research project successfully, you will need to anticipate routine constraints affecting the research process. This will help you to develop strategies to cope with these limitations and avoid difficult situations arising in the first place.

3.6 Overcoming obstacles

The process of successful research will, inevitably, involve developing problem-solving techniques. Problems that may arise can range from those relating directly to the research process (for example, the response rate is very low), to problems in other areas in your life (for example, changing your job or falling ill). Blaxter et al (1996: 137) suggest the following ways of coping with difficulties:

• remind yourself that the purpose of carrying out research, particularly as a new researcher, may be as much to develop your understanding of the research process and/or the use of particular research methods as to explore substantive issues;
• remember that it may be just as valid to write up your research in terms of, for example, the problems of gaining access to a particular group, or of getting an adequate response from that group once access has been gained;
• as part of writing your research report, reflect on your research strategy, explore what went wrong and why and include recommendations for improvement;
• view research as being about the skills you have learnt and developed on the way. Part of doing research is about appreciating what is involved and where it may be leading you;
• if you have time and resources, you may choose to redirect your research strategy when you become stuck.