2023 2 1 What is research The good researcher is not one who knows the right answers | Assignment Collections

• 2.1 What is research? The good researcher is not ‘one who knows the right answers’ but ‘one who is struggling to find out what the right questions might be’. Phillips and Pugh (2005: 48) CHAPTER 2 Research Aims: To introduce research in the context of computing projects. Learning objectives: When you have completed this chapter, you should be able to: n Discuss what research means. n Understand the research process. n Classify research and understand the different research methods available. n Understand the issues surrounding interviews, questionnaires and observational studies. n This chapter is highly relevant for research degrees. n This chapter is highly relevant for research-based taught degree projects and provides useful reading for other taught degree projects. 15 M02_DAWS1314_02_SE_C02.QXD 3/25/09 1:14 PM Page 15 16 Chapter 2 n Research 2.1.1 A definition A useful starting point from which to discuss research is to examine a seminal definition of the term. Research is defined by the Higher Education Funding Council for England (HECFE) as ‘original investigation undertaken in order to gain knowledge and understanding’ (RAE, 2008). Three key terms in this definition have been italicised for emphasis; original, gain and knowledge and understanding. These terms are essential to the definition of research and will be discussed in turn. 2.1.2 Originality There is no point in repeating the work of others and discovering or producing what is already known. Quite simply, originality is doing something that has not been done before. While this remains a relatively simplistic definition, it is important to discuss how originality relates to projects. What can you do that is original? What type of things can you produce that are original? You can be original in two ways. First, you can be original in the way you do things – for example, doing something someone has done before but using a different technique or approach. Second, you can be original by producing or developing something that has not been produced before. In terms of originality in the way you do things, Cryer (2006: 193–195) identifies a number of areas in which your project can be original: n Tools, techniques, procedures and methods. You may apply new tools and techniques to existing problems or try new procedures and methods in contexts where they have not been applied before. Whether these investigations prove successful or not, you will still be doing something that is original and discovering why these approaches are suitable in certain circumstances or why they aren’t. n Exploring the unknown. Although rare, you may investigate a field that no one has thought to investigate before. Recent discoveries in scientific fields may open up new possibilities and unexplored avenues of research to pursue. n Exploring the unanticipated. Although you may investigate a field of research that has been looked at many times before, you may come across unexpected results or exciting new directions as yet unexplored. Investigating these ‘side tracks’ may prove fruitful, but take care that they don’t lead to dead ends. You might also be able to contribute to these fields by further developing original work. Exploring a field that has already been investigated does not necessarily fail to be original. You may be able to improve on something that already exists, provide a new perspective or interpretation, or produce a unique in-depth study of the field. n The use of data. You can interpret data in different ways, use them in new ways or apply them in alternative areas that have not yet been investigated. In terms of your project’s outcomes, Cryer (2006: 196) identifies originality from the perspective of the results themselves and, also, any original by-products of the research. Thus, original outcomes might include a new product, theory, model or method. Where the intended outcomes are not achieved, by-products might still represent originality – for example, an understanding of why a particular experiment failed or why a particular technique did not work in a new area. M02_DAWS1314_02_SE_C02.QXD 3/25/09 1:14 PM Page 16 Figure 2.1 Contributions to knowledge 2.1 What is research? 17 2.1.3 Gain/contribution ‘Gain’ is, perhaps, an unfortunate term in the HEFCE definition because it does not allude to the fact that research should actually lead to a contribution to knowledge. It is all very well performing an exclusive piece of research and learning something new for ourselves, but unless you can disseminate this knowledge to others, the results of your research are somewhat wasted. With this in mind, the following discussion will focus on the term ‘contribute’, which gives the much clearer message that research should add to world knowledge so that it is accessible to all and not just yourself. Figure 2.1 depicts the world’s body of knowledge and how contributions can be made to it. This body of knowledge represents world understanding, theories, concepts, models, the sciences, the arts and so forth. This knowledge is stored in books, journal articles, conference proceedings, documents, reports, the Internet, art, peoples’ minds and more. Your own knowledge, portrayed as the shaded region, is shown as subsumed within this domain. You can obviously learn things that others already know; shown as expansion to your own knowledge ‘cloud’. Likewise, you can make contributions to world knowledge from your research, such as inventions, new theories and so on. These are shown as expansions to the world’s body of knowledge by the dashed lines. Thus, ‘contribution’ refers to a sharing of new ideas, theories and results with the rest of the world and expanding what is already known. 2.1.4 Knowledge and understanding To explain what is meant by knowledge, it will be discussed in terms of a hierarchy consisting of data, information, knowledge and wisdom. Post and Anderson (2006: 5) identify the meaning of these terms as: n Data. Data are the factual elements that describe objects or events. They represent the raw numbers and raw text you gather from your investigations. For example, as M02_DAWS1314_02_SE_C02.QXD 3/25/09 1:14 PM Page 17 18 Chapter 2 n Research part of your research project, you may need to gather rainfall data from various sites around the country. These data, providing daily rainfall totals at 50 sites, are gathered as raw numbers that mean virtually nothing as they stand. n Information. Information represents data that have been processed in order to provide you with some insight into their meaning. In other words, the data have been analysed, summarised and processed into a more understandable and useful format. In this form, information can be passed to other people; for example, in books, articles, recordings, speech and so on (Orna and Stevens 1995: 36). Converting your rainfall data into information may lead to graphs summarising monthly totals, charts presenting seasonal fluctuations and text or tables summarising average daily rainfall at different sites. In these formats the data have some meaning and you now have some insight into what these data represent. n Knowledge. Knowledge is your higher-level understanding of things. While information provides you with an idea of the ‘what’ (i.e., what is happening in the real world), knowledge represents your understanding of the ‘why’. Knowledge is your personal interpretation of what you gain from information as rules, patterns, decisions, models, ideas and so on. According to Orna and Stevens (1995: 35), knowledge represents the ‘results of experience organised and stored inside each individual’s own mind’. While your information about rainfall provided you with an overview of what was happening to weather over a period of time, knowledge represents your understanding of why rainfall might have changed during this period. For example, your knowledge would be your understanding of why rainfall had increased in particular parts of the country since 1900. n Wisdom. Wisdom represents your ability to put your knowledge into practice. It represents your ability to apply your skills and experiences to create new knowledge and adapt to different situations. With reference to the rainfall data example, wisdom would represent your ability to predict likely changes to rainfall and climate in the future or enable you to understand why rain falls at particular levels in entirely different parts of the world. One more category that is worth mentioning here is theory. While data, information, knowledge and wisdom represent a relatively ‘firm’ understanding of what is going on and how things can be applied, theory represents ideas, opinions and suppositions based on your observations of the world. A theory is not necessarily true but, at the moment, it represents the best explanation of what you observe. Although knowledge has been defined from a personal viewpoint, world knowledge can be defined along much the same lines. In this case, world knowledge relates to world understanding, wisdom and interpretation by everybody and everything that is recorded or documented somewhere and somehow. Collecting data and information on their own is termed as ‘intelligence-gathering’ by Phillips and Pugh (2005: 47). These data are used to answer what Phillips and Pugh term the ‘what’ questions – i.e., what is happening in the world, what don’t we know and what can we find out? Research, however, must go beyond merely gathering data and describing what you see. It must make a contribution to knowledge. It looks for ‘explanations, relationships, comparisons, predictions, generalisations and M02_DAWS1314_02_SE_C02.QXD 3/25/09 1:14 PM Page 18 2.1 What is research? 19 theories’. Research thus addresses what Phillips and Pugh term the ‘why’ questions – why do things happen the way they do? Why is the situation the way it is? And so on. While data and information on their own can only answer the ‘what?’, knowledge and wisdom address the ‘why?’. Finally, it is worth discussing how we interpret our understanding of our own knowledge. At the start of the research process, we have no real understanding of how little we know about our subject – we are not conscious of our own incompetence (blissful ignorance). As our research progresses, we move into a (perhaps somewhat demoralising) phase when we become conscious of our incompetence – we become aware of just how little we know about the subject area we are investigating and can become overwhelmed by how vast that area is (demoralised). As our research progresses and we begin to understand the field, we move into the next phase – becoming competent in our subject and aware of this understanding (conscious of our competence – confident). Finally, we move into a phase in which we lose sight of just how much we do know – we become unconscious of our competence (complacent). Figure 2.2 illustrates this development in terms of competence and consciousness. 2.1.5 Summary Now that the three main aspects of research have been looked at in detail, one other definition of research is presented to see if it encapsulates the essence of the term. As an example, take the definition of research by Sharp et al. (2002: 7), which is: ‘seeking through methodical processes to add to one’s own body of knowledge and to that of others, by the discovery of non-trivial facts and insights’. Once again, the important terms within this definition have been italicised, a number of which relate directly to those that have been discussed already. ‘Add’, for example, relates to the discussion of ‘contribution’, and ‘discovery’ appears to imply some form of ‘originality’. ‘Non-trivial facts’ and ‘insights’ relate to ‘knowledge’ and ‘wisdom’, not data or information. One element that this definition contributes, that the HEFCE definition did not, is the idea of a ‘methodical process’. This points out that research is not something done in an Demoralised Conscious Blissful ignorance No Yes Yes No Competent Complacent Confident Figure 2.2 Self-awareness of a research field M02_DAWS1314_02_SE_C02.QXD 3/25/09 1:14 PM Page 19 20 Chapter 2 n Research ad hoc manner but is something that is planned and pursued in a considered way. Thus, the process of performing research, which is discussed in the following section, is methodical. Drawing these points together results in the following succinct definition of research, which encapsulates all the elements discussed so far – consideration, originality, contribution and knowledge: Research is a considered activity, which aims to make an original contribution to knowledge. • 2.2 The research process 2.2.1 Overview One thing that the above definition of research recognised is that research must be a considered activity. In other words, your research activity should not be performed as and when you feel like it, but it should follow a recognised process. Blaxter et al. (2006: 8–9) identify four common views of the research process: sequential, generalised, circulatory and evolutionary: n Sequential. The sequential process is the simplest view of all. In this process a series of activities are performed one after another as a ‘fixed, linear series of stages’. An example of such a process is the systematic process model of Sharp et al. (2002: 17). This process consists of seven unique, sequential steps: 1. Identify the broad area of study. 2. Select a research topic. 3. Decide on an approach. 4. Plan how you will perform the research. 5. Gather data and information. 6. Analyse and interpret these data. 7. Present the results and findings. Although this model appears entirely sequential, Sharp et al. admit that repetition and cycles may take place during this process. However, how and when this repetition takes place is not explicitly identified. Another, perhaps simpler, example of a sequential research process is that defined by Greenfield (1996: 7). Greenfield breaks the research process into four steps: 1. Review the field – i.e., perform a literature survey. 2. Build a theory – based on your understanding and interpretations of the field. 3. Test the theory – does it work? 4. Reflect and integrate – i.e., update your ideas based on your ‘tests’ and contribute your newfound knowledge to others. n Generalised. The generalised research process is identical to the sequential process in that a defined sequence of activities is performed one after the other. However, the generalised model recognises that not all stages are applicable and some steps may require performing in different ways depending on the nature of the research. Thus, the generalised model identifies alternative routes that may be taken at different stages M02_DAWS1314_02_SE_C02.QXD 3/25/09 1:14 PM Page 20 2.2 The research process 21 depending on the nature and outcomes of the research. An example of such a model is that of Kane (1985: 13), which has 11 distinct stages and a number of alternative research methods. n Circulatory. The circulatory approach recognises that any research is really only part of a continuous cycle of discovery and investigation. Quite often, research will uncover more questions than it answers and, hence, the research process can begin again by attempting to answer these newfound questions. Experiences of research might lead you to revisit or reinterpret earlier stages of your work (Blaxter et al., 2006: 9). The circulatory interpretation also permits the research process to be joined at any point and recognises that the process is never-ending. An example of a circulatory process is Rudestam and Newton’s Research Wheel (2007: 5), which suggests a ‘recursive cycle of steps that are repeated over time’. n Evolutionary. The evolutionary concept takes the circulatory interpretation one step further and recognises that research must evolve and change over time, not necessarily following a defined circulatory pattern or repeating the same forms of analysis and interpretation that were performed before. The outcomes of each evolution impact on later ones to a greater or lesser extent. Perhaps one of the more appropriate examples of the research process is that defined by Orna and Stevens (1995: 11). They define a process that is circulatory at the top level and evolutionary within the main search/investigation stage of the process. Figure 2.3 is an adapted interpretation of this model. Figure 2.3 The real research process M02_DAWS1314_02_SE_C02.QXD 3/25/09 1:14 PM Page 21 22 Chapter 2 n Research Figure 2.3 shows a circulatory research process that begins in the top left-hand corner with a definition of your search. Orna and Stevens (loc. cit.) identify this search definition as an attempt to answer the following questions: n ‘What am I looking for?’ n ‘Why am I looking for it?’ n ‘How shall I set about it?’ n ‘Where shall I start looking?’ Following on from this stage you begin your evolutionary investigation of the chosen research area. This investigation will take place within the current boundaries of world knowledge as you search through, digest and evaluate the material available. This search/investigation is not clear-cut and will evolve over time. It will take time for your ideas to mature, you may find yourself pursuing dead ends and you might create more questions than answers. Eventually your diligence will hopefully pay off and you will discover something of value. This discovery must then be disseminated to others through your reports, conference and internal presentations, journal articles and discussions. There is no point in keeping discoveries to yourself as to do so ignores a fundamental purpose of the research process – that of disseminating your new-found ideas and results to others. Through this communication you are able to make a contribution to world knowledge and understanding – shown by the shaded area in Figure 2.3. However, although you may have discovered something of value and contributed it to world knowledge, the research process might be only just beginning. These discoveries might lead to new questions, new avenues of research and so on. Thus, the research cycle is entered once again as you redefine your search and continue your voyage of discovery. 2.2.2 Intellectual discovery While the research process can be represented by a model of one kind or another, your own reasoning processes and intellectual discoveries are often much more complex and personal. When you are looking for questions to answer and answers to those questions, you will often follow a complex process of inductive and deductive reasoning. n Inductive reasoning. You start with your observations of the world and come to general conclusions about it. In other words, you build models and theories based on your interpretation of the world. Clearly, this interpretation will depend on the data and information you can draw from the world, the subject/problem you are studying and, importantly, what you already know and believe. The knowledge that you can obtain from what you are studying is referred to as epistemology (Cornford and Smithson, 2006: 61). You can either draw general conclusions from what you observe and from what you are studying and apply them to other things (positivism), or you can only induce knowledge unique to yourself and the particular situation under study (anti-positivism). n Deductive reasoning. You start with your knowledge and understanding of the world and predict likely observations within it, even though you might not have encountered them before. M02_DAWS1314_02_SE_C02.QXD 3/25/09 1:14 PM Page 22 2.3 Classifying research 23 Deductive reasoning is affected by your theory of reality, your own personal understanding of the world and your underlying assumptions about what you are investigating. This is referred to as ontology. Different people might deduce different things as their understanding differs from your own and they see things in different ways. To solve complex problems you might need to follow a complex chain of inductive and deductive reasoning. As discussed earlier, knowledge is what you derive from inductive reasoning. In other words, you build your ideas, models, theories and understanding based on your inductive reasoning about the world. Wisdom, on the other hand, is evident from your abilities of deductive reasoning – applying what you know to other situations and problems you haven’t yet encountered. There is more to intellectual discovery than inductive and deductive reasoning alone. If you are having difficulty solving a problem, two interesting methods of intellectual discovery listed by Greenfield (1996: 5) that might help are: ‘The method of Pappus: assume the problem is solved and calculate backwards’ ‘The method of Terullus: assume a solution is impossible and try to prove why’ In addition, Greenfield also suggests trying techniques such as: n Random guesses. This is a similar technique to brainstorming whereby you try to solve a problem by generating a number of potential solutions at random. Hopefully one of them will make sense and work. n Analogy. Is the problem similar to anything else that already has a solution or explanation? n Inversion. Try to look at things from the opposite angle. For example, instead of asking ‘which computer language should I use?’ ask ‘why shouldn’t I use Java?’. n Partition. Break the problem or situation down into smaller, more manageable and understandable parts. It is also worth considering where you are heading with your research before you spend several months pursuing it. For example, quite often research students will get an idea for their investigation and pursue it enthusiastically. However, when they finally obtain the ‘answer’ they realise that it was of little value in the first place. Try to think of where you are going, assume you have obtained the answer already, and ask yourself ‘so what use is this to me?’ • 2.3 Classifying research 2.3.1 Introduction Research can be classified from three different perspectives; its field, its approach and its nature. These three categories are adapted from the four categories discussed by Sharp et al. (2002:13) and Herbert (1990:1). These authors identify an additional category called purpose. However, as the purpose of research is arguably to contribute M02_DAWS1314_02_SE_C02.QXD 3/25/09 1:14 PM Page 23 24 Chapter 2 n Research to knowledge, the way that research achieves this contribution has been identified here subsumed within its nature. n Field. The field of research is ‘little more than a labelling device which enables groups of researchers with similar interests to be identified’ (Sharp et al., loc. cit.). For example, in the topic of computing you might identify research fields in areas such as information systems, artificial intelligence, software engineering and so on. These topics may be further sub-divided into more specific topics to aid the more specialist researcher or expert distinguish aspects of the field. n Approach. Approach represents the research methods employed as part of the research process – for example, case study, experiment and survey. These methods are discussed in more detail in the following section. n Nature. The type of contribution that research makes to knowledge depends upon its nature. Sharp et al. (2002: 15–16) identify three categories that can be used to classify the nature of research: Category 1. Pure theoretical development. Category 2. Research that reviews and assesses pure theory and evaluates its potential for practical application. Category 3. Applied research that has some practical application or outcome. The nature of research can also be identified according to the following common classifications, which are adapted from Sharp et al. (loc. cit.), Herbert (loc. cit.) and Saunders et al. (2007: 133–135): n Pure theory – developing theories to explain things without necessarily linking them to practice. This can be based on your own inductive reasoning which leads you to make conclusions and theories about the world as you see it. n Descriptive studies – reviewing and evaluating existing theory and knowledge in a field or describing particular situations or events. This might include testing existing theories, describing the state of the art, or looking for limits in previous generalisations. n Exploratory studies – exploring a situation or a problem. These studies are useful for finding out ‘what is happening; to seek new insights; to ask questions and to assess phenomena in a new light’ (Robson, 2002: 59 cited by Saunders et al., 2007: 133). Exploratory studies can be performed through literature searches, open questionnaires and interviews. These studies can start out by exploring particularly broad areas, concepts and ideas, before focussing in and narrowing down to specifics as the research progresses. The process is thus an iterative and flexible one that seeks new information and ideas. n Explanatory studies – explaining or clarifying something or some phenomena and identifying the relationships between things. n Causal studies – assessing the effects that one or more variables have on another. The independent variables are those which might be having an influence on the dependent variable in which you are interested. In these studies you would manipulate the independent variables and monitor changes to the dependent variable. For example, does the size of software product (independent variable) affect how difficult it is to maintain (dependent variable which is measured in some way)? M02_DAWS1314_02_SE_C02.QXD 3/25/09 1:14 PM Page 24 2.4 Research methods 25 In these studies it is important to ensure that extraneous factors do not influence your results. For example, software size appears to be influencing maintainability but, in fact, maintainability might be due to a range of other factors you were unaware of and did not control. n Resolving a problem with a novel solution and/or improving something in one way or another. n Developing or constructing something novel. 2.3.2 What is good research? You should now have an idea of what research is about and how to classify it, but what is meant by good research? Phillips and Pugh (2005: 48–49) identify three characteristics of good research: n Open minds. You should work with an ‘open system of thought’. Be open minded to the questions posed. ‘Conventional wisdom and accepted doctrine . . . may turn out to be inadequate’. n Critical analysis. Examine data critically. Are these figures correct? Have they been affected in some way? What do these data really mean? Are alternative data available? Can these data be interpreted differently? n Generalisations. Researchers generalise and specify limits on the generalisations they identify. Generalisation allows research to be interpreted and applied to a wide variety of situations. However, researchers must know the limitations of these generalisations. Generalisations stem from your own wisdom and evolve from your deductive reasoning which leads you to develop ideas about things you have not encountered before with certain caveats. Failure to apply these characteristics perpetuates the status quo – everything remains unchallenged and stays the same. Without an open mind to things, without a critical eye and without an ability to generalise your understanding to different things, you will not make a contribution to knowledge. This is, after all, the main aim of your research. • 2.4 Research methods 2.4.1 Overview Berndtsson et al. (2008: 13) identify two main classes of research methods – quantitative and qualitative. Quantitative research methods are associated with measuring things on numeric scales. According to Berndtsson et al., these methods stem from the natural sciences and are concerned with understanding ‘how something is constructed/built/works’. In the natural sciences one is usually concerned with testing hypotheses and Berndtsson et al. point out that ‘repeatability of the experiments and the testing of hypotheses are vital to the reliability of the results’. Qualitative methods, on the other hand, have their origins in the social sciences. Berndtsson et al. state that these methods are ‘primarily concerned with increasing understanding of a substantive area, rather than producing an explanation for it’. Qualitative M02_DAWS1314_02_SE_C02.QXD 3/25/09 1:14 PM Page 25 26 Chapter 2 n Research methods are more common within the field of information science and involve methods such as case studies and surveys. These methods, along with a number of others, are discussed in the following section. 2.4.2 Research methods Four of the most common research methods that you might use (either individually or combined) are action research, experiment, case study and survey: Action research Involves ‘the carefully documented (and monitored) study of an attempt by you. . . to actively solve a problem and/or change a situation’ (Herbert, 1990: 29). Sometimes referred to as participant observation, it involves working on a specific problem or project with a subject or, more usually, an organisation and evaluating the results. With action research you must ensure that you do not become too obsessed with completing the action itself and neglect the real reason for doing it – i.e., evaluating it as part of your academic project. Experiment Involves an investigation of causal relationships using tests controlled by yourself. Quite often quasi-experimental research will have to be performed due to problems of insufficient access to samples, ethical issues and so on. According to Saunders et al. (2007: 137), experiments typically involve: n defining a theoretical hypothesis; n selecting samples from known populations; n allocating samples to different experimental conditions; n introducing planned changes to one or more variables; n measuring a small number of variables; n controlling all other variables. Experiments are usually performed in development, evaluation and problem-solving projects. Case study A case study is ‘an in-depth exploration of one situation’ (Cornford and Smithson, 2006: 71). It involves the investigation of a particular situation, problem, company or group of companies. This investigation can be performed directly, for example, by interviews, observation, etc.; or indirectly by studying company reports or company documentation. Berndtsson et al. (2008: 62) point out that you should not merely report on the outcome of the case study investigation, but also attempt to ‘generalise from the specific details of the examined setting, attempting to characterise the situation for which the studied organisation is typical’. Case studies usually generate a large amount of subjective data – data that you must sift, analyse and interpret in order to produce meaningful, accurate and fair conclusions. You should also be aware of your own influence on the case study if it is performed directly. For example, when you interviewed staff within a local company, did they M02_DAWS1314_02_SE_C02.QXD 3/25/09 1:14 PM Page 26 2.4 Research methods 27 tell you what they felt you wanted to hear rather than the facts of the situation? Is there any means of ‘triangulating’ your data – i.e., obtaining the data from two or three different sources to confirm the truth in what you are told (and thus eliminating the possible influence you might be having on the data capture)? For more information on case study research refer to texts such as Gerring (2006) and Yin (2008), which are entire books, devoted to this issue. Survey This is usually undertaken through the use of questionnaires or interviews. It allows ‘the collection of a large amount of data from a sizable population in a highly economical way’ (Saunders et al., 2007: 138). As part of a survey you might have to identify samples and sample sizes, design questionnaires and define interviews as appropriate. While questionnaires and interviewing are covered in the following sections, Czaja and Blair (2005) and Groves et al. (2004) are two texts that cover the survey research method in detail. Research methods can also be classified according to their ‘time frame’. In other words, does the study result in a snap shot of what you have observed or do your data provide an insight into events over a period of time? A snap shot of a situation or event is referred to as cross-sectional study. A long-term picture, on the other hand, in which data are gathered continually over a period of time, is called a longitudinal study. Which kind of study you use will depend on the nature of your research and what you hope to achieve. For more information on these kinds of study refer to texts such as Saunders et al. (2007: 135–145) and Cornford and Smithson (2006: 70–71). Three research techniques that crop up again and again in both case study research and surveys are interviews, questionnaires and observation. While the detail of these techniques is covered in many other texts, it is worthwhile including a brief overview of these techniques here. 2.4.3 Interviews Interviews can be performed during a number of stages of your project. For example, from the preliminary requirements gathering stages of a software development-type project, to the evaluation of a product you have developed. Interviews are undertaken in either a structured or unstructured way. Structured interviews involve a series of preset questions that you go through with the interviewee. Unstructured (or open) interviews, on the other hand, are less formal and you will ask open-ended questions that allow the interviewee to give extended answers without constraint. In unstructured interviews, you may find that the interviewee is moving away from the agenda you have in mind and you will need to draw them back towards the purpose of the interview. Be aware that unstructured interviews may provide some very interesting data but you may well have to ‘dig deeply’ through the responses you receive to get to the useful information you require. You need to put a lot of thought and preparation into your interviews. The interviewee is probably giving up valuable time to help you out so be well prepared and professional M02_DAWS1314_02_SE_C02.QXD 3/25/09 1:14 PM Page 27 28 Chapter 2 n Research in your approach. Before you conduct an interview be clear in your own mind what you hope to achieve from it. If you lack confidence, or are new to interviewing, then a structured interview with specific questions is probably an appropriate way to start. For more experienced interviewers, unstructured interviews can provide lots of useful data. Berndtsson et al. (2008: 61–62) discuss the following points for preparing and conducting interviews: n Selecting interviewees. If you are performing a case study within a company you might not be able to interview the people you really need to. Be aware that people are usually very busy and might not always be available when you need them. Try to be flexible in your approach and consider alternative interviewees who might provide you with the same information (for example, two or three people might cover the area you are primarily interested in). n Preparation. Don’t just expect to interview somebody ‘off the top of your head’. Have some opening questions to ‘get the ball rolling’ in an unstructured interview or carefully prepare a set of questions for a structured interview. n Questionnaires. You could prepare a questionnaire and use this as a basis for your interviews. This would ensure consistency when interviewing a number of people, it would provide a useful basis for discussion and it would focus your preparation – forcing you to prepare an appropriate set of questions beforehand. Section 2.4.4 provides more detail on questionnaire design. n Have an agenda for the interview. You will have in your mind an idea of what you hope to achieve from the interview so keep this in mind as the interview progresses and pull the interviewee back on track if they start to move away from the intended topic. n Structured replies. In some cases you might wish to quantify an answer from the interviewee (for example, On a scale of 1 to 5, what do you feel about the company’s approach to staff development?). This will enable you to perform more objective analyses of the responses – perhaps through some statistical analyses of the results. n Note taking. Consider how you will make notes during the interview. If you have some pre-defined, structured questions, can you simply note the response on your questionnaire? Would it be appropriate to record the interview? Sometimes recording an interview can stifle the openness of the interviewee. You will need to get their permission to do this. n Confidentiality. Not only must you guarantee the interviewee’s confidentiality if appropriate, but you may also need to consider whether the information they are providing is confidential to their company too. In this case you may need to arrange some sort of confidentiality clause or disclosure agreement with the company beforehand. In these cases you will need to consider what happens to the transcript of the interview (whether written or recorded) and also what you will be able to reproduce in your report. You supervisor should be able to help you with this issue. n Logistics of the session. You will need to arrange the interview; the time, location and duration. Be aware that this can take time to organise as people are very busy so leave plenty of time in your project to arrange your interviews. n Characterisation. You might be interviewing several people during the course of your study and they may have different backgrounds (technical, personal, etc.). You should consider how their background might influence their responses and take this M02_DAWS1314_02_SE_C02.QXD 3/25/09 1:14 PM Page 28 2.4 Research methods 29 into account when reporting your findings. You should also consider how you might preserve an interviewee’s anonymity if it might be obvious who gave you a particular response based on their background. n Ethical issues. Sometimes you may need to arrange ethical clearance for interviews (for example, if you were interviewing children). Most universities will have guidance on this area and you should consult your supervisor for advice. Berndtsson et al. (ibid) go on to point out that a fundamental issue surrounding the interview process is that of trust. Interviewees need to be able to trust you and you should not betray the trust they have in you. Without trust your interviewee may not provide you with the depth or breadth of detail you hope to achieve. 2.4.4 Questionnaires Questionnaires are useful means of obtaining data from a large (possible cross section) number of subjects. Like interviews they can also be undertaken at different stages of your project and can also be used for case studies. Before you undertake a questionnaire, however, you should consider what you hope to achieve from it and whether it is an appropriate technique to use. Be aware that questionnaires are notoriously volatile in terms of replies – most of which achieve only around a 5% response rate. There are a number of texts devoted to the issues surrounding the development and use of questionnaires (for example, Brace, 2008; Bradburn et al., 2004). We cannot hope to cover all of these issues within this text but the following are guidelines on points you should consider when using questionnaires within your own project. Remember to consult with your supervisor for their advice on this issue. The target audience Who are you intending to send your questionnaire to? How will you target them? Medium What format will your questionnaire take? Will you send out postal questionnaires? Will you send the questionnaire through email (embedded within the email or as an attachment)? Will you establish the questionnaire as a page (or series of pages) on a web site? Some websites that you can use to set up questionnaires for you include: n Survey Monkey: http://www.surveymonkey.com/ n UCCASS: http://www.bigredspark.com/survey.html n Thesis Tools: http://www.thesistools.com/ The advantages of sending out questionnaires via email can be summarised as follows (adapted from Loughborough University, 2008): n Quick and free to send out; n Large populations can be targeted easily; n Respondents can reply relatively easily by returning an email. M02_DAWS1314_02_SE_C02.QXD 3/25/09 1:14 PM Page 29 30 Chapter 2 n Research Unfortunately, email questionnaires do have some disadvantages too (adapted from Loughborough University, 2008): n They may be filtered as spam and easily deleted. n If the questionnaire is sent as an attachment it may not be readable or take too long to download. n If the questionnaire is embedded in the email it may not appear clearly as formats might change from one system to another. n It can be difficult for respondents to reply anonymously which may put them off. Response rate How will you persuade your subjects to complete and return the questionnaire? You could offer a prize to a randomly selected respondent to encourage people to complete the questionnaire. You could contact your subjects beforehand (by telephone for example) to get their agreement to respond. This may encourage more people to complete your questionnaire but this is difficult if the target audience is large. Another way of improving your response rate is to offer to send any respondents a summary of the results. For example, if you are evaluating different companies’ approaches to software development, they all may be interested in a summary of your findings. Including a covering letter is also a good way of introducing yourself, the purpose of the questionnaire and any benefits the respondent might gain if they complete it. You can also include a reply-paid envelope with the questionnaire – this often persuades people to reply as they feel they are wasting your time and money if they do not. Layout and size For paper-based questionnaires the layout is important as it can put people off completing the questionnaire if it appears too long or complex. Try to keep your questionnaire as short and as simple as possible. One way to achieve this is to consider carefully every question you are putting into the questionnaire. Only include those questions that will provide you with something you can use and cut out those questions that will not provide anything useful. In terms of layout, try to keep the questionnaire uncluttered and make sure that the questions are unambiguous. If you have a series of closed, ordinal scale questions (for example, Rate this on a scale of 1 to 10), consider how you might lay these out so that the respondent can easily move from one question to the next. Question types Consider the type of questions you will include. Open questions require extended answers from the respondent (for example, What do you think about the user interface?). Closed questions provide specific answers to questions that the respondent can select from (for example, Do you feel the interface was poor, satisfactory or good?). Respondents are usually quite happy to tick some boxes within a series of closed questions but are more reluctant to complete open questions. Try to mix your use of open and closed questions so the respondent does not become bored ticking boxes or put off by a number of long, open questions. In addition, try to keep the style of questions consistent. For example, don’t swap from asking respondents to tick boxes in one question to underlining or circling M02_DAWS1314_02_SE_C02.QXD 3/25/09 1:14 PM Page 30 2.4 Research methods 31 Open questions Closed questions ‘Elicit “rich” qualitative data’. ‘Elicit quantitative data’ – data that can be easy to categorise, measure and quantify. Can discourage people from answering, as they might be reluctant and unmotivated to provide long answers. Are easy to respond to and can be answered quickly. Difficult to analyse as wordy responses can be misinterpreted. Easier to analyse and quantify. Allow freedom of expression. ‘Can encourage “mindless” replies.’ Can stifle responses and not provide options that the respondent wants to give. Table 2.1 Comparison of open and closed questions answers in another. The advantages and disadvantages of open and closed questions are shown in Table 2.1 (adapted from Loughborough University, 2008). For closed questions, consider the type of scale you will use. For example, do you want the respondent to have a ‘don’t care’ or ‘average’ option? For example, ‘rate the software system on a scale of 1 to 5 where 1 is poor, 3 is average, 5 is excellent’. If you made this a 1 to 4 scale there would be no ‘average’ and the respondent would be forced to select either above average (3 or 4) or below average (1 or 2). While questions can, on one hand, be classified as open or closed, another way of viewing question types is whether they are recall or thought questions. Recall questions simply require the respondent to remember a fact, some information, perhaps explaining the way something was tackled, programmed, etc; thought questions require some consideration and perhaps involve opinions, points of view and political interpretations. While respondents might answer recall questions quickly and easily, thought questions may take longer to answer as the respondent weighs up their ideas and interpretations. It might be a good idea to mix recall and thought questions in a questionnaire as the respondent could soon get bogged down with several thought questions following one another. Blaxter et al. (2006: 181) identify seven types of questions that you might use in a questionnaire. They are: 1. Quantity or information. Used to gather specific data that can be quantified or classified (as opposed to qualitative, open-ended questions). When did you join the company? _________________ Although this could cause confusion – are we expecting a date (and if so, in what format?) or a time frame (for example, five years ago)? It might be useful to provide an example to show the respondent what we are expecting (for example, October 2005). 2. Classification. Used to categorise responses into two or more groups. The categories have no ordering to them – i.e., one category is no ‘better’ than the next. Are you male or female (please tick)? Male Female M02_DAWS1314_02_SE_C02.QXD 3/25/09 1:14 PM Page 31 32 Chapter 2 n Research When asking people what age group they fall into it is common to group the ages across decadal boundaries. For example, 18–24, 25–34, 35–44, etc. rather than 20–29, 30–39, 40–49, etc. People sometimes feel bad about being classed in an ‘upper’ age group when they just fall into that group. For example, 30-year-olds do not like to think of themselves as being their thirties and being grouped with 30- to 39-year-olds. By the time they reach the middle of the decadal range they are resigned to the fact that they are in that group (for example, 35-year-olds know they are in their thirties) and are happy to tick that box. Also make sure that categories are mutually exclusive to avoid confusion. For example, age ranges of 20–25, 25–35, 35–45, etc. would be confusing for those aged 25, 35, 45, etc. 3. List or multiple choice. Provides the respondent with a number of options to choose from (they can select more than one). What sources of literature have you used for you literature review (please tick all that apply)? Books Journal articles Conference proceedings Newspapers Case studies Company reports Company documentation Sales literature The Internet Internal reports Past projects Other (please specify) _____________________ 4. Scale. Used to rate the respondent’s feelings towards something. How easy do think the software is to install? Very easy Easy Average Slightly difficult Difficult 5. Ranking. Used to order a series of options. You should not provide too many options for the respondent to rank (ten is probably an upper limit) as this can make it difficult to complete accurately and honestly. What did you spend the most time on during your software development? Please rank the following in order from 1 (spent most time on) to 7 (spent least time on). Requirements capture Design Coding Testing Documenting software Debugging Other (please specify) _________________ 6. Complex grid or table. Used to gather similar responses on a range of questions. Please complete the following table (enter ticks where appropriate) with your views on different aspects of the completed software system. M02_DAWS1314_02_SE_C02.QXD 3/25/09 1:14 PM Page 32 2.4 Research methods 33 Very poor Below average Average Above average Excellent Interface Installability Functionality Ease of use Robustness Accuracy 7. Open-ended. Used to obtain extended, qualitative answers. Do you have any further comments you would like to make? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ Question order The order you place your questions in a questionnaire can influence respondents and affect your response rate. Loughborough University (2008) suggests the following principles to follow when deciding on your question order: n Place the important questions in the first half of the questionnaire. Respondents do not always complete questionnaires so if there are some important questions you would like answered, make sure that these come early on. n Don’t put awkward, difficult or embarrassing questions at the start of the questionnaire. This may put people off completing the questionnaire. This implies that you should put easy, non-threatening questions at the start to draw the respondent in. n Go from general to specific questions. n Go from factual to abstract questions. n Go from closed to open questions. n Leave questions regarding personal information until the end. Anonymity Do you wish the respondent to remain anonymous or do you wish (need) to know who they are? If they are anonymous you may not be able to contact them for clarification or to provide them with a prize for completing the questionnaire. If they are asked to supply their name they may be reluctant to complete the questionnaire. You should always assure respondents of confidentiality and explain how the data you gather will be used. M02_DAWS1314_02_SE_C02.QXD 3/25/09 1:14 PM Page 33 34 Chapter 2 n Research Respondent details In addition to their contact details would it be appropriate to ask the respondent information about themselves – for example, their age, professional standing, qualifications, position within the company, etc.? Gather these data only if you are going to do something with them. For example, you may need to know how different levels or experience of employee within a company react to a software system, so knowing their job title and/or when they started working for the company is important. Draft and redraft Don’t expect to get your questionnaire right at the first attempt. You might want to develop a questionnaire, check it with your supervisor, and send it out to a limited number of respondents to see what feedback you receive. You might then redraft the questionnaire in light of these responses – rephrase questions, restructure the layout, and so on. Flay et al. (1983) show how a question might evolve over a period of time into something that provides a response that can be analysed (in this case it took eight attempts to reach the final version of the question): First attempt: ‘How much time have you been trained here?’ Eighth attempt: ‘I think the amount of training given here is (1) too much; (2) quite a lot; (3) just right; (4) not much; (5) too little.’ In summary, there are a lot of issues surrounding questionnaire design and deployment. Even with a considerable effort to produce a ‘perfect’ questionnaire, response rates are usually poor. Be prepared for this and have a contingency plan in place to deal with this when it happens. To conclude this section a few tips on questionnaire design are presented (adapted from Loughborough University, 2008): n ‘Avoid leading questions.’ Avoid questions that can lead the respondent to a particular answer – for example; “Wouldn’t you agree that…?” or “Is it fair to say that…?” n ‘Be specific.’ Try to avoid words that could be interpreted by people in different ways such as often, locally, etc. n ‘Avoid jargon.’ The respondent may not understand what you mean or have a different interpretation to you. n ‘Avoid double-barrelled questions.’ Avoid asking questions that could be difficult to answer, interpreted in different ways or try to elicit two or more answers at a time. For example, “Did you like the system’s interface and its functionality?”. n ‘Avoid double negatives.’ For example, “Do you agree that the system was not designed badly?” would be better phrased as “Do you agree that the system was designed well?”. 2.4.5 Observation Another research technique often used for case studies is observation. According to Blaxter et al. (2006: 178) observation ‘involves the researcher in watching, recording and analysing events of interest’. It can, for example, involve you entering an M02_DAWS1314_02_SE_C02.QXD 3/25/09 1:14 PM Page 34 2.5 Summary 35 organisation and observing the workings of that organisation within the area of interest. There are clearly a lot of issues surrounding observational research. The more important points that you should consider include (adapted from Blaxter et al., 2006: 178): n Arrangements. How will you go about arranging an observation? What permission will you need to obtain? Are there any ethical issues that you will need to address (for example, if you were observing children)? When will the observation take place and how long will you be there for? Is there a good time for undertaking the observation when certain events may be happening? n Recording the observation. How will you record the situation or event that you are studying? You might record your observations in real time – by tape recording, videoing or making notes. Alternatively, you may simply observe the situation and record what you remember of that situation later. n Participation. Will you actually get involved in the situation you are studying or will you act as a ‘disinterested’ observer (Blaxter et al., 2006: 178)? If you are involved in the situation, how involved will you be – will you ‘work shadow’ or actively participate? Will you need any training or health and safety instruction beforehand? If you are actively involved this may be part of a larger action research method that you are undertaking. n The Hawthorne Effect (Roethlisberger and Dickson, 1939). Between 1924 and 1933 a series of experiments were undertaken at the Hawthorne works of the Western Electric Company, Chicago. These experiments involved the manipulation of various working conditions to investigate their effect on worker productivity – for example, changes to lighting levels, rest breaks, etc. In brief, it seemed that no matter what the investigators did, productivity improved – even when changes appeared intuitively to worsen the workers’ conditions (for example, dimming the lights). One of the main conclusions drawn from this study (see Mayo, 1933) was that, despite these detrimental changes, the workers felt valued by being involved in the experiment. Consequently, the workers worked harder because they felt important. The main point to draw from this study is that you should be aware of your influence on the situation you are observing. What you observe will not be under ‘normal’ conditions as anyone you look at will be conscious of your presence and may well act differently compared with how they normally behave. • 2.5 Summary n Research is defined as ‘a considered activity which aims to make an original contribution to knowledge’. n The research process can be sequential, generalised, circulatory or evolutionary. n Research can be classified according to its field, approach and nature. Approaches to research include case studies, experiments, surveys and action research. n Research techniques that are used within the above approaches include interviews, questionnaires and observational studies. M02_DAWS1314_02_SE_C02.QXD 3/25/09 1:14 PM Page 35 36 Chapter 2 n Research • 2.6 Further reading Brace, I. (2008) Questionnaire design: how to plan, structure and write survey material for effective market research (2nd Edition), Kogan Page, London. Bradburn, N.M. Sudman, S. and Wansink, B. (2004) Asking questions: The definitive guide to questionnaire design, Jossey Bass, John Wiley, New York. Czaja, R. and Blair, J. (2005) Designing surveys: A guide to decisions and procedures (2nd Edition), SAGE Publications, London. Gerring, J. (2006) Case study research principles and practice, Cambridge University Press, UK. Groves, R.M. Fowler, F.J. Couper, M.P. Lepkowski, J.M. Singer, E. and Tourangeau, R. (2004) Survey methodology, Wiley Blackwell, Oxford, UK. Yin, R.K. (2008) Case study research design and methods (4th Edition), SAGE Publications, London. • 2.7 Action points n Try to formulate your own definition of research and ask yourself what research means to you. n Decide which of the research methods outlined in Section 2.4 is appropriate for your own project. M02_DAWS1314_02_SE_C02.QXD 3/25/09 1:14 PM Page 36