IB Docs (2) Team
Teacher guide: Qualitative research
Tips for qualitative research
The following page is to guide teachers through the qualitative research methods module. You will find suggestions for the assessment components.
Each activity submitted by the student will appear in your online grade book. It is up to you how you will give feedback and final assessment marks in your school environment.
Goal: To differentiate between qualitative and quantitative research data. In addition, to evaluate sampling techniques used in qualitative research.
Checking for understanding: This is a drag-and-drop quiz with two sections. In the first section, students need to choose whether each is an example of quantitative or qualitative data. In the second section, they should choose the sampling technique that best describes the scenario provided.
The quiz is automatically scored and entered into your grade book upon submission.
Goal: To define naturalistic observations and understand the different ways that they may be done: overt vs covert and participant vs non-participant. In addition, students should be able to evaluate the use of a naturalistic observation.
Short answer question: Students are asked to watch Deb Roy's "Birth of a Word." Then they are asked the following question: Write a paragraph in which you describe the study to a friend. Be sure to identify this as a lab or naturalistic observation; overt or covert; participant or non-participant. Explain at least one strength and one limitation of the study.
You may either give feedback or a mark for this task. I use criterion D on the ERQ rubric:
The response consistently demonstrates well-developed critical thinking.
Evaluation or discussion of relevant areas is consistently well developed.
Responses that are in the top markband earn a "6" or a "7" - in other words, I take the score they would have received for criterion D and add 1 mark to give them a score on this assignment. Students that only state limitations but do not explain them may not earn higher than a 4. Remember, to get into the high markband, the evaluation should be explanatory - so not just, the study has high ecological validity - but why that is true and what it means with regard to the findings.
Potential strengths include: the naturalistic environment in which the study takes place; the study is a longitudinal study that allows the research to observe change over time; the use of technology allowed for an objective collection of the data.
Potential limitations include: only a single participant, so it is difficult to generalize from this study; there is potential researcher bias in the fact that the researcher is studying his own child; there is an ethical question about a researcher studying his own child in such a manner.
Checking for understanding: This is a task where students answer questions about observations in order to practice terminology and demonstrate conceptual understanding. Answers are provided for students to self-assess their own understanding. No grade is entered into your grade book.
Goal: To evaluate three types of interviews: unstructured, semi-structured, and focus groups. In addition, questionnaires are discussed.
Assessing your understanding: Students are given an abstract and asked to answer questions 1a, 1b, and 1c from Paper 3, as well as a modified question 3.
To assess this, I mark questions 1a, b, and c as if it were a Paper 3 - awarding up to three marks apiece. For the final question, I also award 3 marks. 1 mark for a correct bias and up to 2 marks for a correct explanation of why this may be a bias.
In awarding marks, I use the following markbands: 7: 11 - 12; 6: 10; 5: 8 - 9; 4: 7 ; 3: 5 - 6; 2: 3 - 4; 1: 0 - 2.
Sample answers and tips for marking are below.
1a. Identify the method used and outline two characteristics of the method.
The method used is a focus group. One characteristic of a focus group is that it is a group interview, usually consisting of between 5 and 8 participants. A second characteristic is that an interviewer poses questions or topics for discussion, but the conversation is natural between the members of the group.
One mark is awarded for correctly identifying the method and then one marks for each of the correct characteristics.
1b. Describe the sampling method used in the study.
The sampling method is a bit complex. It is a sample of opportunity in that it was taken from a single clinic in Paris. The researchers then asked for recommendations from the clinic, which shows that the sample was not randomly selected. Using this list, they then asked these participants to volunteer to be in the study. Of those that volunteered, the researchers had exclusion criteria which meant that those with other health issues could not take part in the study.
If the method is identified and simply defined with no link to the text, a maximum of one mark should be awarded. Two marks should be awarded if there is a basic answer with a link to the study. Three marks should be awarded if the description shows some understanding of the complexity of the sampling technique.
1c. Suggest an alternative or additional research method giving one reason for your choice.
There are several potential answers here, but the most logical would be either a one-to-one interview (either unstructured or semi-structured) with each participant, to see if they give the same responses when they are not in the group. This is good to control for conformity effects. The other potential response is to use a questionnaire. This can also be anonymized so that demand characteristics are lowered. Questionnaires also provide a standard set of questions that allow for easier analysis than a focus group.
The alternative method has to make sense in the replication of the study with the same aim. Changing the aim of the study results in a score of zero marks. One mark is awarded for identifying an appropriate method; one mark for describing how it would be used; one mark for the explanation as to why it would be used.
2. What would be one potential bias of this sample?
The researchers noted in the original that all of the participants were religious. This is most likely why they chose to volunteer for this study, as the researcher would have had to identify the aim of the study when asking for their participation. The findings may have been different if the participants had not self-identified as being spiritual/religious. Other potential responses could be that there was a cultural bias or gender bias.
One mark is awarded for identifying a bias (e.g. sampling bias) but not linking it to the study. Up to two more marks are awarded for the "unpacking" of the bias.
Goal: To evaluate the use of the case study method and to differentiate between the different methods of triangulation.
Thinking about case studies: This task asks students to watch two videos on hyperthymesia - people with perfect recall of autobiographical memory. Students are then asked to explain why these studies would be considered case studies. The question is self-assessed with an answer provided. No information is entered into your grade book.
Checking for understanding: This a set of 10 multiple choice questions about the three methods required for paper 3: interviews, case studies, and observations. The questions are automatically scored and entered into your grade book. Explanations of the correct responses are provided for the students.
Goal: Revise ethical considerations for all research. Explain ethical considerations in socially sensitive research with regard to the publication and application of research findings.
Applying terminology: Students are given five scenarios of socially sensitive research and asked what ethical considerations there would be in the publication and application of the findings. They are asked to use the terminology of Sieber & Stanley (1988) in their responses. There are sample answers for them to self-assess their understanding. Nothing is entered into your grade book.
Checking for understanding: Students are given an abstract and asked to write a response for both ethics questions from Paper 3. Each response is worth six marks.
In awarding marks, I use the following markbands: 7: 11 - 12; 6: 10; 5: 8 - 9; 4: 7 ; 3: 5 - 6; 2: 3 - 4; 1: 0 - 2.
Sample answers and tips for marking are below.
Describe the ethical considerations that were applied in the study and explain if further ethical considerations could be applied.
The researchers would need to get consent from the women in this study. They should be made aware of the aim of the study and how their data will be used. They should also be reminded of their rights as a participant. For example, the identities of the women will be confidential - that is, when the study is published, no information that would identify the women would be included. This is important because they may face stigma as a result of the findings. They should be reminded that they have the right to withdraw. They should be allowed not only to decline participation in the interview, but they may have their data removed even after completing the interview. There is no deception used in this study. The aim is honestly communicated as the researcher wants to do a valid evaluation of the mental health of the participants. Researchers have to consider potential undue stress to the women, carrying out a telephone interview in which they are asked personal questions about their partner and their own mental health. Perhaps by having the interview over the phone, there will be less stress for the participants. Finally, the researcher should debrief the participants. Based on the findings of this study, in addition to sharing the findings, the researcher should suggest ways in which the women may receive support.
When awarding marks, one mark is awarded for each description/explanation. If the ethical consideration is only identified without reference to the actual study, then no marks are awarded. For example, participants have the right to withdraw.
Describe the ethical considerations in reporting the results and explain additional ethical considerations that could be taken into account when applying the findings of the study.
The identities of the women must be kept confidential. This is important because they could face stigma in their community if their identities were revealed. In addition, it would also reveal the mental health status of their partner, possibly leading to discrimination. It is important that the publication does not imply that this is true of all women married to combat veterans. The study should not result in a general statement about the lives of military couples, potentially stereotyping such couples. As the findings suggest that women married to veterans with PTSD may need psychological support themselves, another ethical consideration is that the researchers advocate for such mental health care for women in this situation. In presenting their findings, it is important that they are clearly stated and that any potential bias or error in their findings is clearly communicated to the reader. It is also important that the researchers use reflexivity, documenting their own potential biases - eg. my father was a combat veteran or I work with veterans with PTSD. This will help the reader to decide on any potential bias in the reporting of the data. Finally, it is transparency is important. Is the study funded by the military? If so, the goal may be to help support spouses of combat veterans. Is the study funding by a drug company that markets anxiolytic drugs (anti-anxiety)? If so, there may be some concerns about the findings.
When awarding marks, award one mark for each relevant point with regard to ethical considerations up to a maximum of six marks. In this example, one mark for each of the following: the need for confidentiality for the women; confidentiality for the partner, avoid stereotyping; advocacy for spouses of combat veterans with PTSD, revealing any personal biases; explaining any potential errors in the study; transparency of funding.
Goal: To differentiate between generalizability and transferability. Discuss the extent to which the findings of a study may be generalized or transferred to another population.
Presentation: The presentation explains the three different types of generalizability applied in qualitative research: representational, inferential, and theoretical (analytical). The presentation ends with three scenarios, asking students to apply the terminology to answer questions. There are answers provided so that students may self-assess. Nothing is added to your grade book.
Short Answer Question: Students are presented with a fictitious study of cheating. The study is based on information in this article, but it is not meant to be an actual study. It is designed to make it possible for students to answer questions about the three types of generalization.
You may choose to only give feedback or to give marks for this task. If you give marks, you may want to give 3 marks for each question, for a total of 9 marks. Marks should be awarded based on how well developed the response is. This would be equal to the marks awarded for question 3 on Paper 3. You may want to award marks as follows: 7: 8 - 9 marks; 6: 6 - 7 marks - and then all other marks are equal to an IB grade - that is, 5 marks is a 5.
Suggested answers to the questions are below.
1. What questions or concerns do you have about the sample in order to determine whether it is representative of the population from which it was drawn? (representational generalization)
The sample was a volunteer sample on a university campus. The participants knew that the study was about why people cheat. This may mean that "cheaters" are over-represented in the sample. In addition, we would want to know the gender breakdown, as well as the diversity of the sample compared to the population of the university. We would also want to know the majors (subjects) that the students are studying. Are they all psychology majors? Mostly maths majors? It would also be important to know the age range of the sample and whether they are living on campus or whether they live with family off campus; whether they are working or not working.
2. What variables would we have to consider with regard to the sample or the environment of the study in order to determine if we could transfer the findings to another population? (inferential generalization)
Inferential generalization would mean to what extent can we apply the findings from this university population to another university population? Variables that would be important include, the size of the university, the competitiveness of the university, the location of the university (the country with regard to cultural dimensions; rural vs urban); socioeconomic status of the student body; diversity of the student body; whether the university has strict penalties for cheating; whether the university integrates academic integrity into the curriculum.
3. To what other contexts do you think that we could generalize the findings? (theoretical generalization)
Cheating is not just about university. The findings that social or community norms may encourage cheating could be applied to paying taxes or traveling without paying for public transportation. It might also be generalized to cheating in relationships - that is, when internal motivation for the relationship decreases (e.g. love) and external motivation increases (financial benefit, keeping family happy), then cheating may increase.
Goal: To differentiate between validity and credibility. Discuss the ways in which the credibility of a study may be ensured.
Presentation: The presentation explains three key strategies for ensuring credibility: triangulation, member checking, and the credibility of the researcher. The presentation ends with a research scenario, asking students guiding questions to apply the terminology and concepts. There are answers provided so that students may self-assess. Nothing is added to your grade book.
Short Answer Question: Students are presented with a fictitious study of community service and prosocial behaviour. The study is based on information in this study, but it is not meant to be an actual study. It is designed in line with IB exam expectations.
You may choose to only give feedback or to give marks for this task. If you give marks, you may want to give 3 marks for each of the three main points about credibility (member checking, the credibility of the researcher and triangulation), for a total of 9 marks. Marks should be awarded based on how well developed the response is. This would be equal to the marks awarded for question 3 on Paper 3. You may want to award marks as follows: 7: 8 - 9 marks; 6: 6 - 7 marks - and then all other marks are equal to an IB grade - that is, 5 marks is a 5.
Suggested approaches to the questions are below.
Students should write this as a paragraph and not just list ideas. The IB requires a "well-developed argument." Making a list will mean that only mid-range marks will be awarded.
Triangulation
The study has used data triangulation in that there are 15 schools represented. This means that there are several different perspectives represented in the sampling of the data. That being said, the sample is restricted to a certain part of the United States. A broader sampling of data may give a more holistic picture of attitudes toward community service. In addition, only the students were interviewed. Interviewing parents and/or teachers may give richer data for analysis and confirm or challenge the perceptions of the students. Only choosing to look at students at the end of their high school career may also have an effect on the findings. Since they are perhaps nostalgic as they end their time in high school, the results may be less credible than if students still in the process of high school were interviewed.
Researcher triangulation should be used - both in the carrying out of the interviews and the interpretation of the data. This limits potential researcher bias. It also confirms that different researchers reached the same conclusions when looking at the same data.
Finally, by only using one-on-one interviews, it is possible that demand characteristics or interviewer effects played a role in the findings. Method triangulation would increase the credibility of the study. Using questionnaires would allow the researchers to see if they had the same responses when the participants were anonymous. In addition, running focus groups would potentially result in different responses as participants listen to the thoughts of their peers. If the same results are obtained using different methods, then we know it was not the choice of face-to-face interviews that led to the results.
Member checking
A key aspect of credibility is that the results reflect the intention and meaning of the participants' data. It is important that the researchers take the interpretation of their data back to the participants (members) and asks them if they feel that their feelings/opinions have been fairly represented. This is also a chance for the participants to correct misperceptions or clarify their responses.
The credibility of the researcher
When evaluating the credibility of this study, one would want to know what the experience of the researcher is. Are they a teacher? A psychologist? And do they have experience working with teenagers? One would want to know that if several researchers would used, was there training for how to carry out the semi-structured interviews?
Goal: To explain the different types of bias in research. Discuss how a researcher may avoid bias in a study.
Presentation: The presentation examines three key sources of bias: selection (sampling) bias, participant bias, and researcher bias. For each type of bias, strategies for avoiding the bias are discussed. Nothing is added to your grade book.
Checking for understanding: A set of multiple-choice questions to check for understanding of key vocabulary relevant to question 3 on Paper 3. The questions are automatically scored and entered into your grade book. Explanations of the correct responses are provided for the students.
There are three final assessments for this unit. One for each of the three qualitative methods, each with a different question 3. You may choose to give one or all three of the final assessments over time. Each assessment is one hour.
If you want students to do this remotely, you can assign this task using the "exam mode" which will limit their time to one hour.
Mock Paper 3 A
This stimulus piece is based on this study. Details have been altered in order to align with IB assessment practice.
Markscheme
1a. Identify the method used and outline two characteristics of the method
One mark is awarded for identifying the method as a naturalistic observation. One mark is then awarded for each correct characteristic. This may include - the children are observed in their natural environment; there is no manipulation of an independent variable; researchers take field notes on behaviours and then carry out a content analysis to look for trends in behaviour; in this case, this was a non-participant naturalistic observation so the researchers did not interact with the children.
1b. Describe the sampling method used in the study.
The sampling method used was an opportunity sample. The researchers chose a local preschool. An opportunity sample was used because it was readily available and was willing to give consent. An opportunity sample also may have specific traits - and in this case, the sample was predominantly white and university educated. One mark is awarded for identifying the sample; one mark is awarded for a simple description of the traits of an opportunity sample, and one mark for a link to the text.
1c. Suggest an alternative or additional research method giving one reason for your choice
An appropriate response would be a lab observation. In a lab observation, the researchers could have some level of control in the study - for example, how long the parents would stay or the number of peers in the room. In a naturalistic observation, there is no control over other variables that could influence the child's behaviour. Another potential response would be a case study. In a case study, the researchers would carry out a longitudinal study that would look at the child's social skills over a period of several years. This would allow the researchers to determine the potential long-term effects of parental behaviour on the child's social and academic success. One mark for an appropriate method and up to two more marks for a well-developed justification.
2. Describe the ethical considerations that were applied in the study and explain if further ethical considerations could be applied.
One mark should be awarded for each ethical consideration that is directly linked to the study. For example, informed consent would have to be obtained from the participants and they would have to be informed about the goal of the study. The six ethical considerations are informed consent, anonymity, right to withdraw, deception, undue stress or harm, debriefing. Simply defining the ethical considerations without linking them explicitly to the study should not earn more than three marks.
3. Discuss the possibility of generalizing the findings of the study.
When discussing the generalizability of the findings of the study, there are two different "levels" of generalizability to consider. Representational generalizability is the extent to which the findings can be applied to the larger population from which the sample is taken - in this case, all families at the pre-school.
In addition, inferential generalizability - also known as transferability - is the extent to which the findings could be applied to other populations. When discussing this, students should make note of variables that may affect the researchers' ability to generalize the findings - e.g. the size of the preschool, the culture of the parents, the socioeconomic status of the parents, or the type of program at the school.
Finally, students may also write about theory generalization - or the ability to generalize the findings to other contexts. Here they may discuss the role of parental attention and risk-taking, for example.
In order to earn in the top markband, responses must directly address the text. A summary of the three types of generalizability without reference to the text cannot earn more than 5 marks. It is recommended that at least two of the three types of generalizability be discussed. Remember that this is a qualitative study so responses that focus on statistical generalizability and discuss factors like sample size will not earn marks.
There are three final assessments for this unit. One for each of the three qualitative methods, each with a different question 3. You may choose to give one or all three of the final assessments over time. Each assessment is one hour.
If you want students to do this remotely, you can assign this task using the "exam mode" which will limit their time to one hour.
Mock Paper 3 A
This stimulus piece is based on this study. Details have been altered in order to align with IB assessment practice.
Markscheme
1a. Identify the method used and outline two characteristics of the method
One mark is awarded for identifying the method as a semi-structured interview. One mark is then awarded for each correct characteristic. This may include - they are given one to one and they use an interview guide with specific questions but the researcher may ask follow-up questions.
1b. Describe the sampling method used in the study.
The sampling method is complex. It is a purposive sample since they were looking for people who had taken part in a Gacaca trial. In addition, the sample was a self-selected sample. Information was sent out to people that took part in the trials based on information provided to the researchers by local leaders. 1 mark for the identification of a purposive or self-selected sample. Up to two more marks for a more in-depth description.
1c. Suggest an alternative or additional research method giving one reason for your choice
Appropriate responses would be a focus group, questionnaire, or an unstructured interview. Quantitative research methods would not be appropriate. One mark for identifying an appropriate method; up to two marks for the justification of the choice - one marks for a generic justification; full marks if the justification is explicitly linked to the study.
2. Describe the ethical considerations that were applied in the study and explain if further ethical considerations could be applied.
One mark should be awarded for each ethical consideration that is directly linked to the study. For example, informed consent would have to be obtained from the participants and they would have to be informed about the goal of the study. The six ethical considerations are informed consent, anonymity, right to withdraw, deception, undue stress or harm, debriefing. Simply defining the ethical considerations without linking them explicitly to the study should not earn more than three marks.
3. Discuss how a researcher could ensure that the results of the study are credible.
There are several potential answers here. These include researcher triangulation, data triangulation (getting people from around the country and not just one village), and method triangulation (potentially a focus group or written questionnaire). The researchers also used back-translation in this study to make sure that the translated questions were understood correctly. The researchers should also take their interpretations back to the participants to see if they feel they reflect their feelings and opinions. This is called member-checking. Finally, we would want to know the experience of the researcher. The fact that the team had a lot of experience working in Rwanda raises the credibility of the study. In order to earn in the top markband, responses must directly address the text. A summary of factors that increase credibility without reference to the text cannot earn more than 5 marks.
There are three final assessments for this unit. One for each of the three qualitative methods, each with a different question 3. You may choose to give one or all three of the final assessments over time. Each assessment is one hour.
If you want students to do this remotely, you can assign this task using the "exam mode" which will limit their time to one hour.
Mock Paper 3 A
The third paper 3 is focused on a case study.
This stimulus piece is based on this study. Details have been altered in order to align with IB assessment practice.
Markscheme
1a. Identify the method used and outline two characteristics of the method
One mark is awarded for identifying the method as a case study. One mark is then awarded for each correct characteristic. This may include - the fact that it was longitudinal - taking place over one sport season; the use of method triangulation - in this case, interviews and observations; the idea that as focused on a specific "case" - in this study, a single team.
1b. Describe the sampling method used in the study.
The sampling method is complex. It is a purposive sample since they were looking for student-athletes who played a team sport. It was also an opportunity sample as they went to a school where there was a strong sports program. In addition, the sample was a self-selected sample. After obtaining consent, the researchers asked for volunteers from the team to take part in the interviews. 1 mark for the identification of a purposive, opportunity, or self-selected sample. Up to two more marks for a more in-depth description.
1c. Suggest an alternative or additional research method giving one reason for your choice
In this case, an "additional" research method should be mentioned. The most appropriate responses would be a questionnaire, unstructured interviews, or a focus group. A questionnaire would allow responses to be anonymous. Since they had great respect for their coach, there may have been social desirability bias. This would help to control for that. In addition, it would have saved the researchers the task of having to transcribe all the interviews. Unstructured interviews would avoid the problem of leading questions. In semi-structured interviews, both the interview agenda and the follow-up questions could result in demand characteristics. Focus groups could lead to more information as ideas/memories are sparked by listening to other members of the team. One mark should be awarded for identifying an appropriate method and up to two marks awarded for the justification of the method.
2. Describe the ethical considerations in reporting the results and explain ethical considerations that could be taken into account when applying the findings of the study.
There are several potential ethical considerations with this study. First, the coach and the team members should be anonymized. This means not only that the names of the coach and players should not be in the report, but that the school should not be identified. Even though there may not be anything negative in the report, the anonymity of the participants must be maintained. It is also important that the researchers attempt to identify any potential bias in their report before publication. In this case, the lead researcher was a former soccer player and coach. There could be a bias in the reporting of the findings. To avoid this and to fairly represent the findings, the researchers should practice reflexivity and should also be transparent in their report with regard to their own relationship to sports. It is important that the findings not be overstated. In this case, comments like "the researchers did not find evidence to show that respect transferred out of the immediate soccer context," should be clarified so that it is not assumed that outside of sports, student-athletes are disrespectful. It is also important to share the rationale of the study and the funding or support for the study. It is possible that this study has the intention of supporting tax increases to continue to support after-school sports programs in the area. If that is the case, then this should be revealed in the report.
One mark should be awarded for every point made - not necessarily every "ethical consideration." A single ethical consideration could make more than one point with regard to the study. A maximum of six marks.
3. Discuss how the researchers in the study could avoid bias.
There are several potential biases in this study; students should consider how to minimize the bias. First, there is selection bias.. The researchers should reflect on the nature of their sample and how this may have influenced the results. The study was made up of a group of highly experienced high school athletes, with an average playing time of over 11 years. This may mean that their findings are biased and cannot be applied to other student-athletes who may still be learning team skills, even at the high school level. These concerns should be noted in the report.
As mentioned above, social desirability bias may play a role in the interviews. The participants obviously liked coach and may have attempted to portray him in a positive light during the interviews. This is why it was important to have observational data to establish the coherence between the student‐athletes’ words and their actions. Data triangulation helps to control for social desirability bias. In addition, having anonymous questionnaires would have also helped to avoid this bias.
Student-athletes are often very aware that their programs need funding. They are also well aware of the negative stereotypes about athletes that often exist in skill. The researchers would have to be careful to avoid leading questions that may lead to the students giving the answers that they are "expected to give" to make their sports program look positive and worth funding. An unstructured interview that asks students to simply discuss what they have learned on the team and how it affects their life outside of sports may have been less open to bias.
Researcher triangulation is also important. In this case, two researchers worked together to collect and analyse the results. Bias could have been further reduced by having other researchers carry out the analysis of the interviews, looking for themes that emerged from the transcripts. Having spent a sports season with these students, the researchers had to be careful of not overly interpreting the transcripts based on their interactions with the team.
Finally, keep a journal of personal reflexivity encourages the researchers to reflect on their own personal biases during the data collection process. Since the lead researcher had also been a coach, it was perhaps a good idea that he was not the one that carried out the interviews - but rather, his assistant. In the report, it would be important for the researchers to discuss how their own personal biases may have affected the interpretation of the findings.
In order to earn in the top markband, responses must directly address the text. A summary of strategies to avoid bias without reference to the text cannot earn more than 5 marks. It is recommended that at least three examples be used to develop a strong response.
Twitter
Facebook
LinkedIn