E. Evaluation
The evaluation is the final part of the paper. In this section of the report you are expected to:
- Discuss your results in comparison to the results of the original study. If they differed from the original study, why do you think that this happened?
- Discuss your findings with reference to the theory outlined in the introduction.
- State and explain the strengths and limitations of your design, sample, and procedure.
- Suggest modifications to address the limitations of your own investigation.
- A final statement of conclusion with regard to your hypothesis.
In this section of the report, it is important that you evaluate the strengths and limitations of your methodology. You should also explain how the limitations may have affected the outcome of your study. The strongest reports will identify possible extraneous variables that may have influenced the study, and will not rely on a simplistic evaluation, such as “the experimental study should have used a larger sample”. Since it is always true that we could have more participants, this is not a very rigorous analysis.
In addition, you should not include examples of “experimenter error.” For example, if for one group you forget to read the directions from the briefing notes, this is not a valid limitation. This is a mistake. If you make a mistake, the expectation is that you redo the experiment. A discussion of such errors will not earn any marks.
Finally, it is not appropriate to discuss the assumed behaviour of the participants. For example, do not assume that the participants did not follow directions. Use your debriefing to gather evidence to support this so that you can make informed comments about the experiment.
Suggestions for relevant modifications for future replications must also be included. These should only address the limitations you have identified in your own investigation.
Sample evaluation
As can be seen by the results stated above, we were able to support the findings of Tversky & Kahnemann (1974) and the prediction that an anchor influences a person’s estimate of a value. They found a much lower difference between the two groups than we did. The difference between the two medians in the original study was 1738; in our study, it was a difference of 9202.50. This could be because our participants were younger and may not have as much conceptual understanding of mathematics. This means that we support the Dual Process Model and the use of system 1 thinking. Since the students didn't have much mathematical understanding, they relied on the anchor to make an estimate.
One of the strengths of our procedure was that we had the participants practice estimation first so that we could rule out that they didn’t understand instructions. In addition, we had them answer the key question on the back of the paper so that they did not see the previous numbers on the other side of the paper. We did this so that those numbers did not influence their estimate. However, a limitation of the procedure is that we probably should have carried out a distractor task - such as reciting the alphabet backward – between tasks. We cannot rule out that the memory of the previous estimates may have influenced their final response.
There were also several variables that could not be controlled. For example, it is not possible to know to what extent the participants were trying to quickly calculate the response and whether they were really estimating. We also do not know if they read the equation from left to right or right to left. During the debriefing, the participants did not show that they knew the goal of the study, but some indicated that they found it uninteresting. This may have resulted in a high variation of the responses.
A limitation of the design was that we used an independent samples design. Our differences may be due to participant variability. We do not know how many of the participants have experience with factorials or with estimating values. It may be more appropriate to first test the students for their ability to estimate the values of equations or their knowledge of factorials and then eliminate them from the sample if necessary. The strength was it allowed us to use the same equation with both groups and they did not guess the hypothesis.
Although the strength of the sample was that it was easily obtained, there were limitations of the sample. It is difficult to generalize our findings as the sample was made up of only grade 9 maths students. Older participants or participants with lower math abilities may not demonstrate the same results.
One modification for future replication would be to create a more realistic context. Rather than simply giving them a math problem, several studies give suggested prices for goods. The participants then have to determine what a fair price would be. If anchoring works, then when the given price should have an effect on the concept of a “fair price.” We would also try to match the math abilities of both groups to make sure that this does not have an effect on the difference in estimates.
From our study, we are able to conclude that in a sample of grade 9 mathematics students, the value of the anchor in a math equation affects the estimate of the answer to the equation.
NB: The entire sample in this chapter is only 1660 words in length. The maximum word count is 2200. Many ideas could and should be added to the evaluation, this is only used as an example of the way to approach the writing of the IA.
Comments on the sample
Paragraph 1: This paragraph compares the findings of the study to the original study. The goal is to link the theory to their own study. Explaining why their results occurred (the lack of mathematical understanding) is essential for earning the top marks here.
Paragraph 2: Strengths of the study are outlined; there is a suggestion for improvement.
Paragraphs 3 & 4: Limitations of the study are explained and then suggestions are made for improvement.
Paragraph 5: A suggestion is made for how to study this topic in a different manner - furthering our understanding of the role of anchors in decision making.
Paragraph 6: The report ends with a statement of the conclusion that reflects the hypothesis.
How you are assessed
The following table is the assessment rubric used to award marks for your evaluation.
Marks | Level descriptor |
0 |
|
1 - 2 |
|
3 - 4 |
|
5 - 6 |
|