In many psychological studies, data is collected by means of interviews, surveys and/or questionnaires. These research methods are heavily dependent on self-reported data. Often this is the only reasonable type of data that can be collected; however, there are several limitations to the use of self-reported data. It is important that when we evaluate research we recognize these limitations.

Remember, self-reported data is often the only way we can gather data - for example, when studying cognitive processing, non-physical symptoms of a disorder or attitudes.  Since self-reported data on its own has many limitations, it is important that psycholgists use data triangulation before drawing conclusions.

As part of any research in psychology, demand characteristics may play an important role in undermining our results. The expectancy effect is when a participant thinks that they know what a researcher is hoping to achieve and then tells the researcher what s/he wants to hear. Another demand characteristic is the social desirability effect, where a participant says what s/he believes will make them look good - or at least not make them look bad! For example, when people are asked how they feel about a social issue, they may simply say what they believe is “socially appropriate” because they do not want to seem to be uneducated, racist, sexist, or old-fashioned.

Health psychologists often have the problem that participants demonstrate optimism bias. In this case, the participants believe that their health behaviours are better than they actually are. This is similar to overconfidence bias in which we assume that our behaviour is better than average - for example, when most people are asked about their driving skills, they assume that they are a better driver than most people.  But, of course, we cannot all be better than everyone else! When participants show optimism bias, they believe that their health behaviours put them at a lower risk than other people for a negative health outcome.  They also tend to overestimate their healthy behaviours.  When asked how often they exercise - or how much alcohol they drink on a weekly basis - their answers may not reflect reality.  It is important to remember that this is the participants' actual perception of their behaviour.  They are not actually "lying" - they actually believe that they are better drivers, frequent exercisers, and responsible drinkers.

Conformity effects are more likely to happen when in a focus group or experimental situation with other participants. Participants may simply conform to what others are saying, even though they will believe that what they are saying is, in fact, their own opinion.

Another problem with self-reported data is that it is often retrospective in nature - that is, people are asked to think back on moments in their lives and give the researcher information. This may include, for example, a description of their past relationships, time in school or childhood trauma. Many of these memories may have been distorted over time because of the reconstructive nature of memory. When it is not possible to verify the accuracy of the information, then the study's validity may be compromised.

In addition, the availability heuristic plays a key role in how we assess the past. This bias means that we form an opinion based on what information readily comes to mind. So, if you are able to more quickly recall the times when a teacher scolded students rather than praised them, then the perception would be that the teacher was rather mean and unfair to students. So, it may make a difference when doing a student feedback survey if it is done after a big exam or if it is done after the class has had a series of fun experiments and field trips.

Finally there is the peak-end rule. According to the peak-end rule, we judge our experiences almost entirely on how they were at their peak and how they ended. This is because it is not possible for us to figure out an "average of our behaviour over time."  Daniel Kahnemann (1993) argues that people seem to perceive not the sum of an experience but its average.

Peak-end rule is often a problem in self-reported data. Suppose a couple goes on a holiday and has a great 12 days in the sun on a spectacular beach. For the last two days of the holiday, not only does in rain incessantly, but their hotel room was robbed while they were at the all-you-can-eat buffet for dinner. When asked about the holiday, it is highly likely that the couple will judge the holiday as a “disaster,” even though the vast majority of the holiday was excellent. In the study of why relationships fall apart, a 20-year marriage may be judged as horrible and unfulfilling, even though it was only the last two years that were difficult for the couple. Comments like, “I don’t think that I ever loved him” may reflect peak-end rule rather than fairly represent the overall marital experience.

So, when evaluating a study and commenting on the use of “self-reported data,” there are different reasons why this type of data may be problematic for a particular study. Always try to clarify why the use of this data is limited in order to show the highest levels of critical thinking.

Checking for understanding

For each of the following scenarios below, drag the answer into the box that best describes the researchers' problem.