Examples of Research Limitations

Use of historical control

An internal validity threat in which prior knowledge about the expected course of a condition influences the interpretation of study data.

This validity threat can introduce bias because prior knowledge may lead the research team not to investigate possible confounding factors. It also makes it more likely that the research team will find expected results.

Poor study design

A type of internal validity threat in which the study fails to address how it will answer its research question adequately.

Poor study design may increase the possibility of biases and pose other threats to internal validity. An example is a case-control study; people with one outcome are compared to others who are similar but not the same.

A poorly designed study may fail to match groups for essential variables that could affect the comparative outcome.

Failure of randomization

This happens when participants are not randomly assigned to treatment groups due to either chance or error. The lack of randomization can introduce bias because groups of participants may not be equal or equivalent.

For example, we may notice that one treatment group has more men than the other. This discovery may lead to the suspicion that something about the men (or the women) affects the outcome. The resulting research findings and conclusions are likely to be biased.

Inadequate sample size

An extremely small sample size may have results that are not generalizable to the population. E.g., if a study investigates the opinion of 20 people, the results may not reflect the opinion of the whole population.

In this case, a very small sample size can reduce statistical power and increase the chances of making a type II error.

Type II errors occur when there is no difference between groups, but the null hypothesis was false.

The researcher is unable to identify significant relationships between the sample items inhibiting the research process.

Faulty data collection methods

The lack of validity in the data collection process can lead to biased results. This is because if there’s no consistency in the data collected, variables will be missed.

For example, it won’t be easy to compare results if data is collected using different methods (e.g., via an online survey and face-to-face interviews).

Such limitations can lead to a lack of validity, which going back to our earlier point, can lead to bias.

Inappropriate statistical tests

It is a type of internal validity threat in which the choice of a statistical test is inappropriate for the research design. The use of incorrect statistical tests can lead to inaccurate results and misinterpretation of data. Also, the researcher is unable to acquire critical data on a research problem.

For example, if a researcher conducts a study comparing the effects of two drug treatments for depression. The researcher tests whether one group has higher levels of serotonin than the others.

The appropriate statistical test, in this case, is a parametric one, such as the t-test. However, sometimes it’s not possible to assume that the depression levels in both groups are normally distributed. In such a case, non-parametric tests should be used instead.

Limited scope

A study may be limited in scope if it’s investigating only one aspect of a broader topic. Example: Studying whether people above 40 years old are more likely to say yes to organ donation than people in their 20s.

However, the scope of this study is limited because it only focuses on one factor, age. A more in-depth study will consider other factors, e.g., how do levels of empathy affect the tendency to say yes or no?

The generalizability of this study will also be reduced because the results only apply to people in their 40s and not to other age groups.

Lack of ecological validity

The lack of ecological validity can be an issue if the study limits what participants are asked to do. If the study is reliant on participants completing a questionnaire, its ecological validity will be lower. People respond differently when filling in questionnaires than they do in the real world.

The lack of ecological validity can lead to a lack of generalizability and artificiality. Research should be conducted in real-life conditions to understand the behavior of participants truly.

Data management limitations

Poor data management can lead to loss or distortion of critical data. For example, if the data is not entered correctly or it is lost in storage.

Likewise, any errors made during data analysis can lead to incorrect results. These limitations are extremely common in medical research, where patient data is paramount to improving public health.

Limitations in data management can lead to limitations in generalizability, as well as statistical errors. Poor data management can also result in bias if the accuracy of the data is compromised.

Wrong data analysis methods

When researchers are not aware of the appropriate statistical methods, they can make mistakes in data analysis. This will lead to incorrect results, reduced validity, and a loss of information.

For example, a researcher is studying the relationship between attractiveness and life satisfaction levels. One study found that, on average, more attractive people were happier than less attractive individuals.

However, if the researcher incorrectly analyzes this data, they might find that life satisfaction levels are not correlated with attractiveness. This is because the researcher used incorrect statistical methods, which led to invalid results.

Unreliable measurements

Unreliable measurements can lead to inaccurate results. For example, if data is collected using the wrong units of measurement or incorrect scales. The study would not be replicable because it’s impossible to know what caused the error.

Other examples of unreliable measurements include unreliable questionnaires and biased sampling methods. For example, the researcher only gets responses from people who are easily accessible.

This unreliability leads to a non-representative sample and biased results. Also, sometimes participants may lie during questioning. It will also influence the results and make them unreliable.

Visual limitations

A study can be limited by the type of chart it uses. For example, the study looks at people’s mood swings over a month. The researcher can either use a bar chart or line graph to visualize the results.

The bar chart will only show mood swings’ starting and ending points, but it won’t reveal anything about their length or duration.

The line graph can show all of this information, but its interpretation might be difficult.

If the researcher uses a scatterplot instead, they will see the mood swings in more detail. If they use a combination of charts, they can create an even more comprehensive visual.

Practical limitations

Practical limitations in research are often associated with time and budget. For example, a study might not have enough funding to run all the procedures. Researchers might need to compromise and cut out measures that may not have a lot of impact on the results.

This compromise can reduce the validity of the study and the reliability of its findings. Journals and funding bodies should consider practical limitations when evaluating studies. They should ask themselves if a study with similar results could be conducted in the study’s condition.

Theoretical limitations

Many research questions cannot be answered by empirical data alone. Researchers might need to rely on theoretical frameworks for guidance.

For example, a study is looking at what triggers people’s charitable behavior. The results show that more people are likely to donate more money on sunny days.

The researchers do not have a theory that would explain why this occurs. They know empirical data can provide valid results, but they cannot interpret it without help from a theoretical framework.

External Study Limitations

These are factors beyond the researcher’s control ability that can still affect the study results. External limitations are sometimes unavoidable, despite best efforts. Some of the common limitations of this nature are:

Lack of adherence to the treatment protocol

It is when the participants in a study do not adhere to the treatment protocol. If this occurs, it may affect how we interpret the study outcomes. An example is a study that compares two drugs in two groups of people.

If participants in one group stop taking their medication sooner than the others, the results may be biased.

Attrition bias

Attrition refers to the loss of participants over time, e.g., due to dropout or death. In a research study, attrition bias can occur if the characteristics of those who remain in the study are different from those who leave.

These differences may affect the internal validity of a study because it becomes harder to compare groups in these circumstances.

Unavailability of prior research

If there’s no relevant literature published in a particular study, the researcher will lack relevant and reliable data to use in their research project. This lack of existing literature can have a significant effect on the design of the study.

Unavailability or inadequacy of previous research studies affects the literature review and the study design. For example, a researcher may not have enough materials on a topic to develop a sound hypothesis. The research methodology will also be wanting in this case.

Availability of self-reported data

The availability of self-reported data can limit a study. Researchers are only able to collect subjective measurements from participants. This collection might involve memory recall and self-assessment.

For example, a study looks at why people have different coping strategies when stressed out. People might have a different perception of their coping strategies.

These differences make it difficult to compare results across participants, as they may have used different criteria to assess their coping strategies. A study that uses self-report data can be highly subjective and limited.

Cultural and language limitations

Cultural and language limitations can influence the accuracy of data analysis. For example, A researcher’s background can prevent them from fully understanding the meaning behind the data. In this case, the researcher will be limited in their interpretation of the study analysis results.

To avoid cultural limitations, it may be necessary for a researcher to take a cross-cultural approach to their study. This helps to negate the possible effects of cultural and language boundaries.

Qualitative Research Limitations

A qualitative research study can also experience limitations that hinder a thorough analysis of issues. To avoid these types of limitations, researchers should collect data from a larger sample size.

For some qualitative research studies, it is problematic if the researcher is unable to make inferences. This forces the researcher to rely on subjective interpretations of data.

Examples of qualitative research limitations include:

The researcher collects data using unstructured interviews.

• Small sample size may cause difficulties in generalizing the results.
• Participants may hold different interpretations of the questions asked.
• There may be a lack of trustworthiness and dependability in the data.
• Participants may behave differently than they would in real life.
• Researchers may not be able to make inferences or generate conclusions based on the data.

Qualitative researchers are more likely to offer suggestions for further research rather than definite answers. Study limitations make it difficult to make these suggestions.