AI is embedding itself into clinical research, mostly indirectly at this stage. From patient recruitment to data cleaning, protocol optimisation to predictive analytics, the upside is clear: faster trials; better targeting and reduced cost. But when you step into the literature, a more balanced picture emerges...

AI does introduce a new layer of risk, one that is somewhat less visible, harder to validate, and more difficult to govern.

Here are five of the most evidence-backed risks shaping the conversation today.

1. Bias Isn’t Removed, It’s Amplified

AI systems learn from historical data. And in healthcare, that data is often incomplete, imbalanced, or skewed toward certain populations.

The result is not neutral automation, it is the amplification of existing bias at scale.

In clinical research, this has direct implications:

• Underrepresentation of certain populations in trials

• Skewed eligibility or matching algorithms

• Reduced external validity of study outcomes

Celi et al. (2022) highlight that bias can enter at multiple stages, from data collection to model deployment, making it difficult to detect and even harder to correct once embedded.

https://pmc.ncbi.nlm.nih.gov/articles/PMC9931338/

2. “Black Box” Models Limit Trust and Adoption

There is a growing assumption that explainable AI will solve the transparency problem. The reality is more nuanced. Many high-performing AI models remain inherently opaque, and even explainability techniques can fall short of providing meaningful clinical insight.

In a regulated environment like clinical trials, this creates a fundamental tension:

• If you can’t explain it, can you validate it?

• If you can’t validate it, can you defend it?

Di Martino et al. (2022) show that despite advances in explainability, trust, interpretability, and usability remain major barriers to adoption in healthcare settings.

https://pmc.ncbi.nlm.nih.gov/articles/PMC9607788/

3. Performance in Theory ≠ Performance in Practice

AI models often perform well in controlled environments with set up parameters. But clinical trials often dynamic and frequently change (think amendments etc). They are also operationally complex, multi-site, and highly variable.

This creates a gap between model performance in development and real-world effectiveness.

Ahmed et al. (2023) identify key barriers that directly impact AI deployment:

• Poor integration into clinical workflows

• Lack of user trust

• Inadequate validation in real-world settings

• Organisational resistance

https://pmc.ncbi.nlm.nih.gov/articles/PMC10623210/

4. Regulation Is Catching Up — Slowly

AI is evolving faster than regulatory frameworks.

While guidance is emerging, there is still significant ambiguity around:

• Model validation standards

• Lifecycle management (updates, retraining)

• Documentation expectations

• Accountability for AI-driven decisions

The U.S. Food and Drug Administration has begun addressing this through a risk-based credibility framework for AI in regulatory decision-making. But the key takeaway is clear:

There is no universal standard yet, only direction of travel.

For sponsors and CROs, this creates a strategic risk: building AI capabilities that may not withstand regulatory scrutiny later.

https://www.fda.gov/media/184830/download

5. Data Risk Expands with AI, Not Reduces

AI systems depend on large, interconnected datasets, often across vendors, platforms, and geographies.

This increases the exposure surface for:

• Data breaches

• Re-identification of anonymised patient data

• Unintended data leakage during model training

• Third-party risk through AI vendors

Conduah et al. (2025) highlight that data privacy challenges are intensifying as digital health ecosystems grow more complex, particularly where advanced analytics and AI are involved. In clinical research, where data sensitivity is exceptionally high, this is not a secondary concern. It is foundational.

https://pmc.ncbi.nlm.nih.gov/articles/PMC12138216/

Conclusion

AI is not a future concept in clinical research. It is already here and in many cases, already delivering value. But the evidence is clear:

• It can amplify bias

• It can obscure decision-making

• It can underperform in real-world settings

• It can outpace regulation

• It can increase data risk

None of these are reasons to avoid AI. But they are strong reasons to approach it with discipline, governance, and healthy scepticism.