Insights - AI Models Agree With Each Other More Than Toxicology Experts Do

A Eurotox 2026 study found LLMs agree more consistently than expert toxicologists do. Here is what that means for legacy IUCLID dossier data.

· Svetlana Galakhova · 4 min read

Toxicology runs on some of the strictest lab standards in science: OECD guidelines, GLP protocols, validated assays. A study presented at Eurotox 2026 found that the inconsistent part isn’t the lab work. It’s the reading.

The case

At the HESI CITE Lecture, Jess Ewald’s team at EMBL-EBI faced a familiar problem: over 100,000 free-text toxicology findings, collected across decades of animal studies from different labs, in different formats, under different conventions. All of it needed to be mapped onto one standardized ontology (the Mammalian Phenotype Ontology) before it could be reused at scale.

Manual mapping at that volume isn’t realistic, so the team built an LLM-assisted pipeline. Building it was expert work in its own right: toxicologists shaped how findings were matched to the ontology and how the output was judged. Then the team tested it against seven expert toxicologists doing the same mapping by hand.

The verdict

Measured in mean pairwise agreement (Jaccard similarity):

ComparisonAgreement
LLM ↔ LLM0.85
Human ↔ LLM0.60
Human ↔ Human0.55

Every difference was significant at p < 0.001. Every individual model landed closer to the average human answer than any single human expert did to their own colleagues.

The models were more consistent than the experts.

Why this matters beyond one conference talk

We spend a lot of energy asking whether AI is reliable enough for regulatory work. This result asks the harder question the other way: how reliable is the expert judgment that existing dossiers are already built on?

That question isn’t abstract for anyone managing a legacy IUCLID instance. Every dossier accumulated over years holds the same kind of material: endpoint summaries and classification rationale, written by different reviewers, in different years, in slightly different words. None of it is technically wrong. Very little of it agrees perfectly with the record next to it. A typical case: one skin sensitization summary concludes “non-sensitizing”, the next one “lacks skin sensitizing potential”. Both mean the same thing, but a search, a filter or an automated export treats them as two different answers.

Nobody flags this as an error, because it isn’t one — it’s drift. And drift stays invisible until someone has to read two files side by side and decide which version is actually correct. During a migration, an audit, or a submission review, that decision has real consequences.

The study also shows where consistency comes from. It didn’t come from a model left to its own devices. It came from experts deciding once, carefully, how a finding should be read, and then building that decision into a process that applies it the same way every time. For a different regulatory question, they would have to make those decisions again.

Legacy dossiers have the opposite problem: the expert judgment is all there, but it was applied again and again, by different people, in slightly different ways.

There is no magic LLM wand. Consistency comes from experts defining the decision context, setting the rules and checking the output. AI then applies that judgment the same way every time. It doesn’t replace it.

What this means for a migration

Structured, well-managed infrastructure is built to close exactly this gap. Before a migration, we work through it in three steps:

  1. Experts set the rules. Your regulatory team decides how findings and conclusions should be expressed for your substances and your regulatory questions. We help by showing what the legacy data actually contains.
  2. The rules are applied across the whole instance. We go through every dossier and flag records that say the same thing in different words, or contradict each other. Your experts decide which version stands.
  3. One authoritative record. Each substance ends up with a single dossier, current picklists and one record that flows outward instead of being re-keyed at every step, so new drift has fewer places to hide.

If you’re evaluating how much inconsistency is sitting in your own legacy dossiers, that’s exactly the kind of review we do before a migration to 4chems.com. Talk to us about what a structured, current IUCLID environment would surface in your data.

Source: Jess Ewald, HESI CITE Lecture, “Connecting chemical effects across biological scales with AI,” Eurotox 2026, Vienna.

Related Posts

Your next step

The tools and solutions behind these insights — managed IUCLID, CHESAR and QSAR Toolbox in the EU cloud.