Estimated Read Time: 4 minutes
Last week, we hosted a discussion with three experts about a big concern for health system leaders: how to properly validate and manage clinical AI.
The discussion was open. Everyone agreed that most health systems are adopting AI quickly, but lack the right infrastructure to do it safely. This is a real problem.
Here are the main takeaways.
Health systems face a tough situation. Clinical AI is being used in many areas, radiology, cardiology, oncology, and admin tasks. But there’s little coordination. Each department uses its own tools while IT tries to connect them. Health systems are very slow to know what’s working or at what cost.
This is more than just an operational hassle. It leads to blind spots. Clinical insights get stuck in silos while performance is tracked on vendor dashboards that few trust. When problems happen it’s hard to know who is responsible.
The webinar made it clear: fragmentation hurts trust and trust is essential in healthcare.
Rifat Atun, Vice Dean and Professor at Harvard T.H. Chan School of Public Health, explained what really matters when validating an AI solution:
"I think the first is the provenance of the data used to develop and assess AI algorithms. Secondly, the AI solution's accuracy. But that doesn't mean that the third dimension—safety—is met. A model can be very accurate in predicting harm all the time. So accuracy does not mean safety. The fourth is algorithmic bias. Fifth is the solution's fidelity post-deployment. And finally, privacy and security."
The assessment of most AI solutions developed for use in health systems today focus almost exclusively on accuracy. They check if the model works in a test environment, sign the contract, and move on. What Atun outlined is different: validation is multidimensional. It includes safety, bias, durability, and the ability to catch ‘drift’ after go-live.
Atun also pointed out that there’s no standard approach yet. Different organizations use different metrics, so it’s hard to compare solutions. This lack of consistency is risky, especially when patient safety is involved.
Darren Skyles, Partner at Nelson Mullins, highlighted the governance issue. He has helped many health systems build AI governance structures and often sees the same problem: committees exist but lack an apparent centralized authority.
"I do think AI councils do have authority, and I think they have authority at least on paper," Skyles said. "What I think inevitably happens is the process begins to move, and it might be driven too much by the IT team. It might be driven too much by compliance and legal, making it too restrictive. It's so important to have that AI governance council be very much a part of the process."
The difference between a governance committee that matters and one that doesn't is simple: does it actually make decisions, or does it just review them after the fact?
Skyles described AI Councils in some health systems that commit to meeting once or twice a month as gatekeepers. They decide what tools staff can access. They set the rules before deployment happens and stay involved through the lifecycle.
In contrast, some organizations let IT departments make decisions before governance is involved which turns the committee into a formality.
"It takes a village," Skyles emphasized. "It takes your C-suite to be involved in that AI governance committee. It takes somebody from your legal team. Somebody from compliance. Somebody from clinical. Everybody is going to have a perspective on these issues, and you're really going to need to have everybody's input so that decisions aren't made that have unintended consequences down the road."
A common theme was that accountability must be shared, but that doesn’t make it simple.
Developers are responsible for training and design, while health systems handle implementation. Neither side fully understands the other’s work. Regulators are still finding their role in the rapidly changing AI landscape. Smaller health systems without strong informatics or legal teams are at a disadvantage compared to large academic centers in establishing robust governance and accountability mechanisms.
This gap between well-resourced and less-resourced health systems is important. Not all can validate, monitor, and manage AI equally, which leads to fairness issues beyond just technology.
Both panelists kept repeating the same practical advice: slow down.
"The most common mistake I see leaders make with AI governance is trying to address everything at once," Skyles said. "You need to understand the problem that you're trying to solve. You're operating and managing a very large system that has a ton of issues day in, day out. You have many potential solutions out there. But really, it is so important to take that step back to think about what your plan is for your overall adoption of AI."
This isn't an argument against AI adoption. It's an argument for intentionality. Start with an inventory. Know where AI is actually running. Understand the clinical problem you're solving. Build a governance structure with real decision-making power before you deploy. Then commit to monitoring through the entire lifecycle.
Atun put it simply when asked what one thing he'd change about how health systems approach AI: "I'd like to be able to compare solution A versus solution B versus solution C against current practice. To be able to do that, I need a uniform set of principles and metrics."
The development toward this is ongoing. On a macro scale, it would take health systems working together, validators using the same standards, and vendors focusing on transparency. At the local level, some developers are promoting platforms that allow real-time comparison amongst models as well as integrative solutions for the multi-AI applications utilized by a health system.
The webinar showed that the main barriers to responsible AI governance are not about technology. They are structural and cultural.
Health systems can set up governance committees, write policies, and run pilot projects. The real challenge is building and monitoring systems and incentives through diligent, concerted efforts that ensure strong validation, clear accountability, and the ability to share evidence across organizations.
That’s the real work ahead, not just creating more algorithms or deploying faster but building better foundations for the AI tools that matter.
In the coming years the most successful health systems won’t be the fastest. They’ll be the ones that validate before scaling, measure what works for their patients, and include governance from the beginning.
This means taking time to pause, think, and act carefully. It also means working with people who will speak up if speed or ROI concerns are put ahead of safety.
We'd like to thank Rifat Atun, Glenn Cohen, and Darren Skyles for a thoughtful and challenging conversation about governance, validation, and accountability in clinical AI.