Your Hospital Already Knows How to Govern AI
Treat AI like a formulary item, and use a Reverse Business Model Canvas to expose what a product demonstration will never show.
Pharmacy would never put a medication on formulary because the manufacturer gave a polished demonstration. So why would we approve an AI system that way?
AI needs an indication, evidence, restrictions, monitoring, accountability, and a clear reason to remove it. In other words, AI needs a formulary.
The discipline is already in the building
Health systems keep being told they need to invent AI governance. They do not. Pharmacy already runs one of the most disciplined evidence and safety systems in healthcare, and it has been running for decades: formulary management.
Walk through what happens before a drug reaches unrestricted use. The P&T committee reviews a monograph. The drug gets a defined indication. Someone weighs the evidence, names the appropriate prescribers, sets restrictions, defines monitoring parameters, considers cost and alternatives, and documents how the decision can be reversed. Approval is not the end of scrutiny. It is the beginning of surveillance.
ASHP has already connected these dots. Its updated Statement on Artificial Intelligence in Pharmacy says pharmacy professionals should evaluate AI deployment “in the same way they apply scientific rigor to medication formulary decision-making.” The original 2020 statement drew an equally hard line on automation: full automation belongs only to tasks where AI demonstrably performs as well as or better than pharmacists.
That is not a new governance framework. That is your existing one, pointed at a new category of risk.
Start with the vendor, not the product
Here is where most AI evaluations go wrong before they begin. They start with the product. The demonstration shows the product on its best day, on curated data, run by the person who built it. The business model shows you the product on every other day.
The gap between pitch and practice is not hypothetical. An ASHP national survey of 1,497 pharmacy directors found that 5.7 percent of hospitals had actually deployed AI or machine learning in pharmacy. Adoption is early, and the pitches are not. The public record already shows what happens when claims outrun evidence: a chatbot that answered only 10 of 39 drug-information questions acceptably, the first state attorney general settlement over a hospital AI tool’s overstated accuracy claims, a transcription model found inventing content in medical notes across roughly 40 health systems, and a payer’s care-denial algorithm ordered into federal court discovery.
The regulatory ground shifted this year too. FDA narrowed its oversight of clinical decision support software in January 2026, which formally moved more of the diligence burden onto the buying organization. There is help on the other side of that shift: for AI-based decision support inside certified EHR technology, the ONC HTI-1 rule requires vendors to publish 31 standardized source attributes describing how the tool was built, trained, and validated. Ask for them. Silence is an answer.
So before the monograph, map the vendor. I use a Vendor Reverse Business Model Canvas: the classic nine-block canvas, worked backward from the vendor’s side of the table. Read in reverse, it surfaces the questions a pitch is built to avoid. After thirty years of sitting on the health system side of vendor negotiations, I can tell you that five blocks carry most of the weight for pharmacy buyers:
Who actually powers the technology? Many AI vendors are a thin workflow layer over someone else’s model. If the upstream provider changes pricing, terms, or the model itself, your tool changes with it, whether or not anyone tells you.
What data does the vendor depend on, and where does yours go? Training rights, retention periods, de-identification claims, who owns the model after it learns from your pharmacy, and what happens to your data if the relationship ends.
How does the vendor make money, exactly? Per seat, per transaction, share of savings, or paid in your data. Pricing is the vendor’s incentive structure, and the unit of revenue tells you what the product is optimized to produce.
Where do the vendor’s incentives diverge from your outcomes? A tool paid per alert has a relationship with alert volume. A tool paid per conversion has a relationship with utilization.
Are you the core customer, the beta site, or the training set? Sometimes the honest answer is all three. You should know which before you sign.
The first block you cannot complete is probably your next due-diligence question.
From canvas to monograph
The completed canvas should feed a record that looks familiar to anyone who has sat on a P&T committee: an AI formulary monograph. Mine has twelve fields. The approved indication, meaning the specific use rather than the category. The exact product and model version. Intended users. Excluded uses. Evidence required for approval. Local validation results, on your patients and your data. Human verification points. A named monitoring owner. Performance thresholds that trigger review. An incident reporting pathway. A reapproval date. Exit criteria.
Two of those fields do work that drug monographs never had to. Model version matters because drugs do not silently reformulate overnight. Models do, sometimes without notice, which is exactly why the first canvas block exists. And exit criteria matter because the most expensive AI failure is not the tool that gets rejected. It is the weak tool that becomes an embedded workflow dependency nobody can unwind.
Monitoring already has a name
The post-approval half of this lifecycle has a direct lineage from pharmacy. In 2021, Peter Embi proposed the term algorithmovigilance in JAMA Network Open: a discipline for detecting, understanding, and preventing the adverse effects of algorithms in clinical use, modeled deliberately on pharmacovigilance. Researchers have since transposed the core pharmacovigilance practices to AI systems: post-approval evaluation, case reporting, standardized terminology, causality assessment, and dissemination of adverse events. Pharmacy did not borrow this playbook from the AI field. The AI field borrowed it from pharmacy.
National guidance is converging on the same lifecycle. The Joint Commission and the Coalition for Health AI released their Responsible Use of AI in Healthcare guidance in September 2025, and its seven elements read like a formulary system for algorithms: governance structures with board visibility, local validation, ongoing quality monitoring, risk assessment, and voluntary blinded reporting of AI safety events. It is not binding today, but it is the clearest public signal of where accreditation expectations are heading.
Put it all together and the lifecycle is one approval gate and one sentence. Map the vendor. Review the evidence. Approve a bounded use. Monitor real-world performance. Renew, restrict, suspend, or retire.
This is a faster yes, not a slower no
The objection I hear most is that this adds bureaucracy. In practice it does the opposite. A defined indication, a visible evidence standard, and explicit stop criteria give strong vendors a clear path forward. They also keep weak claims from calcifying into systems your staff depends on. Vendors with real evidence tend to welcome the monograph. Vendors without it tend to reschedule the meeting.
Pharmacy buyers are uniquely qualified to lead this work because it is the balancing act they already perform every day: evidence, safety, operations, economics, and continuity of care. AI governance is not foreign territory. It is a familiar discipline applied to a new category of risk.
Before your next vendor meeting
Map three things: how the vendor makes money, who owns the model after it learns from your pharmacy, and whether you are the core customer, the beta site, or the training set. The block you cannot fill is your next question.
To make that first pass easier, I built Pharmacy Buyer AI Guardrails, a free, vendor-neutral resource for hospital pharmacy leaders. It includes a Reverse Canvas prompt builder that runs entirely in your browser, a blank canvas worksheet and a fictional worked example, and a six-tool buyer toolkit covering the five categories of AI risk, vendor due diligence, the approval gate, role-based AI literacy, the incident-response first hour, and a first-90-days plan. Everything opens and prints without an email address or an account. Through September 30, 2026, pharmacy buyer leaders at hospitals and health systems can also claim three free consultation hours against a live vendor decision or a governance question.
None of it makes the purchasing decision for you. It reveals the questions the vendor presentation did not answer. It was built for the people who sign, vote, and answer for the outcome.
This piece accompanies “AI Risk Management for Pharmacy Leaders: Building Guardrails Before You Need Them,” presented at the Hospital Pharmacy Buyer Conference, August 2026.
Sources
The living source register for this work, with publication dates, review dates, and what each source is and is not, is maintained at Pharmacy Buyer AI Guardrails: source register. Primary sources from that register:
ASHP. Digital Health and Artificial Intelligence Resource Center. Updated continuously.
FDA. Clinical Decision Support Software Guidance. January 2026 revision.
ONC (HHS). HTI-1 Final Rule and the Decision Support Interventions certification criterion, including the source-attribute disclosures.
Joint Commission and the Coalition for Health AI. Guidance on the Responsible Use of AI in Healthcare. September 2025.
Cited directly in this piece:
Bastow S, et al. ASHP Statement on Artificial Intelligence in Pharmacy. American Journal of Health-System Pharmacy. 2025;82(19):e853–e858.
Schutz N, et al. ASHP Statement on the Use of Artificial Intelligence in Pharmacy. American Journal of Health-System Pharmacy. 2020;77(23):2015–2018.
Embi PJ. Algorithmovigilance: Advancing Methods to Analyze and Monitor Artificial Intelligence-Driven Health Care for Effectiveness and Equity. JAMA Network Open. 2021;4(4):e214622.
Algorithmovigilance, lessons from pharmacovigilance. npj Digital Medicine. 2024.