Summary
Agentic AI systems act on patient records, send messages, and write to your EHR, so governing them takes more than a model review. This guide gives hospice and healthcare leaders one program that combines NIST, Joint Commission and CHAI guidance, and HIPAA through autonomy tiers, least-privilege access, and agent-level audit trails. It also covers who owns governance, how to find shadow and vendor-embedded AI, and a 90-day plan for a right-sized start.
Introduction
Healthcare AI has moved from tools that suggest to systems that act. AI in hospice care now reads referral packets, drafts HOPE documentation, matches caregivers to visits, and messages families. When an agent gets one of those steps wrong, the error repeats across every referral and every family it touches before a person sees it. Patients and clinicians feel that first.
Oversight is arriving in pieces. The Joint Commission launched a voluntary Responsible Use of AI in Healthcare (RUAIH) certification on June 1, 2026. NIST opened its AI Agent Standards Initiative in February. And on October 28, HHS published the HTI-5 final rule, which strips the model card and risk management requirements for predictive decision support out of ONC’s certification criteria. The transparency requirement inside your EHR just got smaller, so more of the proof now has to come from you.
This guide is for COOs, CIOs, compliance officers, and boards at hospice care agencies and other regulated providers who need a defensible answer to one question: what governance has to exist before an agent is allowed to act? It’s part of our series on AI agents for hospice care operations, and the hub article, How AI Agents Improve Hospice Operations While Staying CMS-Compliant, covers the wider agent layer. Here we focus on the program behind it: the frameworks, controls, and evidence that make AI-powered automation for hospice defensible.
What is agentic AI governance in healthcare?
Agentic AI governance in healthcare is the combination of policies, named owners, technical controls, and evidence that decides what an AI agent may do, who answers for it, and how you prove it afterward. It needs its own treatment because of how agents work. Generative AI produces text or a summary for a person to use. An agent plans a multi-step task, calls tools, reads records from several systems, and writes results back. Governance has to attach to what the agent does.
Picture a healthcare ready AI agent in a hospice workflow. It reads a referral, scores it, updates a status field, and emails a nurse. That’s four actions, three systems, and one chain of accountability. The table shows how the control point shifts compared with the AI you may already govern.
| Predictive model | AI scribe | AI agent | |
| What it produces | A score or forecast | A draft note or summary | Actions: reads, decides, writes, and sends across systems |
| Where the risk sits | Accuracy and bias of the score | Accuracy of the draft and clinician sign-off | Permissions, tool use, chained errors, and who approved what |
| Main control | Validation and drift monitoring | Clinician review before signing | Autonomy tiers, scoped identity, action-level logs, kill switch |
| Evidence an auditor asks for | Validation results | Review and edit history | A full action trail per run, tied to a named owner |
Which framework should you build on: NIST AI RMF, Joint Commission and CHAI, or ISO 42001?
A responsible AI governance framework for healthcare doesn’t need to be invented from scratch. Three sources cover most of it, and each does a different job.
The NIST AI Risk Management Framework gives you the shared language for risk: Govern, Map, Measure, and Manage. It’s voluntary and technology-neutral, so it tells you to map and measure risk without telling you how to give an agent its own identity or cap what it can touch. That’s the gap NIST is working on now. Its Center for AI Standards and Innovation launched the AI Agent Standards Initiative on February 17, 2026, with a request for information on agent security and a concept paper on agent identity and authorization. Until those standards land, NIST AI RMF healthcare programs need agent-specific controls added on top.
The Joint Commission and CHAI guidance, released in September 2025, supplies the healthcare operating checklist. It sets out seven elements. The table translates each one into what it means when the AI in question can act.
| Element | What the guidance asks | What it means for an agent |
| 1. AI policy and governance structures | Policies, plus a governance body that keeps leadership informed of uses and outcomes | A committee, an agent register, and a named owner for every agent |
| 2. Patient privacy and transparency | Data access and use policies; tell patients when AI directly affects their care | Disclosure language for patient-facing agents; minimum-necessary access |
| 3. Data security and data use protections | Security policies and limits on how exported data is used | Agent credentials, encryption, and vendor terms that cover agent data |
| 4. Ongoing quality monitoring | Vendor validation information and risk-based monitoring after deployment | Drift and override tracking after go-live, tied to pause thresholds |
| 5. Voluntary, blinded reporting of AI safety events | Share events with an independent body such as a patient safety organization | An internal event process that feeds external reporting |
| 6. Risk and bias assessment | A process to find and address risk and bias in AI tools | Bias checks on any agent that influences access to care |
| 7. Education and training | Train users on benefits and risks | Role-specific training for staff who approve or override agent actions |
In June 2026 the Joint Commission turned that guidance into the voluntary RUAIH certification. It’s organized around five areas: governance; effective data management; risk and bias reduction; monitoring, evaluating, and validating performance; and transparency, education, and training. It assesses how an organization uses AI and doesn’t certify individual products. You don’t need Joint Commission accreditation to apply. CHAI, the author of the Blueprint for Trustworthy AI and the governance playbooks the certification builds on, said the two are tightly aligned.
Should you pursue RUAIH certification or ISO 42001?
ISO/IEC 42001 is the international standard for an AI management system, and it can be certified in any industry. It makes sense when payers, enterprise partners, or a parent company ask for it. Most hospice care agencies will get further, faster by treating the five RUAIH areas as the internal audit checklist for their own program, then adding ISO 42001 later if a customer requires it. Decide on certification after the program has run for a full quarter and produced real evidence.
What makes agentic AI risk management different from other healthcare AI?
Agents add failure modes that a model review never tests. OWASP’s Top 10 for Agentic Applications (2026 edition) lists goal hijack, tool misuse and exploitation, and identity and privilege abuse among its leading risks. Here’s how those and three related risks show up in hospice operations, and the control that answers each.
| Risk | Example in hospice operations | Control |
| Prompt injection and goal hijack | A scanned referral packet or inbound email carries text telling the agent to ignore your criteria | Treat documents as data, keep instructions separate, require human approval on status changes |
| Tool misuse | An agent with EHR write access updates fields beyond its task | Scoped tool permissions, an allow-list of actions, rate limits |
| Identity and privilege abuse | The agent runs on a shared service account or a staff member’s login | A unique agent identity, time-bound credentials, least-privilege access |
| Persistent memory risk | An agent carries one patient’s details into another patient’s session | Task-scoped memory, retention limits, nothing carried between patients |
| Multi-agent delegation | An intake agent hands off to a scheduling agent and nobody owns the combined result | One named owner for the whole workflow, with the full chain logged |
| Drift and bias | Referral formats or patient mix shift after go-live | Sampled review against clinician-confirmed values, subgroup checks |
Fragmented accountability is the risk leaders underrate. When three agents each do something reasonable and the combination harms a patient, an auditor will ask who approved the chain. A workflow owner is the answer.
What autonomy levels should you allow, and what must always need a human?
Autonomy works like a dial, and your committee should set it use case by use case. This four-tier model is a practical starting point for human-in-the-loop design.
| Tier | What the agent may do | Hospice example | Oversight |
| 1. Recommend | Prepares analysis or a draft; a person takes every action | HOPE documentation draft, IDG pre-meeting summary | Clinician reviews and signs |
| 2. Act with approval | Completes the task up to the final step; a person approves before it takes effect | Eligibility screening returns Eligible, Pending Review, or Ineligible; the status applies after approval | Named approver for each decision |
| 3. Act within guardrails | Acts alone inside narrow, pre-approved limits; exceptions escalate | Appointment reminders, document requests to referral sources | Sampled review and exception alerts |
| 4. Monitor and alert | Runs continuously on non-patient-facing data and raises alerts only | Data quality checks, cap exposure alerts | Threshold alerts and periodic audit |
Some actions should stay with a person at every tier. Our recommended minimum list:
- Prognosis and certification of terminal illness. Federal rules put certification with a physician.
- Admit and decline decisions.
- Any entry in the legal medical record without a clinician’s signature.
- Patient or family communication about prognosis or end-of-life decisions.
- Claims or attestations sent to a payer.
- Changes to an agent’s own criteria, permissions, or tier.
The stakes show in the audit data. An HHS OIG audit published in June 2026 found that 45 of 100 sampled initial hospice certification periods, drawn from fiscal year 2021 claims, didn’t meet Medicare requirements. An agent that screens at scale multiplies whatever standard it’s given, good or bad. Build a kill switch and a tested rollback path before go-live: one control that pauses the agent and a way to reverse its recent writes.
What identity and permissions should an AI agent have?
Give every agent its own identity, and give that identity the least access that lets it finish one task. Shared service accounts and borrowed staff logins break accountability, because the log then shows a person doing something a machine did. Least privilege also matches HIPAA’s minimum necessary standard. An agent preparing a discharge summary has no reason to read every chart in the EHR.
NIST’s agent identity work frames the problem as identification, authorization, auditing, and non-repudiation. In practice that means separate read and write scopes, credentials that expire per task, an allow-list of the tools each agent may call, and an approval gate on any write to a legal record.
We build Caregence agents around minimum-necessary access and role-based permissions, with secure identity management in place of shared credentials and every agent action logged. Whatever platform you choose, ask the vendor to show you those controls in a working environment.
What audit trail should every agent action produce?
If an agent’s action can’t be reconstructed, it can’t be defended. Each run should leave a record that answers who, what, when, under which rules, and who approved. The HIPAA Security Rule’s audit controls standard (45 CFR 164.312(b)) already expects activity in systems holding ePHI to be recorded and examined. Agent runs make that expectation concrete at the tool-call level.
| Log field | Why it matters |
| Agent identity, model, and configuration version | Shows exactly which system and rules acted |
| Source records and documents read | Ties the output to its evidence |
| Tool calls: system, action, and fields read or written | Gives you action traceability |
| Criteria or rule version applied, with the factors behind the result | Lets you explain the outcome to a reviewer |
| Human reviewer, decision, and timestamp | Proves a person approved where policy requires one |
| Overrides, exceptions, and escalations | Shows the guardrails worked |
| Tamper-evident storage and retention period | Supports non-repudiation: nobody can quietly alter the record |
Logs show what the agent did. Lineage shows where its inputs came from, and a CMS or MAC reviewer will want both. If your source data can’t be traced, a perfect action log still leaves a gap, which is why audit-ready data lineage belongs in the same program.
How do you handle shadow AI and AI embedded in vendor products?
You can’t govern what you haven’t listed. Start an AI register that covers three groups: tools you built or bought for AI use, AI features switched on inside vendor products such as your EHR, CRM, and billing platform, and public tools staff use on their own, often called shadow AI. Pull from procurement records, vendor release notes, single sign-on and network logs, and a short staff survey.
Each register entry should carry the same fields:
- Business owner and purpose
- Whether it touches PHI, and the status of the business associate agreement
- Autonomy tier and the actions it can take
- Vendor validation and bias information on file
- Last validation date and next review date
Ask every vendor how the tool was tested and validated, how bias was evaluated, and whether they’ll validate on a sample from your own population. The Joint Commission and CHAI guidance recommends exactly that posture in procurement. With HTI-5 trimming model card requirements for certified health IT, write those disclosures into the contract.
A register works best when data owners and stewards already exist. That foundation is the subject of our guide to data governance and CMS compliance frameworks for hospice organizations.
How do HIPAA, FDA rules, HTI-5, and state AI laws fit into one program?
AI governance for regulated healthcare gets simpler when you treat these as lenses on the same agent instead of separate projects. Run every new use case through one screen at intake, then apply the controls from the sections above.
| Authority | What it asks of agent programs | Where it lands in your program |
| HIPAA | PHI used by agents and their vendors falls under the Privacy and Security Rules, including business associate agreements, minimum necessary use, and audit controls | Scoped identity, action logs, vendor contracts |
| FDA | The January 2026 Clinical Decision Support Software guidance clarifies which decision support functions are devices. Updates to AI-enabled device software can be pre-authorized through a predetermined change control plan | A classification question at intake. Administrative agents usually sit outside device rules, but ask it for any agent that recommends patient-specific clinical action |
| ONC HTI-1 and HTI-5 | HTI-1 created decision support intervention transparency criteria for certified health IT. HTI-5 removes the model card and predictive risk management requirements | Contract demands on EHR vendors and your own model documentation |
| Section 1557 (45 CFR 92.210) | Covered entities must make reasonable efforts to identify and mitigate discrimination risk in patient care decision support tools | Bias checks on any tool that influences access to care |
| State AI laws | A moving patchwork. Colorado replaced its 2024 AI Act with the narrower SB 26-189 on May 14, 2026 | A state matrix that counsel reviews each quarter |
| CMS program rules | Hospice certification and documentation standards still apply to agent-prepared records | Physician certification, human sign-off, audit-ready records |
HTI-5 deserves a closer look because it’s brand new. The final rule, published in the Federal Register on October 28, 2026, revises the decision support criterion that HTI-1 introduced in 2024. ONC said it found no evidence that the transparency requirements had improved patient care, and it reminded users of certified health IT of their duty to identify bias in predictive tools and avoid discrimination under Section 1557. The rule adjusts certification criteria for health IT developers. It doesn’t change your own HIPAA or Section 1557 obligations.
Who should own AI governance: compliance, the CIO, the CMIO, or a committee?
A cross-functional AI governance committee owns the policy, and named individuals own each agent. The Joint Commission and CHAI guidance points to executive leadership, compliance and ethics, IT, cybersecurity, safety and incident reporting, and clinical and operational leaders. For hospice care agencies and hospitals alike, the roles divide like this.
| Role | Owns |
| CEO or COO (executive sponsor) | Risk appetite, resourcing, and accountability to the board |
| CIO and CISO | Agent identity, access, logging, and vendor security |
| Compliance and privacy officer | The regulatory map, HIPAA, Section 1557, state law, and audit readiness |
| Medical director or CMIO | Clinical safety, autonomy tier decisions, and what always needs a physician |
| Legal counsel | Vendor contracts, liability allocation, and disclosure language |
| Workflow owner | Day-to-day performance of one agent and its end-to-end workflow |
| Board or board committee | Quarterly oversight report and approval of high-risk use |
Who answers when an agent errs and a clinician or staff member relied on it? The organization that deployed it, because patients and regulators hold the provider responsible whatever the vendor contract says about sharing costs. Liability turns on the facts and on state law, so take specific scenarios to counsel. What you can control is the record: a policy that says staff remain responsible for decisions they sign, that agent output is decision support, and that every reliance and override is logged.
Boards should ask for four numbers each quarter: agents in production by tier, overrides and exceptions, safety events and near-misses, and open vendor gaps.
How do you monitor agents after go-live and report AI safety events?
Go-live is where the real testing starts. Track four signals: accuracy against clinician-confirmed samples, override and exception rates, input drift such as a new referral source format, and outcomes across patient groups. Set thresholds that pause the agent, automatically or through a named person, and test the rollback before you need it.
The Joint Commission and CHAI guidance treats AI safety events like patient safety events and encourages voluntary, blinded reporting to independent bodies such as patient safety organizations. Build the internal side first. You need a simple intake form, a definition that includes near-misses, a review within a set number of days, and a no-blame rule. People report what they won’t be punished for reporting.
What does a right-sized AI agent governance framework look like in 90 days?
A mid-size provider or payer doesn’t need a large team. A committee that meets monthly, one register, and one standard for logs and permissions cover most of the ground.
- Days 1 to 30: Charter the committee and name an executive sponsor. Adopt a one-page AI policy and the four autonomy tiers. Build the AI register from procurement, vendor, and sign-on data.
- Days 31 to 60: Run every register entry through the five-lens screen: HIPAA, FDA, HTI-5, Section 1557, and state law. Set the logging and permissions standard. Send vendor questionnaires covering validation, bias testing, and model documentation.
- Days 61 to 90: Pilot one Tier 2 agent in shadow mode with monitoring thresholds, a tested kill switch, and the safety-event form live. Deliver the first board report.
Before any agent goes live, confirm each item on this checklist:
- A named workflow owner and approver
- An autonomy tier approved by the committee
- A unique agent identity with least-privilege scopes
- Action-level logging verified in a test run
- The HIPAA, FDA, Section 1557, and state-law screen documented
- Vendor validation and bias information on file, and a signed business associate agreement
- Monitoring thresholds, kill switch, and rollback tested
- Staff trained to approve and override
Final thoughts
Agents will keep getting more capable, and regulators are still writing the rules. The programs that hold up share three habits. They tier autonomy by risk, they give every agent its own identity, and they keep a record good enough to hand a surveyor. Start with one agent, one owner, and one audit trail, then widen the circle.
