Security, Compliance & Offshore Delivery

Choosing an AI Development Partner for Healthcare: A Twelve-Point Evaluation Checklist

How to pick the best AI development partner for healthcare: 12 checks covering BAAs, PHI, de-identification, FDA CDS rules, evals, bias, FHIR and incident response.

Fwdpod14 min read
Choosing an AI Development Partner for Healthcare: A Twelve-Point Evaluation Checklist — Security, Compliance &

The best AI development partner for healthcare is the one that will sign a business associate agreement, keep PHI inside your environment and theirs under written rules, prove clinical accuracy with evals on your data, and tell you early whether your product is a regulated medical device. Engineering talent is necessary, but it is the easiest of those four things to find.

This checklist turns that into twelve points you can score, each with the question to ask, what a good answer sounds like and the red flag that should end the conversation. It assumes you have already run a general vendor screen such as how to choose an AI development partner; everything below is what healthcare adds on top. It is part of our Security, Compliance & Offshore Delivery series.

This article explains US federal rules and other regimes in general terms. It is not legal or regulatory advice. Involve your privacy officer, counsel and, where relevant, regulatory affairs before you sign.

Who can build a HIPAA compliant AI system?

Any competent engineering team can, but only if the whole chain of people and services touching PHI is covered by business associate agreements and runs under Security Rule safeguards. HIPAA compliance is a property of how the system and the organizations around it operate. It is not a badge a vendor carries.

That point matters because "HIPAA certified" appears in plenty of vendor marketing. HHS says plainly that no standard requires an entity to "certify" compliance, and that it "does not endorse or otherwise recognize private organizations' 'certifications'" regarding the Security Rule (HHS HIPAA FAQ). A third-party assessment can still be useful evidence. It is not a substitute for the twelve checks below.

A HIPAA compliant AI development team, in practice, is one that can show you three things: the BAA it will sign, the controls it runs on PHI day to day, and the downstream agreements with every model provider, cloud and tool it plans to use.

The twelve points at a glance

#CheckpointEvidence to requestWeight for a PHI-touching build
1Willingness to sign a BAATheir BAA template, or redlines on yoursPass/fail
2PHI handling rulesData flow diagram showing every place PHI landsHigh
3De-identification methodWritten method: Safe Harbor or Expert DeterminationHigh
4Model provider BAAs and retentionProvider BAA scope, retention settings per accountPass/fail
5Clinical safety and human oversightHazard log, review workflow, override designHigh
6FDA device and CDS awarenessTheir read of your intended use against the CDS criteriaHigh if clinical
7Audit loggingSample audit record for one PHI accessHigh
8Evals for clinical accuracyEval plan, test set design, clinician sign-offHigh
9Bias testingSubgroup results on a past project or pilotMedium to high
10EHR integration with HL7 FHIRFHIR resources and auth flows they have builtMedium
11Incident responseRunbook, notification window, last tabletop dateHigh
12Healthcare referencesTwo references you can call, one from complianceMedium

Treat points 1 and 4 as gates. If either fails, the rest of the score does not matter for a build that touches PHI.

What should each checkpoint cover?

1. Will they sign a business associate agreement?

Ask: "Will you sign our BAA, and will you flow its terms down to every subcontractor?" Good answer: yes, with specific redlines rather than a blanket refusal, plus a list of the subcontractors who would also need agreements. Red flag: "We never see PHI, so we don't need one," from a team that will build pipelines, debug production or touch logs.

HHS lists IT vendors, managed service providers and cloud service providers among the examples of business associates, and says a business associate must have a BAA with a subcontractor "before disclosing PHI to the subcontractor" (HHS business associate guidance). Encryption does not get a vendor out of this. HHS's cloud guidance says a provider that stores only encrypted ePHI without the key is still a business associate (HHS cloud computing guidance).

2. How will PHI move, and where will it rest?

Ask: "Show me every place PHI could land in this system: prompts, vector stores, caches, traces, eval sets, tickets, laptops." Good answer: a data flow diagram with PHI kept in your accounts, masked before it reaches observability tools, and never on endpoints. Red flag: a diagram that stops at the database and ignores logs and evaluation data.

AI systems create more copies of data than conventional apps. The general controls (client-owned cloud, SSO, no data on laptops, secrets in a vault) are covered in the 12 data security controls to demand in writing. For healthcare, add one rule: name each store that may hold PHI, and confirm each is covered by a BAA. Where the data physically sits is its own question, covered in our guide to AI data residency.

3. Which de-identification method will they use, and on what?

Ask: "When you say 'de-identified', which HIPAA method do you mean, and how do you handle free-text notes?" Good answer: a clear choice between the two methods HHS recognizes, with a plan for clinical narrative text. Red flag: "We strip names and MRNs."

HIPAA offers two routes (HHS de-identification guidance). Safe Harbor removes eighteen categories of identifiers, including all date elements other than year and geographic units smaller than a state, and also requires that you have no actual knowledge the remaining data could identify someone. Expert Determination relies on a qualified expert concluding the re-identification risk is very small and documenting the methods. HHS also says identifiers must be removed "regardless of its location in a record," which is why clinical notes and transcripts are where de-identification most often fails.

4. Do the model providers sign BAAs, and what do they retain?

Ask: "Which model APIs will process PHI, under whose account, and which BAA covers each one?" Good answer: named providers, the specific services each BAA covers, and the retention setting on each account. Red flag: a personal or vendor-owned API key, or "the provider is HIPAA compliant" with no BAA in hand.

Coverage is narrower than marketing suggests, so check each vendor's current terms:

  • OpenAI says customers may not process PHI with its API services unless a BAA expressly lists them, and requires the account to be set up for a modified retention option, such as zero data retention, before PHI is sent (OpenAI HIPAA implementation and configuration guide, July 2026).
  • Anthropic says it provides a BAA covering its "HIPAA-ready services," such as its first-party API once the BAA is signed and PHI use is enabled for the organization. Not every API feature is covered, and covered models require 30-day retention rather than zero data retention (Anthropic Privacy Center).
  • AWS lists Amazon Bedrock as HIPAA eligible, which means it can be used under an AWS BAA (AWS Bedrock security and compliance).

These terms change. Ask the partner to confirm them against your own accounts at contract time, not from memory.

5. Who checks the model's output before it reaches a patient or chart?

Ask: "Where does a clinician review, edit or reject the output, and what happens when they disagree with it?" Good answer: a designed review step, an easy override, a hazard log listing what could go wrong clinically, and a clinician on the project. Red flag: human review described as "the doctor can always ignore it."

FDA, Health Canada and the UK MHRA's Good Machine Learning Practice principles include "Focus is placed on the performance of the human-AI team" and ask developers to address human factors and how interpretable outputs are (GMLP guiding principles). A good partner designs the review screen as carefully as the model.

6. Have they told you whether you are building a medical device?

Ask: "Based on our intended use, does this software fall inside or outside FDA's non-device CDS criteria?" Good answer: a reasoned first read against the four criteria, with a recommendation to confirm with regulatory counsel. Red flag: "It's just AI, the FDA doesn't regulate that."

FDA's Clinical Decision Support Software guidance, revised in January 2026, explains when software is excluded from the device definition under section 520(o)(1)(E) of the FD&C Act (FDA CDS guidance). In summary, all four criteria must hold. The software does not analyze medical images or physiological signals. It displays or analyzes medical information. It supports a healthcare professional's recommendation. And the professional can independently review the basis for that recommendation rather than rely mainly on it. The January 2026 revision added enforcement discretion for some tools that give a single recommendation when only one is clinically appropriate (Covington, January 2026). Your partner does not make the regulatory call, but it should know enough to flag the question in week one, because a device answer changes the whole delivery plan.

7. Can you reconstruct who saw what PHI, and when?

Ask: "Show me the audit record your system writes when a user or the model retrieves a patient record." Good answer: user, patient, data elements, purpose, model call and output reference, written to logs you own. Red flag: audit logging treated as an infrastructure setting rather than part of the application.

The Security Rule's audit controls standard requires mechanisms "that record and examine activity in information systems that contain or use electronic protected health information" (45 CFR 164.312(b)). For an LLM system that means logging retrieval and generation, not just logins. Our guide to LLM audit trail architecture covers the design.

8. How will they measure clinical accuracy?

Ask: "What is your eval plan, who labels the test set, and what accuracy would stop a release?" Good answer: a test set drawn from your data and kept separate from anything used in development, clinician-labeled reference answers, task-specific metrics and a written release threshold. Red flag: "We tested it and it looked good," or accuracy quoted from a public benchmark.

GMLP principles call for training data sets that are independent of test sets and for testing "during clinically relevant conditions" (GMLP guiding principles). For generative systems, add checks for omissions and invented findings, which matter more in a clinical note than a fluent sentence does. The general mechanics of eval harnesses and guardrails are covered in how to build a production AI agent.

9. Will they test for bias across patient groups?

Ask: "How will you report performance by age, sex, race, ethnicity, language and payer type, where the data allows?" Good answer: subgroup metrics as a standard eval output, with a plan for groups too small to measure reliably. Red flag: "The model is neutral because we don't use race as an input."

For US covered entities, the Section 1557 rule at 45 CFR 92.210 bars discrimination through "patient care decision support tools" and creates an ongoing duty to identify tools using input variables that measure race, color, national origin, sex, age or disability, and to make reasonable efforts to mitigate the risk. That duty falls on you, not the vendor, so the partner's evidence becomes your evidence.

10. Have they built against real EHRs with HL7 FHIR?

Ask: "Which FHIR resources and authorization flows have you implemented, against which EHR sandboxes?" Good answer: specific resources, SMART App Launch or backend services auth, and an honest account of where vendor APIs fell short. Red flag: "We support FHIR" with no named resources or EHRs.

Certified health IT in the US must support a standardized FHIR API under ONC's 170.315(g)(10) criterion, built on US Core, SMART App Launch and Bulk Data (ONC, "Lighting the Way for FHIR API Implementation"). Integration is usually where healthcare AI timelines slip, so experience here is worth more than it looks on a scorecard.

11. What happens in the first hours of an incident?

Ask: "If PHI is exposed through a prompt, a log or a model provider, who calls whom, and how fast?" Good answer: a runbook with AI-specific incident types, a notification window in hours, named contacts and a recent tabletop exercise. Red flag: "We'd notify you promptly."

Under the Breach Notification Rule a business associate must notify the covered entity "without unreasonable delay and no later than 60 days" from discovery (HHS Breach Notification Rule). Sixty days is the legal ceiling, not a target. Set a window measured in hours in the BAA.

12. Can they give you healthcare references you can actually call?

Ask: "Can I speak to a healthcare client, including someone from their compliance or privacy team?" Good answer: two references, at least one who handled the BAA or a security review with them. Red flag: logos without contacts, or references only from engineering.

A compliance reference tells you how the vendor behaves when a control is inconvenient. For a broader question bank to run alongside these reference calls, use our 25 questions to ask an AI development company.

What changes outside the US?

If your patients or data are in the EU or India, HIPAA is not the only frame.

  • EU AI Act. AI that is, or is a safety part of, a medical device requiring third-party conformity assessment falls in the high-risk category. After the Digital Omnibus amendment, in force since 27 July 2026, high-risk obligations for AI embedded in regulated products such as medical devices apply from 2 August 2028 (CMS, July 2026). Ask the partner whether it has produced technical documentation and logging that would support a conformity assessment.
  • India DPDP Act. The Digital Personal Data Protection Rules 2025 were notified on 13 November 2025, with most operational duties, including security safeguards and breach intimation, starting on 13 May 2027 (Hogan Lovells, November 2025). The Act places duties on the data fiduciary, which then flows them down to processors by contract. If an Indian team processes your data, see is it safe to outsource AI development to India for how DPDP sits alongside your own law.

Your scoring checklist

Copy this into your evaluation sheet. Score each line 0 (no evidence), 1 (verbal answer) or 2 (documented evidence).

  • [ ] Signs a BAA and flows it down to named subcontractors (gate)
  • [ ] PHI data flow diagram covers prompts, vector stores, traces, eval sets and tickets
  • [ ] De-identification method named (Safe Harbor or Expert Determination), with a plan for free text
  • [ ] Every model API that will see PHI is covered by a provider BAA, with retention settings documented (gate)
  • [ ] Clinician review step, override path and hazard log designed in
  • [ ] Written first read of FDA CDS criteria for your intended use
  • [ ] Application-level audit record for every PHI retrieval and model call
  • [ ] Eval plan with independent test set, clinician labels and release threshold
  • [ ] Subgroup performance reported as a standard eval output
  • [ ] Named FHIR resources and EHR sandboxes previously built against
  • [ ] Incident runbook with a notification window in hours and a recent tabletop
  • [ ] Two healthcare references, one from compliance or privacy

A partner scoring 2 on both gates and 18 or more overall is worth a paid pilot. Below that, the gaps will show up later as delays in security review. For how the pilot itself should be run in a regulated setting, see from pilot to production in a regulated environment. Our wider guide to HIPAA compliant AI development covers the controls in more depth.

How Fwdpod approaches this

Fwdpod builds dedicated AI engineering pods for startups and enterprises, covering LLM development, RAG systems, AI agents and product delivery, with engineering delivery from India for buyers in the US, UK, EU and Gulf. Each pod is 4–6 experts working around a single measurable outcome, and Compliance AI is one of the pod templates we scope. We would ask any healthcare buyer, including those considering us, to score us against this checklist. Bring all twelve points, including the BAA and model provider questions, to a first call and ask the team to answer each one directly before you go any further. We make no HIPAA or certification claims in this article.

If you are shortlisting partners for a healthcare AI engineering team, talk to the team about your intended use and data flows, or configure your AI engineering pod: answer five questions and receive a custom pod proposal within 48 hours. If your project is still at the scoping stage, our AI development services page outlines how engagements are structured.

Frequently Asked Questions

1. Is there such a thing as a HIPAA certified AI vendor?

No official one. HHS says no standard requires an entity to certify HIPAA compliance and that it does not recognize private organizations' certifications regarding the Security Rule. Third-party assessments can be useful evidence, but they do not replace a signed BAA and controls you can check.

2. Does an AI vendor need a BAA if it only works on de-identified data?

Data properly de-identified under Safe Harbor or Expert Determination is no longer PHI, so a BAA may not be required for that work. In practice, most AI builds eventually touch identifiable data during integration, debugging or production support, so agree the BAA before that point rather than after.

3. Can we use ChatGPT or Claude with patient data?

Only through services the provider has agreed to cover under a BAA, configured as that provider requires. Consumer chat apps and personal accounts are not the same as a business API account set up for HIPAA use. Check the provider's current BAA scope and retention settings for your own account.

4. Will our AI tool be regulated by the FDA?

It depends on intended use. Software that analyzes medical images or signals, or that a clinician is meant to rely on without being able to review its basis, is more likely to be a device. Tools that support a clinician who can independently review the reasoning may fall outside device rules. Confirm with regulatory counsel.

5. Is an offshore team a problem for HIPAA?

HIPAA itself does not ban offshore business associates, but the same BAA, safeguards and subcontractor agreements apply. Some state Medicaid contracts and government programs restrict offshore work, and many buyers add contract terms on where PHI may be accessed. Ask the partner whether engineers ever need direct PHI access, and design the environment so most work happens on de-identified or synthetic data.

6. How long should healthcare vendor evaluation take?

Long enough to get documents for both gate checks and to run reference calls, which is usually several weeks because BAAs and security reviews involve legal and privacy teams. Start the BAA review in parallel with technical evaluation instead of after it.

7. What is the most common reason healthcare AI vendors fail evaluation?

Gaps in the model provider chain. A vendor may be ready to sign a BAA itself but plan to send PHI to a model API, tracing tool or labeling service that is not covered by one. Ask for the full list of services that will see PHI and the agreement covering each.

Planning an AI deployment?

Fwdpod places pre-vetted AI engineers from India inside your team, as one specialist or a full AI pod. Book a call about your roadmap, or browse the pods we deploy most.

View all →
Back to all insights

Tell us the role or the outcome.

We'll send matched profiles or a pod plan within 48 hours.

Book a 30-minute call