AI Vendor Risk Assessment: Questionnaire, Scoring Model, and Evidence Checklist (2026)
An AI vendor risk assessment is the review you run before a third-party AI product or API touches your data: you classify how risky the use case is, ask AI-specific questions (training on your data, retention, sub-processors and model providers, security testing, output risk, regulation), collect evidence for each answer, score the vendor, and set contract terms and monitoring. The difference from a normal vendor review is that AI vendors add model behavior, data reuse, and a chain of downstream model providers. This guide gives a ready-to-use questionnaire, a scoring model, and an evidence checklist, current as of October 26, 2026.
It is written for security, procurement, and platform teams approving AI APIs, AI gateways, and SaaS tools with AI features. It is not legal advice.
Why AI vendor risk is different
A traditional third-party review asks about access control, encryption, uptime, and certifications. AI vendors need all of that plus:
| New risk | Why it matters | Example question |
|---|---|---|
| Data reuse for training | Your prompts could improve a model others use | Is customer data used for training or evaluation by default? |
| Fourth parties | The vendor may forward data to model providers | Which model providers process our data, and in which regions? |
| Model behavior | Outputs can be wrong, unsafe, or leak data | How is the system tested for prompt injection and data leakage? |
| Fast change | Models and versions change without notice | How are model changes announced and versioned? |
| Regulation | EU AI Act and sector rules add duties | How do you classify this system under the EU AI Act? |
| Cost exposure | Usage-based pricing can spike | Are there quotas, spend caps, and alerts? |
The NIST AI Risk Management Framework explicitly covers third-party AI risk in its Govern and Map functions, and the NIST Generative AI Profile (AI 600-1) lists value-chain and component-integration risk as a generative AI risk.
Step 1: Tier the vendor by use case
Not every AI tool needs the same depth. Tier first, then scale the questionnaire.
| Tier | Criteria | Review depth |
|---|---|---|
| Tier 1 (high) | Customer data, personal data, regulated data, or the AI takes actions (sends, pays, changes records) | Full questionnaire, evidence, security testing, legal review |
| Tier 2 (medium) | Internal data, human reviews every output | Core questionnaire + key evidence |
| Tier 3 (low) | Public data only, no integration | Short form + terms review |
Record the tier and why in the vendor file. Re-tier when the use case changes.
Step 2: Send the AI vendor risk questionnaire
Use these questions on top of your standard security questionnaire (for example the Shared Assessments SIG or the CSA AI Controls Matrix).
Data handling and privacy
- Is our data (prompts, files, outputs) used to train or evaluate any model? Default setting? Opt-out or contractual commitment?
- How long are prompts, outputs, and logs retained? Can we set zero or short retention?
- Where is data processed and stored (regions)? Can we pin a region?
- Which sub-processors and model providers receive our data? How are we notified of changes?
- Is a DPA available, and does it cover AI processing and international transfers?
Model and system governance
- Which models power the product, and are they your own or third-party?
- How are model or version changes communicated and tested?
- What evaluation do you run for accuracy, bias, and safety, and can you share results?
- Do you have documentation of intended use and known limits (model or system cards)?
Security
- Which certifications or attestations cover the AI service (SOC 2 Type II, ISO 27001, ISO 42001)? Is the AI product in scope?
- How do you test for prompt injection, data leakage, and excessive agency (for example against the OWASP Top 10 for LLM Applications)?
- How are API keys issued, scoped, rotated, and revoked?
- What do logs contain, who can access them, and can we export them?
- What is your incident response and notification timeline for AI-related incidents?
Operations and cost
- What is the SLA, and does it cover the AI features and upstream model providers?
- What happens when an upstream model provider is down: fallback, degrade, or fail?
- Are there per-key quotas, spend caps, and usage alerts?
- What are the exit terms: data export, deletion confirmation, notice periods?
Regulation
- How do you classify the system under the EU AI Act, and which obligations do you meet as provider or deployer support? The implementation timeline currently shows high-risk Annex III requirements applying from 2 December 2027.
- Do outputs carry disclosure or marking for AI-generated content where required?
Step 3: Collect evidence, not just answers
Every "yes" in a Tier 1 review should point to something you can file.
| Answer | Evidence to request | What to check |
|---|---|---|
| "We don't train on your data" | Contract clause, DPA, product setting doc | Default vs opt-out; covers sub-processors too |
| "Data is retained 30 days" | Retention policy, admin setting screenshot | Applies to logs and backups |
| "SOC 2 Type II" | Report under NDA, bridge letter | AI service in scope; period end date; exceptions |
| "We test for prompt injection" | Test summary or pen test report | Date, scope, findings, retest |
| "Per-key quotas" | Docs or console screenshot; your own test | Limits actually block requests |
| "Model providers listed" | Sub-processor list with regions | Matches your data residency needs |
A certificate is screening evidence, not proof that your use case is safe. For gateway-specific evidence, see SOC 2 evidence for an AI API gateway, the AI gateway DPA checklist, and the AI API data retention checklist.
Step 4: Score the vendor
Score each domain 1-5 (5 = strong, evidenced) and weight by what matters for your use case:
| Domain | Weight (Tier 1 example) | Score 1 | Score 5 |
|---|---|---|---|
| Data use and retention | 25% | Trains on data by default, unclear retention | No training, configurable retention, contractual |
| Security and testing | 25% | No AI-specific testing | Recent tested evidence with retest |
| Fourth parties and residency | 15% | Unknown model providers | Full list, regions, change notice |
| Governance and change | 15% | Silent model changes | Versioned, announced, evaluated |
| Operations and cost controls | 10% | No SLA, no quotas | SLA, fallback, quotas, caps |
| Regulatory readiness | 10% | No view | Documented classification and obligations |
A simple scoring script you can keep with the vendor file:
WEIGHTS = {"data": .25, "security": .25, "fourth_parties": .15,
"governance": .15, "operations": .10, "regulatory": .10}
def vendor_score(scores: dict, evidence: dict) -> dict:
"""scores: 1-5 per domain; evidence: True if the score is backed by filed evidence"""
capped = {d: (s if evidence.get(d) else min(s, 2)) for d, s in scores.items()} # no evidence -> max 2
total = sum(WEIGHTS[d] * capped[d] for d in WEIGHTS)
decision = "approve" if total >= 4 else "approve with conditions" if total >= 3 else "reject or escalate"
return {"score": round(total, 2), "decision": decision, "capped": capped}
print(vendor_score(
{"data": 5, "security": 4, "fourth_parties": 4, "governance": 3, "operations": 4, "regulatory": 3},
{"data": True, "security": True, "fourth_parties": True, "governance": False, "operations": True, "regulatory": False},
))
The cap rule matters: an unsupported claim should never score as well as a tested one.
Step 5: Verify the risky claims yourself
For Tier 1 vendors, test a few things in a sandbox before signing:
- Key revocation: revoke a test key and confirm requests fail immediately.
- Quotas: set a low quota and confirm requests are blocked when it is reached.
- Model allow-list: call a model that should not be allowed and confirm it is denied.
- Log content: check what a request log stores (metadata only vs full prompt).
# After revoking the test key in the vendor console, this should return 401
curl -s -o /dev/null -w "%{http_code}\n" https://api.vendor.example/v1/chat/completions \
-H "Authorization: Bearer $REVOKED_TEST_KEY" -H "Content-Type: application/json" \
-d '{"model":"any-model","messages":[{"role":"user","content":"ping"}]}'
A questionnaire tells you what the vendor claims; these tests check a few claims directly. A full security test of your own AI application is a separate job, covered in our AI security audit guide.
Step 6: Write the controls into the contract
| Clause | Why |
|---|---|
| No training on customer data (including by sub-processors) | Prevents data reuse |
| Retention limit and deletion on exit | Limits exposure |
| Sub-processor and model-provider change notice | Lets you object to new fourth parties |
| Breach and AI incident notification time | Speeds response |
| Audit and evidence rights | Lets you re-check claims |
| SLA covering AI features | Makes uptime enforceable |
| Data residency | Meets transfer rules |
Step 7: Monitor after approval
- Re-review Tier 1 vendors yearly, and when the vendor changes models, sub-processors, or terms.
- Track usage and cost per vendor and per key.
- Watch for incidents and security advisories.
- Keep an approved model list so new models go through review (AI model approval workflow).
Assessing an AI API gateway as a vendor
A gateway sits between your apps and several model providers, so assess both layers: the gateway itself (keys, logs, quotas, billing, support) and each upstream provider it routes to. Ask for the route list and which providers receive which data. With Flatkey, for example, you can check per-request model, tokens, cost, and key in the usage dashboard, export logs as CSV/JSON for evidence, and create or revoke keys per environment in API key management. Put those exports in the procurement file next to the questionnaire. Our AI gateway procurement evidence packet lists the full set, and the model provider evidence review covers upstream providers.
FAQ
What is an AI vendor risk assessment?
It is a structured review of a third-party AI product or API that checks data use, security, fourth-party model providers, governance, operations, and regulatory readiness before approval, backed by evidence and a risk score.
What questions should be in an AI vendor questionnaire?
At minimum: whether your data trains models, retention, processing regions, sub-processors and model providers, certifications covering the AI service, AI-specific security testing, key and log controls, SLA and fallback, cost controls, exit terms, and EU AI Act classification.
How is AI vendor risk management different from regular TPRM?
AI adds data reuse for training, fourth-party model providers, unpredictable model output, frequent model changes, and AI-specific regulation. Standard TPRM controls still apply.
Can AI help with vendor risk management?
Yes, AI tools can summarize questionnaires and flag missing evidence, but a person should make the approval decision and verify the evidence, especially for Tier 1 vendors.
Is a SOC 2 report enough to approve an AI vendor?
No. Check that the AI service is in the report's scope and period, then still ask AI-specific questions on training, retention, model providers, and testing. The AICPA SOC 2 overview explains what a SOC 2 report covers.
Which frameworks help with AI vendor risk?
NIST AI RMF and its Generative AI Profile, the OWASP LLM Top 10 for technical risks, the CSA AI Controls Matrix, ISO/IEC 42001 for AI management systems, and the Shared Assessments SIG questionnaire.
Sources
Checked October 26, 2026: NIST AI RMF, NIST AI 600-1, OWASP Top 10 for LLM Applications, CSA AI Controls Matrix, Shared Assessments SIG, EU AI Act timeline, GDPR AI API gateway checklist.
