Auditing AI Vendors: Third-Party Risk When the Product Learns

AI GOVERNANCE | VENDOR & THIRD-PARTY RISK

Written by: Devensa AI

Figure: 78% of organizations use third-party AI tools and 53% rely on them exclusively, yet those tools account for 55% of AI-related failures. The exposure sits in the AI organizations buy. Source: MIT Sloan Management Review and BCG (2023).¹

Auditing an AI vendor means verifying not only what a product does on the day you buy it, but how it changes afterward: new model versions, new training data, new downstream providers. In research by MIT Sloan Management Review and BCG, 55% of AI-related failures came from third-party AI tools.¹

The Audit Was Accurate. Then the Model Changed.

Traditional vendor due diligence assumes a stable product. AI products break that assumption. A vendor can swap its foundation model, retrain on new data, or add agentic features without a contract amendment, so the onboarding questionnaire can describe a product that no longer exists.

That is what third-party AI risk covers: a system you did not build, and cannot inspect directly, producing outcomes you remain accountable for. It spans the vendor’s model, its training data, the fourth-party model providers behind it, and the paths your own data takes back into it.

What it gets mistaken for

It is often reduced to a security review with a few AI questions appended. That treats AI governance as a point solution bolted onto an existing process, and it misses whether outputs stay accurate and fair as the model evolves, whether your data trains shared models, and who answers when an automated recommendation goes wrong. Nor is it a decision software makes alone. A platform can surface drift and missing evidence; whether a vendor’s AI is acceptable for a high-impact use remains a decision for accountable people.

Where the Exposure Is Growing

AI is bought more often than it is built

The same MIT SMR and BCG study found that 78% of organizations use third-party AI tools, and more than half rely on them exclusively.¹ Verizon’s 2026 Data Breach Investigations Report adds that employee use of unapproved shadow AI tripled to 45%, raising data leakage.²

Supply chain compromise carries a measurable premium

IBM’s 2026 Cost of a Data Breach Report puts the global average breach at a record $4.99 million.³ Business partner or supply chain compromise ranked first among the 30 cost factors IBM analyzed, adding $227,250 above that average, and those breaches averaged 258 days to identify and contain.⁴

The regulatory clock is split, not stopped

The EU’s Digital Omnibus on AI, Regulation (EU) 2026/1744, entered into force on July 27, 2026. It deferred the AI Act’s Annex III high-risk obligations, but the Article 50 transparency duties, took effect as scheduled on August 2, 2026.⁵ High-risk obligations for providers and deployers now apply from late 2027.⁶ Deployers of a vendor’s AI carry obligations of their own and depend on the vendor for much of the documentation.

Figure: The EU AI Act timeline after the Digital Omnibus. Literacy and transparency duties already apply; Annex III high-risk obligations moved to late 2027. Sources: Gibson Dunn; Cloud Security Alliance; Pinsent Masons.⁹⁵⁶

Oversight structures are thin

Cisco’s 2026 Data and Privacy Benchmark Study found that 23% of organizations still lack a dedicated AI governance committee, and only 12% describe their existing committee as mature and proactive.⁷ Without a body that owns AI vendor decisions, each business unit sets its own tolerance. The platform’s vCAIO capability, an AI-powered vOfficer augmented by Devensa subject-matter experts where needed, is structured to hold AI vendor oversight to one consistent standard.

NIST AI RMF Puts Vendors Inside the Framework

The NIST AI Risk Management Framework does not treat third parties as an appendix. Its GOVERN 6 category calls for policies that address AI risks from third-party software, data, and supply chains, and GOVERN 6.2 calls for contingency processes to handle failures or incidents in third-party data or AI systems deemed high-risk.⁸ MAP 4 extends risk mapping to third-party components, and MANAGE 3 covers ongoing management of third-party risk. A vendor’s ISO/IEC 42001 certification is useful evidence, but it does not assess your specific use.

Devensa AI scores this through TruMaturity™, the methodology and six governance domains shared across Devensa. On the platform, the TruMaturity™ Assessment is designed as an AI-mediated, adaptive assessment, triggered on first sign-in and roughly annually after that, with scores mapped to the five CMMI maturity levels. For AI vendor oversight, the levels read like this:

• Level 1, Initial: AI vendors are unidentified, and AI features arrive inside existing software without review.
• Level 2, Managed: Known AI vendors are listed and reviewed at onboarding, usually by security alone.
• Level 3, Defined: A standard AI vendor assessment exists, with risk tiers tied to the impact of each use case.
• Level 4, Quantitatively Managed: Model changes, incidents, and performance are tracked against defined thresholds.
• Level 5, Optimizing: Findings feed back into contract terms, tiering criteria, and procurement standards.

What Changes When the Product Learns

Model versioning

Ask how the vendor versions models, how far ahead it gives notice of material changes, and whether you can pin a version for a regulated workflow.

Training rights over your data

Confirm in the contract, not the marketing page, whether prompts, outputs, and files train shared models, and whether that setting is on by default.

The fourth-party model layer

Many AI products run on another provider’s foundation model, whose outages and incidents become your exposure. Require disclosure of material model providers.

Performance evidence over time

Ask what the vendor monitors after release, such as drift, error rates, and bias metrics, and what it shares when thresholds are crossed.

Incident and exit terms

Define what counts as an AI incident, how fast it is reported, and how your data returns if you switch providers. GOVERN 6.2 is difficult to meet without these terms.

Five Moves That Hold Up Under Audit

1. Inventory the AI you already bought, including AI features switched on inside existing SaaS tools.
2. Tier by use, not by vendor. The same model can be low risk for drafting internal notes and high risk for screening job applicants.
3. Use more than one evaluation method. Organizations using seven evaluation methods were more than twice as likely to uncover AI failures as those using three.¹
4. Write change notification into the contract. A material model change should trigger re-review for high-tier uses.
5. Name a decision owner with authority to accept or decline the risk.

Frequently Asked Questions

What is an AI vendor audit?

An AI vendor audit is a structured review of a third-party AI product’s data practices, model governance, performance evidence, and contractual protections, repeated when the product materially changes.

Is a SOC 2 report enough for an AI vendor?

No. SOC 2 covers security, availability, processing integrity, confidentiality, and privacy controls over a defined period. It does not typically cover model accuracy, bias testing, training-data provenance, or model update management.

Does the EU AI Act apply if we only buy AI and do not build it?

It can. The AI Act assigns obligations to deployers as well as providers, on the timeline described above.⁵⁶ Confirm with counsel which uses fall in scope.

How often should an AI vendor be reassessed?

Tie it to change as well as the calendar: review high-impact uses after a material model change, a new model provider, or an incident, with an annual review as the floor.

Who should own AI vendor risk?

A defined AI governance function that can approve, condition, or decline a vendor, with security, privacy, legal, and the business owner contributing. When ownership is split by default, no one answers for the model’s outputs. Who in your organization could say today which of your vendors changed their model this quarter?

Sources

1.  MIT Sloan Management Review and Boston Consulting Group, “Building Robust RAI Programs as Third-Party AI Tools Proliferate” (2023), summarized by MIT Sloan. https://mitsloan.mit.edu/ideas-made-to-matter/third-party-ai-tools-pose-increasing-risks-organizations

2.  Verizon, “Breach entry point, 2026 DBIR finds” (2026 Data Breach Investigations Report news release, May 19, 2026). https://www.verizon.com/about/news/breach-industry-wide-dbir-finds

3.  IBM and Ponemon Institute, Cost of a Data Breach Report 2026, key findings as reported by eSecurity Planet (July 30, 2026). https://www.esecurityplanet.com/cybersecurity/ibm-2026-cost-of-a-data-breach-report-key-findings/

4.  IBM and Ponemon Institute, Cost of a Data Breach Report 2026, supply chain cost-factor findings as analyzed by Cycode (August 6, 2026). https://cycode.com/blog/ibm-cost-of-data-breach-2026/

5.  Cloud Security Alliance, “EU AI Act High-Risk Deadline: Deferred, Not Cancelled,” research note on Regulation (EU) 2026/1744 (August 2026). https://labs.cloudsecurityalliance.org/research/csa-research-note-eu-ai-act-high-risk-deadline-omnibus-20260/

6.  Pinsent Masons (Out-Law), “Rules on high-risk AI to be delayed under EU omnibus deal” (May 2026). https://www.pinsentmasons.com/out-law/news/rules-high-risk-ai-delayed-under-eu-omnibus-deal

7.  Cisco, 2026 Data and Privacy Benchmark Study (2026). https://www.cisco.com/c/en/us/about/trust-center/data-privacy-benchmark-study.html

8.  NIST AI Risk Management Framework, GOVERN 6.2 subcategory text, via myctrl.tools framework reference. https://myctrl.tools/frameworks/nist-ai-rmf/govern-6-2

9.  Gibson Dunn, “EU AI Act Omnibus Agreement: Postponed High-Risk Deadlines and Other Key Changes” (May 27, 2026). https://www.gibsondunn.com/eu-ai-act-omnibus-agreement-postponed-high-risk-deadlines-and-other-key-changes/

This article is provided for informational purposes only and does not constitute legal or compliance advice. Consult qualified counsel before making governance decisions.