Building Vendor Risk Assessment In-House With AI: What Works and What Doesn’t

Introduction

With the boom of AI comes the ability to DIY your own tools, including tools to assess the cybersecurity of your vendors. I build AI tools in house myself: our FAIR-CAM assessment workflow runs inside a Claude Project, with the MIT AI Risk Database loaded as a resource every time an analyst scopes a scenario that touches AI. But is it a good idea to use AI to build the vendor risk assessment piece of your third-party cyber risk management (TPCRM) program? It depends on the data your AI is working with.

I’ve talked to people running working systems they stood up themselves, wired into their procurement stack, triaging real vendors. But the trendline is pointing in the other direction. Menlo Ventures found that companies now buy 76% of their AI use cases rather than building them internally, up from 53% a year earlier.

So most companies are buying rather than building. Regardless, when someone tells me they built their own vendor risk tooling, my working assumption is that they built the part that was buildable, and the part they’re missing isn’t a software problem.

What AI Does Well in Vendor Risk Assessment

I wrote last year that AI won’t fix a broken TPCRM program: automating a bad process just gets you to bad results faster. I also said AI helps with document-heavy work.

Parsing a SOC 2 report and mapping controls to your framework is a solved problem, and so is flagging what changed between last year’s report and this one, tiering vendors against criteria you wrote, and routing intake through whatever approval logic your industry demands.

Every one of those tasks has something in common. They run on documents you already possess, about vendors you already know, against frameworks you already wrote.

The Limits of AI Agents That Read Vendor Websites

A security leader, I won’t name him, described his setup to our team recently. His agents assess a vendor by going out and reading that vendor’s website.

That’s not a bad input. We ingest the same material: trust centers, published security documentation, compliance certifications. I don’t say people lie in that material; the answers are aspirational. Self-reported evidence tells you what a vendor believes about its own posture, and what it’s willing to put in writing. Both are worth knowing.

The problem is when it’s the only thing in the system. An agent reading a trust page collects a claim. It doesn’t collect anything capable of checking that claim. You end up with the questionnaire’s core weakness, which is total dependence on vendor self-report, arriving faster and carrying more confidence than a questionnaire.

Reading what a vendor publishes about itself was never the question. What you hold it up against is, and an agent that only reads the trust page is holding it up against nothing.

What Gartner Sees Coming for AI-Written Questionnaires

Gartner reached the same place. Their Predicts 2026 research on third-party cyber risk says that by 2028, 70% of organizations and vendors will be running generative AI on both sides of the questionnaire process. Meanwhile 62% of organizations still place excessive trust in questionnaire answers that are increasingly machine-written.

Gartner calls the result output degradation. Your AI analyzes their AI’s answers. Small errors compound at each pass, and the output drifts further from what’s actually true about that vendor’s security posture while reading more fluently every cycle.

It’s a photocopy of a photocopy. Every pass loses a little accuracy, and nothing in the process tells you how much.

Gartner’s recommendation is to point AI at continuous monitoring, detection and response, not at automating check-box activities. They expect half of all TPCRM programs to be built around continuous monitoring by 2028.

The Vendor Risk Data You Can’t Build In-House

Your AI works on what you already have. What it can’t reach is a decade of scanning, and enough accumulated history to know whether what you’re looking at is normal for a vendor like this one or the start of a problem. That is the line I’d draw for anyone deciding what to build.

• External attack surface discovery across thousands of supplier domains, run continuously instead of at onboarding.
• Breach and leak intelligence that tells you a vendor was compromised before the vendor does.
• Dark web activity, including supplier credentials showing up for sale.
• Nth-party mapping that reaches past your own contracts into the vendors your vendors depend on.
• Ransomware likelihood modeling like our Ransomware Susceptibility Index®, which only works because it’s calibrated against a population and an incident history.

AI can do this work well when what sits underneath it is years of continuous scanning across the whole internet.

Why You Can’t Benchmark Vendors Against Their Sector

Everything above you could build, given enough time and enough money, and this one you can’t build with either.

You cannot compare a vendor against its sector from inside your own vendor list. You see your suppliers, through what your suppliers choose to disclose. Asking whether this vendor’s posture is normal for a payment processor of its size is a question your own data can never answer, this year or in ten years.

The Bottom Line

The security leader I described earlier reached the same conclusion on his own. He told us our data was better than what his agents were collecting.

Deloitte’s research on AI in third-party risk found that combining managed services with in-house capability development is the preferred approach among the leaders they surveyed. Hybrid, not pure build, which leaves the only question that matters: which parts you build and which parts you buy.

I wrote a few months ago that when AI runs your monitoring loop, human judgment becomes the differentiator. The machine handles the pattern, your team owns the exception. The machine can only handle the pattern if someone feeds it signal worth matching against. Build the workflow that fits your regulators and your procurement stack. Nobody knows that better than you do. Then feed it cyber risk intelligence built on the decade of scanning you don’t have.

See what continuous vendor intelligence looks like when it’s collected at scale instead of scraped from a vendor’s homepage.