Are You Measuring the Right Risks — Or Just the Easiest Ones?

Introduction

Your dashboard is green. Your scores are trending up. SLAs are met. The quarterly risk report looks polished and ready for the board.

So why do breaches keep happening?

In the latest episode of the Third Party podcast, Jeffrey Wheatman, Bob Maley, and Ferhat Dikbiyik dig into the uncomfortable truth sitting at the center of most third-party cyber risk management programs: organizations are measuring what's easy to count, not what actually predicts failure. The result is a program that looks healthy on paper while real exposures quietly compound in the background.

It's not a data problem. It's a measurement problem. And it's more widespread than anyone in the industry wants to admit.

The Vanity Metric Trap

According to World Economic Forum’s Global Cybersecurity Outlook 2026 report, 61% of organizations experienced a third-party cyber incident in the past year, despite widespread adoption of formal TPRM programs.

Let that number land. More than half of organizations running structured risk programs still got hit through a third party. If the metrics were working, those numbers would look different.

The problem isn't effort. It's direction. Most TPRM programs default to activity metrics because they're the easiest to produce and the easiest to explain:

• Number of questionnaires completed
• Percentage of vendors assessed
• Volume of findings remediated
• How many vendors have achieved "top tier" scores

These metrics feel productive. They demonstrate work. They look great in slides. But not one of them answers the question that actually matters: are your riskiest vendors getting the right level of scrutiny?

Completing a questionnaire is not the same as reducing risk. A vendor with a high surface-level score can still fail catastrophically. The largest pipeline system for refined oil products in the U.S. had respectable security ratings right before one of the most disruptive ransomware attacks in history. The score didn't see it coming. Neither did the program built around it.

The "Top 10 Risky Vendors" Problem

Most "top risky vendor" lists are built on the wrong foundation, and the teams presenting them often know it.

Here's a scenario that plays out in risk programs every quarter: someone asks for the "top 10 most risky vendors." A list gets generated, sorted by cyber rating or open findings count. It goes into a report. The report goes to leadership. Everyone nods.

But here's what's missing: context.

What does "risky" mean in this specific situation? Risky to operations, risky to data, risky to compliance posture? Is business criticality factored in? A vendor with a mediocre cyber rating but zero access to sensitive systems is not your biggest problem. A vendor with a strong rating and deep network integration very much is.

Third-party cyber risk management exists precisely because surface-level scores were never designed to answer these questions. Risk is not a single number. It's the intersection of vulnerability, threat, and business impact — and all three have to be in frame at the same time.

Removing any one of those variables doesn't simplify the picture. It just makes it wrong.

The Metric That Should Scare You Most

When it comes to what gets surfaced to leadership, the filter should be ruthless: does this help us anticipate and prevent bad outcomes?

Vendor criticality tiers are a prime example of a metric that sounds rigorous but often isn't. Many organizations assign criticality based on contract value or data access alone, without accounting for cyber posture, concentration risk, or supply chain depth. A vendor might be classified as Tier 1 on paper while remaining completely invisible to any continuous monitoring program.

The Ransomware Susceptibility Index® (RSI™) was built precisely because the industry needed a metric that could actually predict outcomes, not just describe current posture. Knowing a vendor is highly susceptible to a ransomware attack in the next 12 months is actionable. Knowing they have 47 open questionnaire findings is a starting point, at best.

The goal isn't to eliminate lagging indicators. It's to make sure leading ones are actually in the room.

Boards Don't Want Metrics. They Want to Understand Risk.

This is where most TPRM programs lose the room.

The disconnect between what security teams measure and what executives need to act on is enormous, and it's not closing on its own. "We have 83% vendor coverage" lands very differently than "we currently have limited visibility into the vendors most likely to cause operational disruption." Both statements can be simultaneously true. Only one creates urgency.

Framing matters more than data volume. A board that hears "three of our top 10 critical vendors show elevated ransomware susceptibility" will respond differently than one receiving a 40-slide deck on assessment completion rates.

Translating technical risk into business language isn't a communication nicety — it's a core program competency. Cyber risk intelligence that maps vendor exposure to potential downtime, financial impact, and regulatory consequence speaks the language that moves budgets and drives decisions. Everything else is reporting theater.

Five Questions to Stress-Test Your Program

Use these to find out fast whether your metrics are measuring what matters:

1. Can you identify your 10 most operationally critical vendors in under five minutes — not by contract value, but by the actual impact to the business if they went offline today?
2. For your highest-risk vendors, are you tracking leading indicators (threat actor activity, dark web credential exposure, financial distress signals) or only lagging ones (assessment scores, questionnaire status)?
3. Does your board-level reporting connect vendor risk to specific financial or operational impact scenarios?
4. Have you ever had a vendor fail or suffer a breach who didn't appear on your watchlist beforehand? If yes: why not?
5. If you had to cut your entire risk dashboard down to five metrics, which five survive?

That last question is harder than it sounds. Most programs would struggle to answer it. That struggle is diagnostic.

Building a Program That Measures What Actually Matters

The goal isn't fewer dashboards. It's better questions.

Start with outcomes, not activities. Define what failure looks like for your organization — a significant operational disruption, a data breach, a regulatory action, a financially devastating third-party collapse — and work backwards to identify which vendor behaviors and characteristics are the strongest predictors of those outcomes.

Then build your metrics around those predictors. Questionnaire completion rates may still belong in your operational reporting. Fine. But when your CISO walks into the board meeting, the story should be about real exposure, concentration risk, and the specific vendors that could hurt the business in ways that matter to the people in that room.

That's not a reporting exercise. It's a risk management program.

The difference between the two? One keeps everyone busy. The other keeps the company safer.

DON'T MISS AN EPISODE!

Subscribe to Third Party on YouTube, the podcast for people who don't need to ask ChatGPT what TPCRM means. New episodes every other week.

Next time on Third Party:

We’re moving from measuring risk to predicting it: the early warning signs that a vendor is headed for financial or cyber collapse and how the best teams spot trouble months before everyone else. If you want to get ahead of incidents instead of reacting to them, you will not want to miss this one.

Subscribe below.