Bob Maley, CSO, Black Kite

CISOs tend to loathe security rating services (SRS). They’re notorious for creating a false sense of security in risk, being pushed into the space by big customers or their boards who don’t understand the underlying methods. Many, including the bandwagon providers themselves refer to them as an “easy button”, building around a poor foundation tailored to a market that’s seeking a quick fix to their cyber hygiene score.

As CSO of a cyber ratings company, why on earth would I be addressing this? The answer is simple: not all SRS providers are built the same. The right questions are not being asked that would inform SRS users toward more effective decision-making in choosing a service that can provide them with more complete, transparent, and accurate information.

In 2017, the U.S. Chamber of Commerce published its brief on Principles for Fair and Accurate Security Ratings based on industry stakeholder agreement for the need to set and follow the principle of fair and accurate rating services.[1] While the Black Kite platform itself was built around these principles, they can also be used as a guide. Identifying the right SRS requires asking the right questions during assessment.

The Right Questions to Ask Before Choosing an SRS Provider

  1. What level of transparency are you comfortable with?

    You should be able to find a SRS that is transparent to the limits of your needs and that includes the need to have defensible methods documented for your program for all your stakeholders.

    An SRS provider that is transparent about using proprietary methods is not transparent about what quantitative analysis is taking place behind the scenes. Are you willing to accept the proprietary method? Or, do you need something completely transparent.

  2. How can you achieve the highest level of accuracy for the best end results?

    Remember that on a smaller footprint,  accuracy has less value. Accuracy on a small number of controls ignores too much of the attack surface. So, being accurate is not the same thing as being useful.

    In fact, services may reduce the total digital footprint in order to improve their accuracy – in other words, they can be more accurate on a smaller footprint. By reducing the footprint to improve accuracy, the number of controls being evaluated is no longer an apples-to-apples comparison and it doesn’t provide a complete picture. In addition, false positives can lead you to reach out to a vendor to mitigate a vulnerability that does not exist.

  3. How are assets classified?

    Weighting without transparency is, at best, a security theater. Asset classification does not improve your view into risk, it masks serious problems, as seen in SolarWinds, Target, and other catastrophic breaches.

  4. How does that classification method impact your ability to show value to management?

    How do you want your board to understand your third-party risk – qualitatively or quantitatively? Do you want to talk about risk in business terms or in technical terms?

    When you are talking with business units, the technical aspects of risk management tend to be an obstacle because it is outside their focus. However, if you can go to your business unit and tell them that the impact of a particular vendor will be a specific dollar amount, then you put them in a position that can better frame their decision making around vendors.

  5. How does the solution help reduce your churn?

    Determine what the basis is of your triage process for determining which vendors you select to examine. If your triage is classification-based vs. risk-based, then your process (and your SRS) will not reduce uncertainty around risk assessment. You must reduce uncertainty because everyone has limited resources.

  6. Will the platform automate your process and, if so, will that add value to my program?

    Automating a process that is inherently flawed does not reduce your effort. Automating questionnaire management may not add value.

    As Venables said, the ability to ingest existing artifacts like questionnaires and infosec policies automatically is valuable. Your SRS should analyze the relevancy of that vendor to your compliance frameworks to provide feedback into your cyber hygiene picture and your overall program.

  7. What is the unit of measurement?

    As Jack Jones (creator of Open FAIR™) likes to say: There’s certainly a ton of confusion about what is and isn’t quantification. Traditional SRS providers often claim their rating scales constitute quantification, as do others. This isn’t just misleading, it’s patently wrong, and it gives people a false sense of security regarding the methods and results used to calculate the final amount.

    You’ve heard me say it before — if there isn’t a unit of measurement (%, frequency, time, $$, etc.), then it isn’t quantification — period. Buyers and users should ask vendors who claim to do quantification to describe their units of measurement. If they can’t, and especially if they’re confused by the question, then you can be certain they aren’t quantifying anything.

The SRS “Easy Button” isn’t Always the Easiest to Implement

Ultimately, the market can demand more of its SRS providers. Your voice as a consumer for your organization can resonate alongside regulatory, compliance, insurance, and other pressures in support of the key principles that mean:

  • Your scores and related data will be more accurate; 
  • Scores are provided with transparency that you understand; and 
  • Then that information can be used more wisely – in the context of a body of other due diligence evidence around that vendor’s security posture and the type of service they provide for your organization.

Your SRS should help you capitalize on the hundreds and thousands of assessments, reports, policies and other artifacts that are part of your due diligence process. An SRS score should help you build on these components that already live in your system.

See What Makes Black Kite Different

References

[1] Principles for Fair and Accurate Security Ratings. 2017. US Chamber of Commerce. https://www.uschamber.com/issue-brief/principles-fair-and-accurate-security-ratings