The Security ratings value debate

Cyber pros have debated the value of cyber intelligence collected to identify potential risks within a third party.

Jonathan Dambrot

Jonathan Dambrot

Principal, Cyber Security Services, KPMG U.S.

+1 908-361-6438

Over the last decade, cyber pros have debated the value of cyber intelligence collected through open-source, dark web, domain sinkholes, and additional sources to identify potential risks within a third party. Startups like SecurityScorecard, BitSight, and RiskRecon have built businesses around the promise of offering visibility, without using more exhaustive assessment techniques. While the value of these technologies continues to be debated by CISOs and 3rd party security professionals, security ratings services have become part of the standard 3rd party security toolbox. The question is how to define the proper use cases and identify the benefits aligned to these technologies.

I have heard 3rd party security professionals want to use these capabilities as their sole source of truth in third-party security assessment. On the other hand, I have listened to others that will not use this data at all. I believe the real value of these services resides within three core use cases: On-boarding and Request for Proposal (RFP) Cyber hygiene indicator, 3rd party cyber indicator monitoring, and Cyber reputational monitor.

As an on-boarding and RFP tool, utilizing cybersecurity ratings makes sense if the goal is to understand the potential cyber risks before engaging with a provider. Before inviting to bid on an RFP, leveraging ratings services can ensure a basic gating for cyber hygiene, if the use and scope of the vendor services are unknown and/or there is an interest in understanding cyber posture from an external perspective. Building visibility into front-end legal and procurement processes can support the use of these tools without impacting business user and cyber professional demands.

It is key for us to have a thorough understanding of the limitations of these services. For instance, the services cannot address many of the cyber control requirements in a security addendum of the contract between you and the third-party. When organizations stretch the use cases for these services it can sometimes misalign what the security ratings service is using to identify cyber risk and the requirements of the security organization in contract.

As an on-going monitoring solution, ratings services are being used to monitor the cyber hygiene between assessments. Many organizations set standard thresholds to trigger alerts when a vendor security rating falls by a certain percentage (I have seen between 5 and 15% as the typical range). This can be useful, but in some cases organizations have a difficult time ingesting the additional notifications and specific asks for vendors to remediate. This can be based on the lack of specificity or lack of transparency in what is creating a reduced score in some cases.

Lastly, organizations are using the ratings services to monitor their own organizational security posture. This helps to better understand what clients are viewing when they purchase a report (please see the section below on the security rating reputation trap) and fundamentally improve the areas that are being reported. The primary challenge here is that the areas being monitored may not represent the entire scope of cyber elements that make up your security program. Typically, security ratings data would be one of multiple mechanisms being used to identify effective cyber program and framework maturity.

Why can't it be like a credit score?

There have been efforts to improve both the efficacy and consistency of cyber information across ratings providers with the goal of making them similar to credit scores. However, these efforts have generally been unsuccessful, and every vendor has introduced different data elements as well as risk models and scoring. The lack of ratings standardization has led to a situation where the different ratings provider data cannot be normalized or compared. Credit scoring is generally on a scale of zero of 850, I have not seen consistency across cyber ratings with scores across providers ranging widely.

Although, the lack of scoring normalization can create challenges, the other major issue is that the data source is generally not primarily sourced. Unlike credit data, which is pooled and collected as primary data from banks and other creditor reporting sources on an individual basis, the security ratings information is based on data derived from secondary sources. For instance, if a sinkholed (or takeover) domain is still communicating with known IP addresses that are correlated with a known vendor IP address, this will generally indicate a potential cyber issue. While this could indicate a problem, the challenge is that it is unknown (in many cases) what type of network or device type it may be communicating with. Bots communicating with servers in a core data center serving thousands of clients will have a significantly different exposure to a bot communicating with a guest device on a guest network. While these are simple examples, it illustrates the difficulty using security ratings like a credit score. Conversely, if lenders were unsure whether the credit data making up a credit score was specific to the applicant, they would face a hard time ensuring lending decisions were sound or fair.

Will it meet my regulatory requirements?

I believe regulators are mixed on the use of security rating services. Most regulatory coverage for third-party security is focused on ensuring, there is a way to align the risks posed by the vendor's use with the firm's appetite for risk and the needs of the industry. For those industries that have identified third party risk as a potential systemic issue, like financial services organizations, there is no prescribed approach, but regulators generally provide guidelines outlining the need to perform assessment as well as ongoing monitoring. This offers flexibility to the regulated entity, allowing for decisions based on their own business requirements and specific situation.

Based on the third party security programs I have worked with, it seems that the current best practices promote regular assessments informed by the scope of the relationship and security ratings as a component of the ongoing monitoring between assessments. Obviously, programs should be aligned closely to regulatory guidance and the advice of both internal and external stakeholders.

The security ratings reputation trap.

One of the initial challenges with security ratings is the complaint that clients will look at the security ratings report as part of their process, but the third party may not be aware of what is being published or worse that the data is inaccurate. I have seen this in a couple of instances where cloud providers, especially Platform as a Service providers could be hindered by the platform's clients and misrepresent the risk of using the provider. In these situations, when the provider looks to review their report from the security ratings provider, they are told that it is not freely available and that they must pay to monitor themselves. They are also told that they should advise and correct the data that is being published to clear up any issues. While there should be a mechanism to update inaccurate security rating data (and almost all of the current providers do this), some providers complain about paying for a report that is being shared with others without their consent that requires them to fix. This creates a potential trap for providers to either pay to monitor themselves or potentially have reputational issues for data that is not accurate being shared with clients.

The US Chamber of Commerce (in conjunction with some of the ratings services) published guidance several years ago* to help alleviate some of these issues, but I still hear some of the same complaints today. In my opinion, there is value in monitoring yourself from both a reputational perspective and better understand what the external security view (or hacker) view might look like.

The next evolution of security ratings.

As organizations, regulators, and security professionals continue to use these services and technologies, it is clear that the ratings services cannot fundamentally solve all 3rd party security use cases by themselves. In my view, the promise and scale of security ratings will not be met until there is broader adoption of a scoring model that can be agreed upon and understood universally; the data becomes more uniform and dependable; and the cost of the data becomes similar to credit reporting and accessible to all size organizations. I believe investments in third-party security, evolving use of utilities and consortiums, and the inevitable commoditization of these services will lead to major changes in how consumable and valuable these services become in the future.