Speed to Modern Tech Podcast Series, Episode 4

On this episode we explore how AI bias can perpetuate “redlining,” and how adopting better governance can help build more responsible AI.

While many think of computers as accurate, neutral, unemotional and purely logical machines, the human-designed AI algorithms they run are susceptible to error and bias, especially if the data they rely on is incomplete or contains inherent biases.

Financial services companies are subject to a significant number of regulatory requirements and compliance mandates. Avoiding unintended biases with age or race in decisions about extending credit or determining credit limits, for example, is key to meeting those mandates. Investment decisions, of course, can also have significant financial consequences. Therefore, it’s essential to have confidence in any decision that AI makes by ensuring that such decisions are free from error or bias.

Understanding, detecting and preventing such errors or biases, however, can be remarkably challenging. There’s often a lack of transparency into how AI is making its decisions — indeed, one of the reasons we turn to AI is precisely because it can detect patterns amid chaos that humans are incapable of seeing or even understanding.

Responsible AI is the practice of bringing transparency, accountability and trust to AI. Our goal was to help our client stand up an effective and efficient responsible AI program, powered by the technology required to validate its AI/ML models and data to support longer term scale.

The challenge in this case was that each of the institution’s five separate business units had already implemented their own AI/ML solutions. But without any formal or centralized responsible AI program, or technology specifically designed to detect AI/ML model or data errors or biases, the client was at risk.

It combines our team of highly skilled AI technology and data science professionals, our advanced and proprietary AI tools and accelerators, our extensive experience with leading AI solutions, our strong industry alliances, and our long-standing experience in governance, risk and compliance (GRC) to form a remarkably broad solution. It’s designed to help organizations stand up a responsible AI program and build and evaluate sound AI and ML models to establish trust in the output of those models.

Beyond the technology used to validate AI/ML models and data, successful responsible AI programs require well-designed operating models and processes that reflect leading GRC practices. We helped define key performance indicators (KPIs) and other metrics that would provide benchmarks for AI/ML model testing and validation.

We also helped stand up an independent AI/ML operations function within the organization. In the same way a firm can’t audit itself, or a software engineer can’t validate that his or her own code is free of bugs or vulnerabilities, an independent validator is required to handle AI-related technologies and deployments.

This included custom Python scripts, open source libraries and other tools designed for this purpose.

We worked with the client’s preferred AI/ML platform solution. This cloud-based AI workbench is designed to help data scientists and developers prepare, build, train and deploy high-quality machine learning models quickly by bringing together a broad set of purpose-built capabilities. However, by default, it’s not configured with any validation capabilities.

We helped develop the scripts for the platform that would enable the client to validate any AI or ML algorithms and the data on which they depend — essentially, “built-in” validation instead of using a separate add-on solution that typically evaluates only the output of those algorithms.

Our validation customizations were designed to help the firm quickly and easily detect, diagnose and correct problems that can arise with AI/ML models, including both internally and externally developed and trained models. It enabled our client to experiment with its models — providing “levers” they can pull to conduct what-if experiments to help determine the effect various parameters or assumptions have on the models that rely on them, and ultimately a way to quantify errors or biases in those models.

We also reverse-engineered several proprietary third-party models to help our client gain insights into those models that were otherwise a black box.

Our team consisted of three data science and analytics professionals skilled in the development of AI models, three risk and compliance professionals with specific knowledge and experience in financial services, and a project lead.

One of the challenges with implementing a responsible AI program is that evaluating AI systems for error or bias often requires more technical skills and experience with AI than were required to develop the AI solutions in the first place. It goes beyond the skills required for many other governance, risk and compliance (GRC) efforts, and includes the ability not only to understand what AI systems are doing, but also to reengineer them when the algorithms are proprietary and therefore inaccessible for direct examination.