Apple CEO Tim Cook introduces Apple Card during a launch event at Apple headquarters on March 25, ... [+]
This weekend, tech entrepreneur David Heinmeier Hansson went on Twitter to say that the new Apple Card provided him a credit limit that was 20 times higher than his wife’s. This was even though she actually had a higher credit rating. But this was not a one-off. Apple co-founder Steve Wozniak had a similar experience with his wife!
Yes, when it comes to sophisticated algorithms and AI (Artificial Intelligence), even some of the world’s most valuable companies can get things wrong. What’s interesting is that one of the selling points of the Apple Card was that it would provide credit to those with little or no credit histories (although, Goldman Sachs did put out a statement that indicated that each person’s application is evaluated separately and the results may different among family members).
Regardless, this episode does point to the inherent risks and complexities of AI. Keep in mind that many other companies have had challenges with the technology, such as Google, Microsoft and Facebook.
So then, what are some of the takeaways with the Apple Card? Let’s take a look:
Krishna Gade, who is the founder and CEO of Fiddler Labs:
“Companies like Apple and Goldman Sachs have a moral obligation to test their products for bias before launching them publicly. They need a better governance process and tools with human oversight over algorithms. For example, was Goldman Sachs validating credit limit differences across gender while keeping other input factors—income, state and assets—the same, to check for bias? Customer representatives should have been empowered to answer questions on ‘Why is my credit limit so low?’ When they respond with answers like ‘It's just the Algorithm,’ it deteriorates the company's trust with customers. Finally, there should be a feedback loop in the algorithm training process to ensure these mistakes are not repeated.”
Chris Nicholson, who is the CEO of Skymind:
“Here’s the thing most people don’t understand: Certain kinds of advanced algorithms, machine-learning algorithms, need a lot of data to train on in order to make predictions. We turn to the past to find that data. But if you’re not careful, the algorithms learn the mistakes of the past. In this case, that would be gender bias. Algorithms that learn from history are doomed to repeat it. That’s the great irony. You have to really work to correct for the mistakes of the past. In life and in algorithms.”
Atif Kureishy, who is the Global VP of Emerging Practices - Artificial Intelligence & Deep Learning at Teradata:
“We need to appreciate the importance of transparency when using AI/ML to predict underwriting that impacts credit lending. Quantifying predictive power is well understood, for example, in terms of a metric like the confusion matrix. Interpretability is less well understood and is in general difficult. When we speak of understanding a model, we need to clarify if we want to understand the model’s overall behavior (global interpretability) or if we want to understand why a specific prediction was made (local interpretability).
“Ultimately, Apple and Goldman Sachs did not emphasize model risk management principles of interpretability. Clearly, their support teams deferred to the ‘algorithm’ for making the determination of credit limit, but hopefully they’ll learn from their mistakes. Most likely, the gender bias was introduced from the data they sourced either internally or externally, but the reputational risk of their credit card will suffer.”
Stuart Dobbie, who is the VP of Product at Callsign:
“The first key process is understanding your dataset. Previous algorithm failures in other industries—insurance and e-Commerce, for example—have occurred due to the predominant bias found in the gender and socioeconomic status of their training datasets. It is important to understand how bias may be found naturally in your own customer base, and how this may affect your inferences, since these naturally need checks and balances to avoid sampling bias. Second, for real-time automated decision systems, it would be wise to create circuit breakers. These will raise alerts when a propensity of the decisions or distribution of outcomes becomes heavily skewed beyond some fail-safe values. This is a good proactive measure to monitor performance and expectations of algorithm decision making.
“AI responsibility is two-fold. We’re looking at the strict implementation of governance processes fortified by a culture of open communication and scrutiny. Data governance best practices are critical, but players implementing AI from across the business ecosystem need to also be vigilant about engaging a diverse range of internal contributors and stakeholders. This could be sharing private and public proofs of concept, working with the academic and research communities to participate in peer reviews, or even involving consenting members of the public to help validate their AI models. “
Tom (@ttaulli) is the author of the book, Artificial Intelligence Basics: A Non-Technical Introduction.
