Machine Learning and AI in Model Risk Management: A Quantitative Perspective – Risk.net

Statistical risk models face validity issues when unprecedented social events and phenomena occur. However, artificial intelligence (AI) and machine learning can help models maximize accuracy. By Tiziano Bellini, Head of the Risk Integration Competency Line, International Markets, at Prometeia

Model risk is generally interpreted as the potential loss that an institution may incur as a result of decisions based on the results of models. Errors can occur throughout the life cycle of the model and can come from two main and general sources: specification errors and calibration. Financial institutions rely on an increasing number of models to inform their day-to-day decision-making processes. These models generally attempt to mimic economic processes and are deeply influenced by social phenomena. Consequently, sudden and profound macro and microeconomic changes undermine the effectiveness of the model, mimicking the dynamics of the economic environment. This raises the following questions: is the model able to function effectively outside the context in which it was trained? Do machine learning and AI help quantify model risk?

The unprecedented Covid-19 pandemic and energy crisis have threatened model developers and validators, as traditional statistical models fail to capture unseen events in the past. In addition, climate change and environmental stress tests are jeopardizing the credibility of some financial models.

In all these cases, the availability of data plays a key role. What is the plausibility of models based on rare historical data? How could statistical models help in case of unprecedented phenomena? Students of econometrics agree that “the past is not the future”. Nevertheless, it is common to look back on history to grasp what will happen next. One may be tempted to discard statistical models when completely new economic conditions take hold. Alternatively, one can think of improving the use of the models by improving their economic foundations. The integration of statistical modeling and economic theory paves the way for a new generation of model management, where it is paramount to assess how a model performs beyond the “environment” in which it was form.

We can think of relying solely on historical observations to assess the uncertainty of a model, for example via bootstrap techniques. A major advantage of this approach in estimating model risk is to rely on inputs and outputs that are already available. On the contrary, the main disadvantage is that these inputs and results are limited to past events. While aiming to assess uncertainty beyond the conditions under which a model was developed, the crucial answer is to test the model under new circumstances. How to approach such a problem?

A method would be needed to consistently simulate the inputs as well as the outputs of the model. Some alternatives are available in the statistical literature. Techniques such as generative adversarial networks or the like may be the maximum realization. Prometeia has implemented solutions to derive error distributions based on various machine learning and AI approaches. Therefore, Prometeia has designed risk appetite frameworks and model monitoring processes by identifying extreme events causing breaches in the model’s risk tolerance. Such approaches have also been applied in cases of what-if scenarios (such as climate risk) where historical data is not strong enough to support effective projections.

The ability to perform analyzes based on simulated contexts where historical structures (correlation and interdependencies, among others) are used or tested constitutes a new paradigm for model developers and validators.

machine learning and AI have been widely used in the recent past due to big data. Their statistical properties now become crucial to support processes where historical data is not necessarily representative of future parameters. Prometeia is a pioneer with applications in the field of model risk and, more generally, in the field of risk, having recently been recognized as a world leader in the Quantification of model risks and Capital optimization categories in the Chartis 2022 RiskTech100® rankings.