Machine learning: When prediction becomes unpredictable

What information security risks can erode by Machine Learning?

Information security risks can erode value added by ML

Such positive potential promotes the transition of ML from a laboratory curiosity to a practical technology that is widely commercialized and deeply embedded in enterprise information systems. While the practical focus in recent years has tended to be on rapid development and the potential positive efficiency and innovation effects, it is now becoming increasingly clear that ML also poses significant risks (Heinrich et al., 2020, Rosenberg et al., 2021). These risks range from the data source, data protection and data sovereignty to adversarial attacks on the ML model itself. According to a recent report by Gartner, 30% of cyberattacks in 2022 will be aimed at data poisoning (manipulation of the training data set), ML model theft or manipulation of input data with the intention of misclassifying a particular input, for example.

This assessment is not merely speculation, as demonstrated by known attacks on ML applications (e.g. the Twitter chatbot scandal in 2016). These attacks are particularly significant when they target critical ML applications (e.g. processing of personal data), which are increasingly used by financial institutions, too. It is with good reason that the European Commission has drafted a law to regulate the use of ML and protect users’ data.

The legislative initiative envisages four classes of AI applications:

• Unacceptable risk: AI applications that pose a threat to the safety, livelihoods and rights of EU citizens are to be banned.
• High risk: AI applications that have adverse effects on people’s safety and fundamental rights are considered high-risk. A set of binding requirements (including a compliance and risk assessment) is to be introduced for these high-risk systems (regulation).
• Limited risk: these AI applications are to be subject to a limited number of obligations (e.g. transparency).
• Minimal risk: for all other AI applications, development and deployment in the EU is to be enabled without additional legal obligations beyond existing legislation.

It will be particularly interesting for financial institutions if, under the current draft legislation, ML applications such as credit assessment of credit scoring are classified as high-risk and risk-reducing transparency and security obligations are demanded. In serious cases, violations can result in fines of up to EUR 30 million or 6% of the company’s global turnover.

AI systems must be made fit for the future to open up new potential:
security & transparency

Based on current cybersecurity threats, increasingly sophisticated attack methods and tightening regulatory requirements, raising awareness of ML security issues is essential. Even today, the information security risk to ML applications is real, and practitioners are facing this challenge.

In order to master these problems in the long term and achieve a strategic competitive advantage through the use of modern ML systems, companies should start building up the necessary skills and implementing an AI risk strategy now. In this context, companies should identify …

1. who can attack their systems and how (threat situation),
2. how resilient their systems are against such a threat and
3. if the resulting risk can be accepted or if risk-minimizing measures must be taken.

Secure and transparent are the key words in the context of ML. Transparency means that ML-based decisions must be traceable. ML applications must not act as a completely inscrutable black box. Moderate, application-based regulation and a thoughtful selection and combination of application components should address this transparency goal. In order to achieve a desired level of security for ML applications, there are various additional options which are indicated in the following overview.