Deep Learning Models for Anomaly Detection in Cybersecurity

Anomalies, often referred to as outliers, abnormalities, rare events, or deviants, are data points or patterns in data that do not conform to a notion of normal behavior. Anomaly detection, then, is the task of finding those patterns in data that do not adhere to expected norms, given previous observations. 

 

The capability to recognize or detect anomalous behavior can provide highly useful insights across industries. Flagging unusual cases or enacting a planned response when they occur can save businesses time, costs, and customers. Hence, anomaly detection has found diverse applications in a variety of domains, including IT analytics, network intrusion analytics, medical diagnostics, financial fraud protection, manufacturing quality control, marketing and social media analytics, and more.

 

The purpose of  Deep learning models for anomaly detection will be clarified in this post.

 

Applications for Detecting Anomalies

 

Before delving into several ways to anomaly detection in the following chapter, we'll start by taking a closer look at some potential use cases. 

 

The detection of network intrusions

 

The operation of a contemporary, successful organisation depends on network security, but all computer systems include security flaws that, once exploited, are both technically challenging and expensive to fix. Business IT systems gather information about their own network traffic, system usage, connection requests, and other topics. Although the majority of activity will be innocuous and normal, this data analysis may reveal information about odd (anomalous) activity occurring within the network after and ideally before a significant attack. 

 

In reality, the damage and expense incurred immediately after an infiltration incident rise quicker than most teams can construct a successful defence. It is therefore crucial to have specialised intrusion detection systems (IDSs) in place that can detect unusual probing and potential threat events early and reliably.

 

Health diagnosis

 

A range of data points (such as X-rays, MRIs, and ECGs) indicative of health condition are gathered as part of the diagnostic processes in many medical diagnosis applications. Medical equipment used by patients themselves, such as glucose monitors, pacemakers, and smart watches, also gather some of these data points. To draw attention to instances of aberrant readings that might be a symptom of a medical occurrence or a warning sign for a health condition, anomaly detection techniques can be used.

 

Detection of Fraud

 

The estimated global financial cost of fraud in 2018 was approximately £3.89 trillion, or $5 trillion USD. It is crucial for service providers to accurately and promptly detect and respond to fraudulent transactions in the financial services sector. In the simplest situations, it is possible to determine whether a transaction is fraudulent by comparing it to previous transactions made by a particular party or to all other transactions made by peers within the same time period. In this situation, fraud can be viewed as a departure from typical transaction data and dealt with utilising anomaly detection techniques.

 

Even if there are several types of financial fraud, such as card-based, check-based, unauthorised account access-based, or authorised payment-based fraud, the fundamental ideas of analysing someone's typical behaviour and watching for signs of odd activity still hold true.

 

Detection of Manufacturing Defects 

 

Quality assurance (QA) in the manufacturing sector depends on an automated method for finding flaws, especially in products produced in huge quantities. The objective of this work, which may be seen as an anomaly identification exercise, is to find manufactured goods that significantly or even slightly deviate from ideal goods that have passed QA tests.

 

Along with industry and regulatory standards, the company's consumers and the company itself decide how much divergence is allowed. Predictive maintenance can also be viewed as an anomaly detection problem. Data from machine sensors, such as vibrations, temperature, drift, and more, can be used as an example for using anomaly detection algorithms. Abnormal sensor readings may be a sign of future failures.

 

These examples show that anomaly detection is helpful in many different contexts. Finding previously undiscovered phenomena and appropriately identifying them as anomalous is a difficult topic that has been approached in a variety of methods over the years. Although there are various methods, typical machine learning (ML) algorithms perform poorly when dealing with high-dimensional data and sequence datasets because they are unable to recognise the intricate data structures.

 

Methods for Detecting Anomalies

 

Based on the kind of data required to train the model, anomaly detection techniques can be divided into different categories. In the majority of use cases, a very tiny portion of the whole dataset is anticipated to be made up of anomalous samples. Therefore, normal data samples are easier to find than aberrant samples even when labelled data is available. For the majority of applications today, this presumption is crucial. We discuss how the choice of strategy is impacted by the availability of labelled data in the sections that follow.

 

Learning Under Supervision

 

Machines learn a function that maps input features to outputs based on sample input-output pairings while they are learning under supervision. Adopting application-specific knowledge into the process of anomaly detection is the aim of supervised anomaly detection algorithms.

 

The challenge of anomaly detection can be reframed as a classification task with enough normal and anomalous instances so that computers can learn to correctly anticipate whether a particular example is an abnormality or not. However, for many anomaly detection use cases, the ratio of normal to abnormal instances is severely skewed; even while there may be several classes of anomalies, each one may be significantly underrepresented.

 

This method implies that the user can accurately classify all possible anomalies and has labelled examples for each kind. As abnormalities can manifest in a variety of ways and new anomalies can arise during testing, this is typically not the case in practise. Therefore, methods that generalise well and are better at spotting anomalies that haven't been seen before are preferred.

 

Unsupervised Education

 

Machines cannot learn a function that translates input features to outputs using unsupervised learning because they lack examples of input-output pairings. Instead, they discover structure within the input features and use that to learn. Unsupervised methods are more widely used in the field of anomaly detection than supervised ones because, as was already said, labelled anomalous data is comparatively uncommon.

 

However, the type of anomalies one expects to find is frequently very particular. As a result, many of the abnormalities discovered in an entirely unsupervised approach may simply be noise and may not be relevant to the task at hand.

 

Semi-supervised Education

 

Semi-supervised learning strategies use a variety of techniques that can benefit from both huge amounts of unlabeled data and sparsely labelled data, acting as a type of middle ground. Due to the abundance of normal instances from which to learn and the dearth of examples of the more unusual or abnormal classes of interest, many real-world anomaly detection use cases are well suited to semi-supervised learning. 

 

Given the presumption that the majority of the data points in an unlabeled dataset are normal, a robust model can be trained on an unlabeled dataset, and its performance can be assessed (along with the parameters of the model) using a small quantity of labelled data.

 

Applications like network intrusion detection, where there may be several examples of the normal class and a few examples of intrusion classes, but new types of intrusions may develop over time, are ideally suited for this hybrid technique. Consider X-ray screening for border or airport security as another illustration. Unusual products that pose a security danger are uncommon and can take many different shapes. Additionally, any anomaly that poses a potential hazard may change in nature as a result of a variety of outside events. Therefore, it may be challenging to get sufficient quantities of useful examples of anomaly data.

 

The identification of novel classes and abnormal classes, for which there is little to no labelled data, may be necessary in such circumstances. In these situations, a semi-supervised classification method that can identify both known and undiscovered anomalies is the best choice.

 

Model Evaluation: Accuracy Is Insufficient

 

As was noted in the preceding section, it is anticipated that the distribution between the normal and abnormal class(es) in anomaly detection applications may be extremely skewed. This is sometimes referred to as the problem of the class imbalance. A model that gains knowledge from such skewed data might not be reliable; it might identify samples correctly when they belong to the normal class but incorrectly when they belong to the anomalous class.

 

Think of a dataset with 1,000 pictures of bags going through a security checkpoint, for instance. 50 of the 950 photographs are of unusual items of luggage. 

 

A similar approach might categorise abnormal cases as normal (false positives, FP), or false negatives, FN), depending on the situation. It becomes clear that the standard accuracy metric (total number of right classifications divided by total classifications) is inadequate in assessing the quality of an anomaly detection model when we take into account both of these sorts of errors.

 

Precision and recall are two significant metrics that have been proposed to help assess model competence. Recall is calculated by dividing the total number of true positives by the total number of true positives plus the number of false positives, whereas precision is calculated by dividing the total number of true positives by the total number of true positives plus the number of false negatives. You now understand the advantages of an unsupervised or semi-supervised approach to anomaly detection as well as the ideal criteria for assessing these models. We concentrate on semi-supervised methods in the next part and go over their operation.

 

Security

 

A challenge of the "different is bad" sort is presented by home or business security systems trained to spot abnormalities, which is similar to the suitcase example stated at the beginning of the chapter. To avoid bias that would make these systems more likely to label individuals of different races, body types, or socioeconomic position as anomalies based on, for example, their skin colour, size, or attire, these systems must contain enough data, both in terms of quantity and variability.

 

Content Control

 

Uncritical reliance on anomaly detection technologies for content moderation might result in false positives that censor or silence system users depending on their language or device preferences. Software for content moderation should have a user appeal/review procedure and be watched for trends of inappropriate blocking or reporting.

 

Open Source Frameworks and Tools

 

The implementations of algorithmic techniques that can be used for anomaly detection jobs are included in a number of well-known open source machine learning libraries and packages in Python and R. Anomaly detection is not a focus of general-purpose frameworks like scikit-learn, but they do contain useful methods (such as clustering, OCSVMs, and isolation forests). Anomalies are recognised based on the difference between the true value and a forecast, hence general tools for univariate time series forecasting (like Facebook's Prophet) have been widely deployed to these jobs. This section focuses on thorough toolkits that are specifically designed to handle the task of anomaly identification.

 

Outlier Detection in Python (PyOD)

 

In order to do scalable outlier identification on multivariate data, PyOD is an open source Python toolbox. Under a single, well-documented API, it offers access to a variety of outlier detection algorithms, including well-established outlier ensembles and more contemporary neural network-based methods.

 

PyOD has several noteworthy benefits, including:

 

• More than 20 algorithms are accessible, ranging from traditional methods like the local outlier factor (LOF) to modern neural network structures like autoencoders and adversarial models.

• It uses combination methods to combine the findings of various detectors with outlier ensembles, a new class of models.

• For simplicity and ease of use, it has a consistent API, thorough documentation, and interactive examples for all algorithms.

• With cross-platform continuous integration, code coverage, and maintainability checks, all models are covered by unit testing.

• When practical, optimisation tools are used; for scalable outlier detection, just-in-time (JIT) compilation and parallelization are enabled in some models.

• It works with Python 2 and Python 3 on all major operating systems.

 

Conclusion

 

The challenge of anomaly detection is a well-known one that affects many commercial disciplines. In this research, we have investigated the semi-supervised application of a number of deep learning algorithms to this job. For the following reasons, we believe this concentration is beneficial:

 

• Although deep learning has proven to be more effective than other approaches for a variety of commercial activities, nothing is known about how deep models do when applied to anomaly detection.

• When handling previously undiscovered anomalies, a semi-supervised technique is preferable since it spares organisations from having to pay astronomical data labelling fees.

• The management of high-dimensional, complicated data and the modelling of relationships between each variable are not well suited for traditional machine learning algorithms.