Snowflake Connect: AI on January 27
Unlock the full potential of data and AI with Snowflake’s latest innovations.
To understand anomaly detection, you must first understand what an anomaly is, and why anomalies are important. An anomaly is anything unexpected or out of the ordinary, a statistical aberration that makes anyone looking at the data curious about why the data might be so unusual. Anomalies can indicate anything from a random chance result to a failing system to malicious fraud.
While it’s easy for us humans to spot an anomaly in a small data set, as these data sets grow to millions or billions of data points, manual oversight and analysis of this information becomes largely impossible, truly becoming a search for a needle in a haystack. As such, the field of anomaly detection has risen in prominence and is now an essential part of the larger discipline of data analysis, using AI to uncover outliers in a data set so organizations can detect issues early and maintain operational quality, preventing small problems from turning into big ones.
Anomaly detection is the identification of data observations that deviate significantly from what is expected. In manufacturing, this could be an abnormally high temperature reading on a production machine on the factory floor. Or it could be a credit card charge processed on your ecommerce website that is much higher than the usual sale amount. Either way, these represent reasons to investigate to determine why the anomaly has occurred. (Is the outsized credit card charge from a lucky lottery winner or a malicious scammer?)
Modern anomaly detection is increasingly driven by AI and machine learning to automatically detect these events, as manual inspection is not practical or efficient in most cases. These automation-driven techniques help organizations determine not just whether an anomaly has occurred, but more importantly why it has occurred — and to what degree that is a cause for alarm.
While an anomaly can be any type of unexpected event, not all anomalies are created equal. Experts generally recognize five different types of anomalies. Note that an anomaly can fit into more than one of these categories, as we’ll outline below:
In the broadest sense, unintentional anomalies are accidental or random in nature. These are non-malicious events that can be caused by a simple, one-time error (such as a part being misaligned in a processing machine) or a human mistake (failure to press the correct button at the right time). Unintentional anomalies are an inevitable part of doing business, and they can have any number of causes; however, this does not mean they should be ignored. Unintentional anomalies can potentially indicate failing equipment, bad programming or workers (literally) falling asleep on the job.
In contrast to unintentional anomalies, intentional anomalies are deliberate. Many intentional anomalies are malicious in nature. This is an especially large problem in the realm of cybersecurity, where intentional anomalies can represent attacks and other exploits. Not all intentional anomalies are bad, however. Seasonal sales spikes such as those on Black Friday or a flood of resumes arriving after a new job is posted would be intentional — and welcomed.
The following three anomaly categorizations can be either unintentional or intentional in nature.
Imagine a series of data points, such as a collection of machine temperature readings, arranged in order by time. (This is known as time-series data.) Point anomalies represent a single data point that resides outside the pattern set by the rest of the data set. In our example, that data may look like a flat, straight line, representing the machine running at an expected, constant temperature. A sudden spike and then a return to normal is a point anomaly. Point anomalies are also known as global outliers.
Data points that deviate from the expected pattern are not point anomalies when they vary within a specific context. For example, in our machine temperature example, if the machine is turned off for an hour each day during lunchtime and then restarted afterwards, a contextual anomaly may see the temperature suddenly rising in the few minutes after it is restarted before returning to normal. High temperatures recorded during this time would be contextually normal, but high temperatures many hours later would be contextual anomalies.
Collective anomalies are interesting because, when someone examines these data points on their own, they don’t look anomalous at all. Again, in our temperature example, imagine that a high and low threshold are well-established, and as long as all temperature readings are fluctuating within that threshold, operations are considered normal. However, if a series of data points in a row shows the temperature pegged at the high threshold, sustained over time and without the normal up-and-down fluctuations, these readings represent a collective anomaly that deserves investigation.
Anomaly detection is a core component of any business data analytics program. Business functions that rely on operational data — such as finance, manufacturing and quality control, cybersecurity and sales analytics — all need to have anomaly detection in their arsenal of tools. Effective use of anomaly detection is associated with:
How do you know your business has a problem? Anomaly detection is the early warning system that lets you uncover issues before they have a chance to spiral out of control.
Similarly, once an anomaly detection system uncovers outlying data, your organization has the chance to address it immediately. For example, a machine part that is showing signs of falling outside of acceptable tolerance levels can be taken out of service for preventative maintenance instead of waiting for the machine to fail completely.
Uncovering anomalies gives organizations the chance to address them. Teams can and should correct anomalies caused by noise or errors in data collection, so they can improve future data collection efforts and ensure accuracy.
Anomaly detection can be a key part of compliance efforts, especially in areas like finance, where fraud can be uncovered more quickly, and cybersecurity, where attempts at data theft can be identified.
While anomalies are obvious in simple time-series graphs, real business data is much more complex and nuanced, and anomalies do not necessarily stand out to the naked eye. Teams can use some of the following methods to surface anomalies:
Statistical testing pairs observed data with what a statistical analysis would expect to see. These tests can be fairly straightforward, such as comparing every measured data point against historical averages and standard deviations.
Additional statistical methods are popular for use with time-series data, which represents a large portion of the information used in anomaly detection. Tools such as Z-score analysis and auto regressive integrated moving average (ARIMA) analysis are used to measure the likelihood that data is anomalous when compared to historical averages and prediction models.
Deep learning is a branch of artificial intelligence that gives researchers the ability to delve into very complex and unstructured data sets. Deep learning often uses neural networks to predict future data points and identify when measured data deviates from those predictions.
API-based models are designed to monitor data streams for anomalous behavior in real time. These tools are trained on historical data but monitor a dynamic stream of information as they search for outliers. Operators are alerted immediately, and may receive suggestions for resolving anomalies.
Visualization methods involve turning raw data into charts and graphs, which makes it easier for analysts to spot outliers. Modern visualization tools can offer insights into much more complex data than a traditional two-dimensional graph. Virtual reality tools can even allow analysts to manipulate visualizations in three dimensions, allowing for even deeper analysis.
Engineers can use machine learning in several different ways to uncover anomalies. The three primary approaches comprise these methods:
As with most forms of unsupervised machine learning, here the model is trained on a vast amount of historical, unlabeled data and tasked with determining anomalies without additional data or context. The data most useful for anomaly detection systems are unlabeled, so this technique is the most used of this trio.
In supervised anomaly detection, algorithms are trained on historical, labeled data that includes normal and anomalous information, correctly tagged by human operators, from which the model can learn. Unfortunately, properly labeled data is rarely available in most anomaly detection use cases, making this methodology uncommon.
This methodology combines unsupervised and supervised techniques, wherein engineers label some of the training data but leave most of it unlabeled. This gives the algorithm some level of context upon which to base its analysis.
Anomaly detection has broad application across a wide range of industries and job functions. Here are some of the most common use cases.
The finance industry is one of the heaviest users of anomaly detection. Credit card processors use anomaly detection to monitor transactions for fraud, while stock traders use anomaly detection to predict when they should buy or sell equities.
Anomaly detection is central to the practice of network intrusion detection and threat intelligence. These systems monitor logs and network traffic in real-time, alerting operators or taking automated actions when unusual traffic patterns emerge.
Anomaly detection systems can monitor networking hardware and servers to determine if a device is at a high risk of failure or is becoming overloaded and in need of an upgrade. Anomaly detection can also help IT teams uncover errors or faults in their IT infrastructure.
Anomaly detection is commonly used to monitor manufacturing production lines. These tools can keep tabs on equipment in real time to ensure a machine is operating within appropriate tolerances, and they can be used to check that finished goods are arriving at the appropriate quality level. Production managers can quickly halt the production line if something goes awry.
When patients are connected to systems that monitor their vitals, anomaly detection determines if, for example, their heart rate or blood pressure is too high or too low. Anomaly detection can also be used on a macro level in healthcare, helping to pinpoint the source of contagious diseases and quickly analyze other global health crises.
Anomaly detection may sound simple, but it’s actually an advanced discipline that requires a significant amount of expertise to master. Here are some of the major challenges in the field.
As mentioned previously, most anomaly detection systems are trained on unlabeled data — because properly labeled data is rare. Organizations collect countless gigabytes of machine logs and transaction data, but very little of it is tagged to denote which readings are true outliers and which are noise (see below). Ultimately most anomaly detection systems have to be trained on unlabeled data, which necessarily results in less accurate results, at least at first.
A large data set will probably have thousands of outliers in it, but there’s no way to know whether any given outlier in that collection is cause for concern. That’s because most data sets are filled with what’s known as noise — random variation — much of which can look alarming to the untrained eye. Advanced statistical methods and data science are needed to determine whether an outlier is a true anomaly.
When thousands of outliers are unearthed in a data set, then what? Investigating each one of them can be incredibly time-consuming — and even impossible if the amount of data is too large. Similarly, the amount of compute resources needed to analyze data for anomalies, especially real-time data feeds, can be high and costly.
The definition of “normal” changes over time in most data sets. Prices rise. Machines wear down. Environmental conditions change. Operational personnel come and go. Any anomaly detection strategy needs to account for changing baselines like these, ensuring those baselines are adjusted properly to avoid false alarms.
Anomaly detection is a cornerstone of data integrity, security and operational reliability across industries in the modern enterprise, and a thoughtful implementation of anomaly detection techniques is a core component of any data monitoring strategy. That said, every organization tracks and uses different types of data from different sources, which means that one single tool or technique for anomaly detection won’t be ideal for every use case. Many organizations find it beneficial to experiment with different tools and techniques to find the best fit for their real-world challenges.
AI has become an essential component of most anomaly detection implementations. It improves on traditional statistical methods that determine which data points are normal and which are outliers and can make more complex associations with outlying data points and the potential root causes for them. AI is also an important part of creating scalable, real-time anomaly detection systems, which aren’t as accurate or robust when traditional methods are used.
Anomaly detection is a valuable pursuit whenever your business collects data which is prone to unusual or unexpected data instances. Use anomaly detection anytime you suspect these patterns could represent either business problems or potential opportunities. A wide range of business functions use anomaly detection, including finance, manufacturing, cybersecurity and more.
The terms anomaly detection and outlier detection are often used to mean the same thing. The difference is nuanced: Anomaly detection focuses on finding unusual events or sequences of events that deviate from expected behavior, while outlier detection refers specifically to the discovery of individual data points that represent a statistical difference from the rest of the data. In other words, anomaly detection is ultimately designed to unearth unusual behaviors that may impact the business, while outlier detection is really a statistical analysis that targets individual data points in a series of data.
Subscribe to our monthly newsletter
Stay up to date on Snowflake’s latest products, expert insights and resources—right in your inbox!
