How to Find Outliers in Python
Find outlier is essential for a host of reasons from not skewing your averages to ensuring that machine learning algorithms function properly. In this tutorial, I am going show you several ways in which you can remove outliers from your data. We explore a few traditional methods such as Z-score and IQR. In addition, we will look at Tukey Ladders and ML algorithms that find the outliers for us.
Watch the video presentation of the how to find outliers in Python
Using Z-Score to Find Outliers in Python
In this method, we calculate the z-score for each data point and identify the ones with a Z-score greater than a certain threshold. The benefit of using this method is super easy to use. However, the negative aspect of this is that this method assumes that your data is normally distributed. Here’s how you can code it in Python:
#import the packages that we need to find outlier import numpy as np from scipy import stats #create some fake data data = np.array([1, 2, 3, 4, 5, 10, 20, 30]) #appy the z-score method and lets get the absolute values z_scores = np.abs(stats.zscore(data)) # lest using a threshold of 1.96 because this 95% of the data threshold threshold = 1.96 outliers = data[z_scores > threshold] print(outliers)
It’s important to note that the data is assume to distributed in a manner seen below in the bell-shaped curve. This allows us to make assumptions on how many standard deviations the data is from the mean which is an element of the normal distribution formula.
Using Standard Deviation to Find Outliers in Python
Using standard deviations to detect outliers is based on the idea that data points that are a certain number of standard deviations away from the mean are considered outliers. This is similar to the method we describe previously with the Z-score with the same limitations.
The general idea is to calculate the mean and standard deviation of the data, and then identify data points that are a certain number of standard deviations away from the mean. The number of standard deviations that are used as the threshold for identifying outliers can be adjusted depending on the desired level of sensitivity and the distribution of the data
#import the packages import numpy as np #create the data and take the mean and standard deviation data = np.array([1, 2, 3, 4, 5, 10, 20, 30]) mean = np.mean(data) std_dev = np.std(data) #More than 3 standard deviations from the mean an outlier threshold = 3 #create the condition to find outliers outliers = data[np.abs(data - mean) > threshold * std_dev] print(outliers)
Using IQR or Boxplot Method to Find Outliers
This method we are evaluating the data into quartiles (25% percentile, 50% percentile and 75% percentile ). We calculate the interquartile range (IQR) and identify the data points that lie outside the range. Here is how calculate the upper and lower data limits
Lower range limit = Q1 – (1.5* IQR). Essentially this is 1.5 times the inner quartile range subtracting from your 1st quartile.
Higher range limit = Q3 + (1.5*IQR). This is 1.5 times IQR+ quartile 3.
Here is how you would implement this Python with some simple to use code
# import the packages import numpy as np # create the data data = np.array([1, 2, 3, 4, 5, 10, 20, 30]) # assign your quartiles, limits and iq3 q1, q3 = np.percentile(data, [25, 75]) iqr = q3 - q1 lower_bound = q1 - 1.5*iqr upper_bound = q3 + 1.5*iqr #create conditions to isolate the outliers outliers = data[(data < lower_bound) | (data > upper_bound)] print(outliers)
Tukey Latter Method
This method is similar to the IQR method, but it uses a different threshold to identify outliers. Instead of 1.5IQR, we use a threshold of kIQR, where k is a parameter that can be set by the user. Also we incorporate the median into this value instead of of using the mean. Here’s how you can implement it in Python
#import the package you need import numpy as np from scipy.stats import iqr data = np.array([1, 2, 3, 4, 5, 10, 20, 30]) # change K to 2 or 1.5. This can be what you need it to be. k = 2.0 q1, q3 = np.percentile(data, [25, 75]) iqr = q3 - q1 lower_bound = np.median(data) - k*iqr upper_bound = np.median(data) + k*iqr outliers = data[(data < lower_bound) | (data > upper_bound)] print(outliers)
Isolation Forest Algorithm
Isolation Forest detects outliers unsupervised. The approach builds many isolation trees, binary trees that iteratively separate data into smaller subsets. Each isolation tree is created by randomly picking a subset of the data and recursively splitting it using a random feature and split value.
Isolation Forest assumes that outliers are isolated and have fewer tree structure paths to reach them than inliers, who are more connected. Hence, the method finds outliers as isolated tree points with short path lengths.
After building isolation trees, the system calculates anomaly scores for each data point to identify outliers. The average path length of the data point across all isolation trees determines the anomaly score. Shorter average path lengths indicate anomalies and outliers.
#bring in the Isolation forest library from sklearn.ensemble import IsolationForest #create your data data = np.array([1, 2, 3, 4, 5, 10, 20, 30]).reshape(-1, 1) #istantiate the algorithm and se estimators and contamination and fit the data isolation_forest = IsolationForest(n_estimators=100, contamination=0.1) isolation_forest.fit(data) # Isolate the negative 1 which would be the algorithms outliers = data[isolation_forest.predict(data) == -1] print(outliers)
Note that these are just a few of the many methods that can be used to identify outliers. The choice of method depends on the specific requirements of your analysis and the nature of your data.