remove the outliers - Winsorization

What is Winsorization

Winsorization is a method to remove the outliers. The process is:

• Set a threshold of outliers(e.g. data under 5% and higher than 95%)
• Change the values of outliers into the threshold value.

That's it, it is a very simple method to deal with outliers.

Simulation

I randomly generate 100 data points from normal with mean 100 and standard deviation 10. Therefore, the mean should be 100 theorically. Also, I add three outliers in the data points. Now, the number of data points are 103.

Since there are some outliers, the mean becomes 102.11. Mean is outlier-sensitive, so we can use it to try the different method for outlier removing. Trimming method is to delete the outliers, right now we set the data below 5% and above 95% are outliers. Winsorization is to set those ouliters to 5% value or 95% value.

In conclusion, we can see both mean are very close to 100.

Why we do not use trimming method?

If we can have the similar results by deleting all outliers, why we need Winsorization. The key difference between two methods is sample number.

Using Winsorization, we keep the same sample number as the original data, while using trimming, the sample number is less than original.

Why we care the sample number? right, we do not need to care about sample number most of the time. However, in hypothesis testing, if we decrease the sample number, the power will also decrease. To keep the same power, we need to find a method to deal with outliers in the same sample number.