Identify outliers of river levels that change continuously over time

yongchuangt 0
Viewed 22

This is a time-dependent measure of the water level of a river measured by an instrument that measures the water level every five minutes. However, due to some interference and other factors, there are often some outliers in the water level data. Since the water level changes continuously with time, the outliers can be obviously found from the water-time scatter plot. Usually the outliers are the outliers, but sometimes the rising water can suddenly change the water level. Since the time data is accurate to minutes, how to take minutes as the independent variable in R? How can R automatically identify and eliminate these outliers?

Some of the data and figures are shown below.The ones selected in red are outliers. enter image description here

outliers r
1 Answers

Univariate Outliers

You didnt provide your data for us to look at, so instead I will use the mpg dataset in R, which measures a number of variables on automobile metrics. I will only use the displ, hwy, and cty variables here for demonstration.

Since you were possibly looking for an R solution, one simple way is to use the is_outlier function in the rstatix package. You can also consider checking the mahalanobis distance with mahalanobis_distance if you are concerned about multivariate outliers. To quickly inspect, you can use is_outlier for generic detection (you can modify the settings to set the criterion for what is an "outlier" too) or is_extreme for extreme outliers.

#### Load Libraries ####
library(tidyverse)
library(rstatix)

#### Check Outliers ####
is_outlier(mpg$hwy) %>% 
  table()

Here you can see in the tabulated outcome that there are three outliers:

FALSE  TRUE 
  231     3

We can plot them by coloring this factor within ggplot in the tidyverse package we just loaded:

#### Plot Outliers on Scatter Plot ####
mpg %>% 
  ggplot(aes(x=hwy,
             y=displ))+
  geom_point(aes(color=is_outlier(hwy)))+
  geom_smooth(se=F,
              color="lightblue")+
  labs(color="Outlier",
       x="Highway MPG",
       y="Engine Displacement",
       title="MPG x Displacement With Outlier Detection")+
  theme_bw()+
  scale_color_manual(values = c("darkblue",
                                "red"))

Which gives us this plot. Notice that what the commentor above is higlighted here as well...using geom_smooth allows us to see that the loess line has shifted towards the outliers at the end of this plot.

enter image description here

You mentioned that you would like to filter these values out. We can see clearly that highway MPG values above 40 are outliers now. So we simply make one switch to the plot code with filter:

mpg %>% 
  filter(!hwy > 40) %>% 
  ggplot(aes(x=hwy,
             y=displ))+
  geom_point(aes(color=is_outlier(hwy)))+
  geom_smooth(se=F,
              color="lightblue")+
  labs(color="Outlier",
       x="Highway MPG",
       y="Engine Displacement",
       title="MPG x Displacement With Outlier Detection")+
  theme_bw()+
  scale_color_manual(values = c("darkblue",
                                "red"))

Then our plot shows no values anymore:

enter image description here

If you would like to save the data to not have these outliers, simply do so with the following code. FYI the ! operator here simply says "dont give me this.":

mpg.no.outliers <- mpg %>%
 filter(!hwy > 40)

Multivariate Outliers

Multivariate outliers in your data can be handled in the same way, and your two variables are no different in that regard. We can try to tabulate them as so:

#### Find MVN Outliers ####
mpg %>%
  select(cty,hwy) %>% 
  mahalanobis_distance() %>% 
  filter(is.outlier == "TRUE")

We thus find three outliers between the city and highway MPG variables:

# A tibble: 3 × 4
    cty   hwy mahal.dist is.outlier
  <int> <int>      <dbl> <lgl>     
1    28    33       16.2 TRUE      
2    33    44       14.7 TRUE      
3    35    44       22.7 TRUE

To plot them, we use a very similar method:

#### Plot Them ####
mpg %>% 
  select(cty,hwy) %>% 
  mahalanobis_distance() %>% 
  ggplot(aes(x=hwy,
             y=cty,
             color=is.outlier))+
  geom_point()+
  geom_smooth(color="lightblue",
              se=F)+
  labs(x="Highway MPG",
       y="City MPG",
       title="Highway x City Mileage with MVN Outliers",
       color="MVN Outlier?")+
  theme_bw()+
  scale_color_manual(values = c("darkblue",
                                "red"))

Which gives us this:

enter image description here

You can see in this case that the loess function does not have radical shifts due to the outliers. This is because the numeric values are extreme compared to the others, but their linearity is still similar to the rest of the data points.

As another has said, a more detailed discussion on the when and how of outliers can be had at Cross Validated.

Shawn Hemelstrand0
Related