How to calculate the overlap between 2 dataset distribution

user_5001665 0
Viewed 1355

Hi How can calculate the overlapping area between 2 columns ( or 2 subsets of a column) in R. Please see the example data below:

set.seed(1234)
df <- data.frame(
  Data=factor(rep(c("D1", "D2"), each=200)),
  weight=round(c(rnorm(200, mean=55, sd=5),
                 rnorm(200, mean=65, sd=5)))
)

library(ggplot2)
plot <- ggplot(df, aes(weight,fill = Data))+
  geom_density() 
plot

This results in the below plot. I am wondering, how to color the overlapping area and calculate the overlapping coefficient (OVL) similar to what is done here Using Monte Carlo Integration? Please note that the link (and example above) provided uses parametric distribution while I am asking if I have a dataset of observed values.

enter image description here

montecarlo integration r
1 Answers

I normally find it easier to work directly with the densities and plot them as geom_area. If you get the x-axis sampling points to match on the two distributions you can find the overlap area using pmin, and the sum of its values divided by the sum of the values for the two curves should give you the proportion of the total area that is overlapped.

d1dens <- with(df, density(weight[Data == "D1"], 
                           from = min(weight), 
                           to = max(weight)))
d2dens <- with(df, density(weight[Data == "D2"], 
                           from = min(weight),
                           to = max(weight)))
joint <- pmin(d1dens$y, d2dens$y)

df2 <- data.frame(x = rep(d1dens$x, 3), 
                  y = c(d1dens$y, d2dens$y, joint),
                  Data = rep(c("D1", "D2", "overlap"), each = length(d1dens$x)))

ggplot(df2, aes(x, y, fill = Data)) + 
  geom_area(position = position_identity(), color = "black") +
  scale_fill_brewer(palette = "Pastel2") +
  theme_bw()

enter image description here

sum(joint) / sum(d1dens$y, d2dens$y)
#> [1] 0.1480701
user_125003150
Related