ggplot2 geom_density_2d allows to fill by factors but not by values

I'd like to use stat_density2D function but coloring the densities by the values of the points. I can color the densities by factors, but not by continous values.

plot_data <-
  data.frame(X = c(rnorm(300, 3, 2.5), rnorm(150, 7, 2)),
             Y = c(rnorm(300, 6, 2.5), rnorm(150, 2, 2)),
             Label = c(rep('A', 300), rep('B', 150)),
             Continous = c(rnorm(300, 3, 2.5), rnorm(150, 7, 2)))

ggplot(plot_data, aes(X, Y)) + 
  stat_density_2d(geom = "polygon", aes(alpha = ..level.., fill = Label))

Obtaining the plot coloured by label: ColorByLabel

but if we aes(alpha = ..level.., fill = Continous) change we obtain: ColorByContinous