How to center dotplot with different fill than boxplot? (ggplot)

New R / ggplot user here.

I'm looking to overlay dots over a boxplot, with a different fill color than the boxplot.

Here's a reproducible example, using the built-in mtcars dataset:

mtcars$am = factor(mtcars$am, levels = c(0, 1), labels = c("A", "B"))
mtcars$vs = factor(mtcars$vs, levels = c(0, 1), labels = c("vs 0", "vs 1"))

ggplot(data = mtcars, aes(x = am, y = disp, fill = am)) +
  geom_boxplot(width = 0.3, outlier.size = 0) +
  geom_point(aes(fill = vs), size = 2, shape = 21, position = position_jitterdodge(0.15))

I would like the dots to be centered over the boxplots. For example, over the center of the pink boxplot, I would like blue and purple dots. Is this possible?