ggplot2: geom_point() dodge shape but NOT color

I'm trying to get a plot in ggplot2 with geom_point() having variables mapped to x, y, color and shape and to dodge the position for color but not shape.

x=tibble(Color=c(rep('A',12),rep('B',12),rep('C',12)),
     Shape=rep(c(rep('A',3),rep('B',3),rep('C',3),rep('D',3)),3),
     xVar=rep(c('A','B','C'),12),
     Value=rnorm(36))

ggplot(x,aes(xVar,Value,color=Color,shape=Shape))+
     geom_point(position=position_dodge(width=.5))

Is it possible to restrict the dodge position to just one aesthetic? I've scoured documentation and stack overflow but haven't found anything yet.