Font spacing whern using cairo_pdf device

I am using {ggtext} and I noticed that there are gaps in the font spacing when I am using smaller font size.

library(tidyverse)
library(ggtext)

ggplot() +
  annotate(
    "richtext",
    x = 0,
    y = 1,
    label = "<span style = 'font-size:6pt;'>123456789</span>",
    family = "Poppins",
    fill = NA,
    label.color = NA
  ) +
  geom_text(
    aes(
      x = 0,
      y = 1.01
    ),
    label = "123456789",
    family = "Poppins",
    size = 6/.pt
  ) +
  scale_y_continuous(expand = expansion(mult = c(1, 1)))

ggsave("~/Desktop/test.pdf", device = cairo_pdf)
#> Saving 7 x 5 in image
ggsave("~/Desktop/test.png", dpi = 600)
#> Saving 7 x 5 in image

^{Created on 2020-12-07 by the reprex package (v0.3.0.9001)}

This is only problematic when using the cairo_pdf device. If we zoom in on the PNG version, the font spacing looks ok. Any ideas?

^{Created on 2020-12-07 by the reprex package (v0.3.0.9001)}