ggforce facet_zoom - labels only on zoomed example

I would like to label points in a scatterplot, but only those within the facet_zoom panel. Here is an example:

library(ggplot2)
library(ggforce)
library(ggrepel)
library(magrittr)

labels <- letters
example_values_x <- rnorm(26)
example_values_y <- rnorm(26)

df <- data.frame(labels, 
                 example_values_x, 
                 example_values_y)
df %>% ggplot(aes(y = example_values_y, 
                  x = example_values_x)) +
  geom_point() +
  facet_zoom(x = example_values_x > 0.5) + 
  geom_label_repel(data = filter(df, example_values_x > 0.5), aes(label = labels))

Any idea how to make it so the labels don't also appear on the non-zoomed panel?