Returning observations that only occur once in a group

I am trying to group a data.frame by a factor variable, and then return rows of the data.frame that correspond to observations that occur exactly once in each group. For example, consider the following data:

x = matrix(c(1,1,2,2,2,3,4,4,5,4), nrow = 5, ncol = 2, byrow = F)
x = data.frame(x)
x

#   X1 X2
# 1  1  3
# 2  1  4
# 3  2  4
# 4  2  5
# 5  2  4

I would like to group the data by the values in column 1, then return the rows for which the value in column 2 occurs only once within a group. Here, the function would return the first, second, and fourth rows.

Desired output

#   X1 X2
# 1  1  3
# 2  1  4
# 4  2  5

I am looking to apply this to a dataset with >1mm rows.