Here is first another approach with geosphere (make a distance matrix with distm) and then I show how you can use the terra::nearby method (which works for both long/lat and planar coordinates).
m <- matrix(c(61, -112, 63, -113, 61, -113, 62, -111, 61, -111, 64, -120), ncol=2, byrow=TRUE)
# note that the order should be long/lat !!!
m <- m[, 2:1]
d <- geosphere::distm(m)
diag(d) <- NA
i <- order(d[4,])[1:3]
i
#[1] 5 1 2
m[i,]
# [,1] [,2]
#[1,] -111 61
#[2,] -112 61
#[3,] -113 63
Now with terra. The below gets the nearest 3 neighbors for all points.
library(terra)
v <- vect(m, crs="+proj=lonlat")
nearby(v, k=3)
# id k1 k2 k3
#1 1 3 5 4
#2 2 4 3 1
#3 3 1 5 4
#4 4 5 1 2
#5 5 1 3 4
#6 6 2 3 4
With terra version 1.3.15 (currently the development version) you can also do
nearby(v[4,], v, k=4)
# id k1 k2 k3 k4
#[1,] 1 4 5 1 2
Taking k=4 neighbors as the first one is the point itself.
To get the development version, do
install.packages('terra', repos='https://rspatial.r-universe.dev')