Is there are version of `setorder` that behaves like `setcolorder`

neworder should be the "lookup index" of the new ordering, e.g. neworder = c(3, 1, 2) gives the 3rd row as the new first row, the 1st row as the new second row, etc...

# example
DT = data.table(mtcars, keep.rownames=TRUE)[1:3]
ord = c(3,1,2)

DT

              rn  mpg cyl disp  hp drat    wt  qsec vs am gear carb 
1:     Mazda RX4 21.0   6  160 110 3.90 2.620 16.46  0  1    4    4   
2: Mazda RX4 Wag 21.0   6  160 110 3.90 2.875 17.02  0  1    4    4   
3:    Datsun 710 22.8   4  108  93 3.85 2.320 18.61  1  1    4    1   

# use DT[ord, do_stuff]:
setorderv(DT[ord, .rn := .I], ".rn")[]

              rn  mpg cyl disp  hp drat    wt  qsec vs am gear carb .rn
1:    Datsun 710 22.8   4  108  93 3.85 2.320 18.61  1  1    4    1   1
2:     Mazda RX4 21.0   6  160 110 3.90 2.620 16.46  0  1    4    4   2
3: Mazda RX4 Wag 21.0   6  160 110 3.90 2.875 17.02  0  1    4    4   3

As noted in a comment, I think it would be a bad idea to get rid of the column capturing the row ordering, but you can make a wrapper to get rid of it as in the other answers.

It may be necessary to use 1:.N in a future version if the behavior of .I changes as discussed here: https://github.com/Rdatatable/data.table/issues/2598

If your integer vector is not inherent to the table, then I don't see a function to do it automatically (I hope others will chime in, I'm not a data.table-guru). Lacking that, here's a quick function, with annoying message calls to show the object memory address, to show this is done internally (and not changing the memory location):

setroworder <- function(DT, vec, verbose = TRUE, vecname = NA) {
  if (is.logical(verbose)) verbose <- if (verbose) message else c
  verbose("# ", data.table::address(DT))
  if (is.na(vecname)) {
    # find an unused name
    vecname <- make.unique(c(colnames(DT), "vec"))[ ncol(DT) + 1L ]
  }
  verbose("# ", data.table::address(DT))
  set(DT, i = NULL, j = vecname, value = order(vec))
  verbose("# ", data.table::address(DT))
  setorderv(DT, vecname)
  verbose("# ", data.table::address(DT))
  set(DT, j = vecname, value = NULL)
  verbose("# ", data.table::address(DT))
  invisible(DT) # convenience only, this function operates in side-effect
}

In action:

x <- data.table(a = 1:10)
setroworder(x, c(3,1,2,4:10))[]
# # 0000000012EFF1A8
# # 0000000012EFF1A8
# # 0000000012EFF1A8
# # 0000000012EFF1A8
# # 0000000012EFF1A8
#      a
#  1:  3
#  2:  1
#  3:  2
#  4:  4
#  5:  5
#  6:  6
#  7:  7
#  8:  8
#  9:  9
# 10: 10

Edit. The original answer did not give behaviour equivalent to secolorder. neworder should be the "lookup index" of the new ordering, e.g. neworder = c(3, 1, 2) gives the 3rd row as the new first row, the 1st row as the new second row, etc...

Here is my solution:

setroworder <- function(x, neworder) {
  # This is assumes that there is some convention that colnames do not start with '.'.
  # I don't know if there is any such convention though.
  x[, .indexcol := sort.int(neworder, index.return = TRUE)$ix]
  setorder(x, .indexcol)
  x[, .indexcol := NULL]
}

Testing it out:

> x <- as.data.table(mtcars)
> head(x)
    mpg cyl disp  hp drat    wt  qsec vs am gear carb
1: 21.0   6  160 110 3.90 2.620 16.46  0  1    4    4
2: 21.0   6  160 110 3.90 2.875 17.02  0  1    4    4
3: 22.8   4  108  93 3.85 2.320 18.61  1  1    4    1
4: 21.4   6  258 110 3.08 3.215 19.44  1  0    3    1
5: 18.7   8  360 175 3.15 3.440 17.02  0  0    3    2
6: 18.1   6  225 105 2.76 3.460 20.22  1  0    3    1
> set.seed(42)
> head(setroworder(x, sample(32)))
    mpg cyl disp  hp drat    wt  qsec vs am gear carb
1: 14.7   8 440.0 230 3.23 5.345 17.42  0  0    3    4
2: 18.7   8 360.0 175 3.15 3.440 17.02  0  0    3    2
3: 21.0   6 160.0 110 3.90 2.620 16.46  0  1    4    4
4: 19.2   8 400.0 175 3.08 3.845 17.05  0  0    3    2
5: 19.2   6 167.6 123 3.92 3.440 18.30  1  0    4    4
6: 21.4   6 258.0 110 3.08 3.215 19.44  1  0    3    1