Since .key is deprecated how it is possible to rename the data column in nest()?

You can use the new nest_by function in dplyr 1.0.0 which works similar to what you had previously with nest.

library(dplyr)
mtcars %>% group_by(cyl) %>% nest_by(.key = "my_name")

#   cyl             my_name
#  <dbl> <list<tbl_df[,10]>>
#1     4           [11 × 10]
#2     6            [7 × 10]
#3     8           [14 × 10]

You can also do the same without grouping.

mtcars %>% nest_by(cyl, .key = "my_name")

You can use group_cols() to refer to grouping variables:

mtcars %>% group_by(cyl) %>% nest(my_name = !group_cols())
#> # A tibble: 3 x 2
#> # Groups:   cyl [3]
#>     cyl my_name
#>   <dbl> <list>
#> 1     6 <tibble [7 × 10]>
#> 2     4 <tibble [11 × 10]>
#> 3     8 <tibble [14 × 10]>

mtcars %>% nest(my_name = !cyl)
#> # A tibble: 3 x 2
#> # Groups:   cyl [3]
#>     cyl my_name
#>   <dbl> <list>
#> 1     6 <tibble [7 × 10]>
#> 2     4 <tibble [11 × 10]>
#> 3     8 <tibble [14 × 10]>

The name can be provided directly in the arguments of nest:

mtcars %>% nest( my_name = -cyl )    # Nest by everything except cyl
#  # A tibble: 3 x 2
#      cyl my_name
#    <dbl> <list>
#  1     6 <tibble [7 × 10]>
#  2     4 <tibble [11 × 10]>
#  3     8 <tibble [14 × 10]>