Error when passing new_data back to transformation in ARIMA model to generate forecast

Question

This is a follow up to the question posed here: new_transformation not working with multiple inputs from tsibble object inside ARIMA

When passing back a CPI forecast as new_data to undo the CPI transformation, I receive an error. Any thoughts?

Here is the code:

library(fpp3)

aus_acc_fit <- aus_accommodation %>% 
  model(ARIMA(resp(Takings)/CPI*100 ~ trend(knots = yearquarter("2008 Q1")) + season()))

cpi_fc <- aus_accommodation %>%
  model(ARIMA(CPI)) %>% 
  forecast(h =6) %>%
  as_tsibble() %>% 
  select(-.model, -CPI) %>% 
  rename(CPI = .mean) %>% 
  relocate(Date, .before = State)

aus_acc_fc <- aus_acc_fit %>% 
  forecast(new_data = cpi_fc)

And here is the error:

Error in `mutate()`:
! Problem while computing `ARIMA(resp(Takings)/CPI * 100 ~ trend(knots = yearquarter("2008 Q1")) + season()) =
  (function (object, ...) ...`.
Caused by error in `hessian.default()`:
! Richardson method for hessian assumes a scalar valued function.
Run `rlang::last_error()` to see where the error occurred.

user_19636006 · Answer

FYI I had to modify the solution a bit for my machine to replicate the results. I updated distributional and fabletools to the GitHub versions, but if I updated fable to the GitHub version, the model() function has trouble fitting for some reason. So I reinstalled the CRAN version of fable and now it works.

Results:

> library(fpp3)
── Attaching packages ───────────────────────────────────────────────────────────────────────────────────── fpp3 0.4.0 ──
✔ tibble      3.1.7     ✔ tsibble     1.1.1
✔ dplyr       1.0.9     ✔ tsibbledata 0.4.0
✔ tidyr       1.2.0     ✔ feasts      0.2.2
✔ lubridate   1.8.0     ✔ fable       0.3.1
✔ ggplot2     3.3.6     
── Conflicts ────────────────────────────────────────────────────────────────────────────────────────── fpp3_conflicts ──
✖ lubridate::date()    masks base::date()
✖ dplyr::filter()      masks stats::filter()
✖ tsibble::intersect() masks base::intersect()
✖ tsibble::interval()  masks lubridate::interval()
✖ dplyr::lag()         masks stats::lag()
✖ tsibble::setdiff()   masks base::setdiff()
✖ tsibble::union()     masks base::union()
> aus_acc_fit <- aus_accommodation %>% 
+   model(ARIMA(resp(Takings)/CPI*100 ~ trend(knots = yearquarter("2008 Q1")) + season()))
> cpi_fc <- aus_accommodation %>%
+   model(ARIMA(CPI)) %>% 
+   forecast(h =6) %>%
+   as_tsibble() %>% 
+   select(-.model, -CPI) %>% 
+   rename(CPI = .mean) %>% 
+   relocate(Date, .before = State) %>% 
+   mutate(Takings = 0)
> aus_acc_fc <- aus_acc_fit %>% 
+   forecast(new_data = cpi_fc)
> aus_acc_fc
# A fable: 48 x 6 [1Q]
# Key:     State, .model [8]
   State                        .model                                                   Date         Takings .mean   CPI
                                                                                          
 1 Australian Capital Territory "ARIMA(resp(Takings)/CPI * 100 ~ trend(knots = yearq… 2016 Q3   t(N(62, 9.9))  67.3  109.
 2 Australian Capital Territory "ARIMA(resp(Takings)/CPI * 100 ~ trend(knots = yearq… 2016 Q4    t(N(59, 12))  64.7  110.
 3 Australian Capital Territory "ARIMA(resp(Takings)/CPI * 100 ~ trend(knots = yearq… 2017 Q1    t(N(59, 13))  65.0  110.
 4 Australian Capital Territory "ARIMA(resp(Takings)/CPI * 100 ~ trend(knots = yearq… 2017 Q2    t(N(59, 13))  65.9  111.
 5 Australian Capital Territory "ARIMA(resp(Takings)/CPI * 100 ~ trend(knots = yearq… 2017 Q3    t(N(61, 13))  67.9  111.
 6 Australian Capital Territory "ARIMA(resp(Takings)/CPI * 100 ~ trend(knots = yearq… 2017 Q4    t(N(59, 13))  65.9  112.
 7 New South Wales              "ARIMA(resp(Takings)/CPI * 100 ~ trend(knots = yearq… 2016 Q3 t(N(791, 1254)) 863.   109.
 8 New South Wales              "ARIMA(resp(Takings)/CPI * 100 ~ trend(knots = yearq… 2016 Q4 t(N(844, 1589)) 926.   110.
 9 New South Wales              "ARIMA(resp(Takings)/CPI * 100 ~ trend(knots = yearq… 2017 Q1 t(N(829, 1679)) 915.   110.
10 New South Wales              "ARIMA(resp(Takings)/CPI * 100 ~ trend(knots = yearq… 2017 Q2 t(N(734, 1703)) 814.   111.
# … with 38 more rows
# ℹ Use `print(n = ...)` to see more rows

Session info:

─ Session info ────────────────────────────────────────────────────────────────────────────────────────────────────────
 setting  value
 version  R version 4.2.1 (2022-06-23 ucrt)
 os       Windows 10 x64 (build 19044)
 system   x86_64, mingw32
 ui       RStudio
 language (EN)
 collate  English_United States.utf8
 ctype    English_United States.utf8
 tz       America/New_York
 date     2022-07-29
 rstudio  2022.07.1+554 Spotted Wakerobin (desktop)
 pandoc   NA

─ Packages ────────────────────────────────────────────────────────────────────────────────────────────────────────────
 package        * version    date (UTC) lib source
 anytime          0.3.9      2020-08-27 [1] CRAN (R 4.2.1)
 assertthat       0.2.1      2019-03-21 [1] CRAN (R 4.1.3)
 cli              3.3.0      2022-04-25 [1] CRAN (R 4.1.3)
 colorspace       2.0-3      2022-02-21 [1] CRAN (R 4.1.3)
 crayon           1.5.1      2022-03-26 [1] CRAN (R 4.2.1)
 DBI              1.1.3      2022-06-18 [1] CRAN (R 4.1.3)
 digest           0.6.29     2021-12-01 [1] CRAN (R 4.1.3)
 distributional   0.3.0.9000 2022-07-29 [1] Github (mitchelloharawild/distributional@f30b8ff)
 dplyr          * 1.0.9      2022-04-28 [1] CRAN (R 4.1.3)
 ellipsis         0.3.2      2021-04-29 [1] CRAN (R 4.1.3)
 fable          * 0.3.1      2021-05-16 [1] CRAN (R 4.2.1)
 fabletools     * 0.3.2.9000 2022-07-29 [1] Github (tidyverts/fabletools@28a9cad)
 fansi            1.0.3      2022-03-24 [1] CRAN (R 4.2.1)
 farver           2.1.0      2021-02-28 [1] CRAN (R 4.1.3)
 feasts         * 0.2.2      2021-06-03 [1] CRAN (R 4.2.1)
 fpp3           * 0.4.0      2021-02-06 [1] CRAN (R 4.2.1)
 generics         0.1.3      2022-07-05 [1] CRAN (R 4.2.1)
 GGally           2.1.2      2021-06-21 [1] CRAN (R 4.2.1)
 ggplot2        * 3.3.6      2022-05-03 [1] CRAN (R 4.1.3)
 glue             1.6.2      2022-02-24 [1] CRAN (R 4.1.3)
 gtable           0.3.0      2019-03-25 [1] CRAN (R 4.1.3)
 hms              1.1.1      2021-09-26 [1] CRAN (R 4.1.3)
 lattice          0.20-45    2021-09-22 [2] CRAN (R 4.2.1)
 lifecycle        1.0.1      2021-09-24 [1] CRAN (R 4.1.3)
 lubridate      * 1.8.0      2021-10-07 [1] CRAN (R 4.2.1)
 magrittr         2.0.3      2022-03-30 [1] CRAN (R 4.1.3)
 munsell          0.5.0      2018-06-12 [1] CRAN (R 4.1.3)
 nlme             3.1-157    2022-03-25 [2] CRAN (R 4.2.1)
 numDeriv         2016.8-1.1 2019-06-06 [1] CRAN (R 4.1.1)
 pillar           1.8.0      2022-07-18 [1] CRAN (R 4.2.1)
 pkgconfig        2.0.3      2019-09-22 [1] CRAN (R 4.1.3)
 plyr             1.8.7      2022-03-24 [1] CRAN (R 4.2.1)
 progressr        0.10.1     2022-06-03 [1] CRAN (R 4.2.1)
 purrr            0.3.4      2020-04-17 [1] CRAN (R 4.1.3)
 R6               2.5.1      2021-08-19 [1] CRAN (R 4.1.3)
 rappdirs         0.3.3      2021-01-31 [1] CRAN (R 4.1.3)
 RColorBrewer     1.1-3      2022-04-03 [1] CRAN (R 4.2.0)
 Rcpp             1.0.8.3    2022-03-17 [1] CRAN (R 4.1.3)
 readr            2.1.2      2022-01-30 [1] CRAN (R 4.1.3)
 reshape          0.8.9      2022-04-12 [1] CRAN (R 4.2.1)
 rlang            1.0.4      2022-07-12 [1] CRAN (R 4.2.1)
 rstudioapi       0.13       2020-11-12 [1] CRAN (R 4.1.3)
 scales           1.2.0      2022-04-13 [1] CRAN (R 4.2.1)
 sessioninfo      1.2.2      2021-12-06 [1] CRAN (R 4.2.1)
 tibble         * 3.1.7      2022-05-03 [1] CRAN (R 4.1.3)
 tidyr          * 1.2.0      2022-02-01 [1] CRAN (R 4.1.3)
 tidyselect       1.1.2      2022-02-21 [1] CRAN (R 4.1.3)
 tsibble        * 1.1.1      2021-12-03 [1] CRAN (R 4.2.1)
 tsibbledata    * 0.4.0      2022-01-07 [1] CRAN (R 4.2.1)
 tzdb             0.3.0      2022-03-28 [1] CRAN (R 4.1.3)
 urca             1.3-0      2016-09-06 [1] CRAN (R 4.1.3)
 utf8             1.2.2      2021-07-24 [1] CRAN (R 4.1.3)
 vctrs            0.4.1      2022-04-13 [1] CRAN (R 4.1.3)
 withr            2.5.0      2022-03-03 [1] CRAN (R 4.1.3)

 [1] C:/Users/mrobinson/AppData/Local/R/win-library/4.2
 [2] C:/Program Files/R/R-4.2.1/library

user_11122210 · Answer

Nice code - there is nothing wrong with your attempt here!

This issue has been fixed in the development versions of the software which I hope to put on CRAN soon. Try updating {distributional} first, and if that doesn't work {fabletools} and {fable}.

library(fpp3)
#> ── Attaching packages ──────────────────────────────────────────── fpp3 0.4.0 ──
#> ✔ tibble      3.1.7          ✔ tsibble     1.1.1     
#> ✔ dplyr       1.0.9          ✔ tsibbledata 0.4.0     
#> ✔ tidyr       1.2.0          ✔ feasts      0.2.2     
#> ✔ lubridate   1.8.0          ✔ fable       0.3.1.9000
#> ✔ ggplot2     3.3.6
#> ── Conflicts ───────────────────────────────────────────────── fpp3_conflicts ──
#> ✖ lubridate::date()    masks base::date()
#> ✖ dplyr::filter()      masks stats::filter()
#> ✖ tsibble::intersect() masks base::intersect()
#> ✖ tsibble::interval()  masks lubridate::interval()
#> ✖ dplyr::lag()         masks stats::lag()
#> ✖ tsibble::setdiff()   masks base::setdiff()
#> ✖ tsibble::union()     masks base::union()

aus_acc_fit <- aus_accommodation %>% 
  model(ARIMA(resp(Takings)/CPI*100 ~ trend(knots = yearquarter("2008 Q1")) + season()))

cpi_fc <- aus_accommodation %>%
  model(ARIMA(CPI)) %>% 
  forecast(h =6) %>%
  as_tsibble() %>% 
  select(-.model, -CPI) %>% 
  rename(CPI = .mean) %>% 
  relocate(Date, .before = State)

aus_acc_fc <- aus_acc_fit %>% 
  forecast(new_data = cpi_fc)

aus_acc_fc
#> # A fable: 48 x 6 [1Q]
#> # Key:     State, .model [8]
#>    State                        .model          Date         Takings .mean   CPI
#>                                                  
#>  1 Australian Capital Territory "ARIMA(resp… 2016 Q3   t(N(62, 9.9))  67.3  109.
#>  2 Australian Capital Territory "ARIMA(resp… 2016 Q4    t(N(59, 12))  64.7  110.
#>  3 Australian Capital Territory "ARIMA(resp… 2017 Q1    t(N(59, 13))  65.0  110.
#>  4 Australian Capital Territory "ARIMA(resp… 2017 Q2    t(N(59, 13))  65.9  111.
#>  5 Australian Capital Territory "ARIMA(resp… 2017 Q3    t(N(61, 13))  67.9  111.
#>  6 Australian Capital Territory "ARIMA(resp… 2017 Q4    t(N(59, 13))  65.9  112.
#>  7 New South Wales              "ARIMA(resp… 2016 Q3 t(N(791, 1254)) 863.   109.
#>  8 New South Wales              "ARIMA(resp… 2016 Q4 t(N(844, 1589)) 926.   110.
#>  9 New South Wales              "ARIMA(resp… 2017 Q1 t(N(829, 1679)) 915.   110.
#> 10 New South Wales              "ARIMA(resp… 2017 Q2 t(N(734, 1703)) 814.   111.
#> # … with 38 more rows

^{Created on 2022-07-29 by the reprex package (v2.0.1)}

Session info

sessioninfo::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#>  setting  value
#>  version  R version 4.1.2 (2021-11-01)
#>  os       Ubuntu 20.04.4 LTS
#>  system   x86_64, linux-gnu
#>  ui       X11
#>  language en_AU:en
#>  collate  en_AU.UTF-8
#>  ctype    en_AU.UTF-8
#>  tz       Australia/Melbourne
#>  date     2022-07-29
#>  pandoc   2.17.1.1 @ /usr/lib/rstudio/bin/quarto/bin/ (via rmarkdown)
#> 
#> ─ Packages ───────────────────────────────────────────────────────────────────
#>  package        * version    date (UTC) lib source
#>  anytime          0.3.9      2020-08-27 [1] RSPM (R 4.1.2)
#>  assertthat       0.2.1      2019-03-21 [1] RSPM (R 4.1.2)
#>  cli              3.3.0      2022-04-25 [1] RSPM (R 4.1.2)
#>  colorspace       2.0-3      2022-02-21 [1] RSPM (R 4.1.2)
#>  crayon           1.5.1      2022-03-26 [1] RSPM (R 4.1.2)
#>  DBI              1.1.2      2021-12-20 [1] RSPM (R 4.1.2)
#>  digest           0.6.29     2021-12-01 [1] RSPM (R 4.1.2)
#>  distributional   0.3.0.9000 2022-06-10 [1] local
#>  dplyr          * 1.0.9      2022-04-28 [1] CRAN (R 4.1.2)
#>  ellipsis         0.3.2      2021-04-29 [1] RSPM (R 4.1.2)
#>  evaluate         0.15       2022-02-18 [1] RSPM (R 4.1.2)
#>  fable          * 0.3.1.9000 2022-07-23 [1] local
#>  fabletools     * 0.3.2.9000 2022-06-13 [1] local
#>  fansi            1.0.3      2022-03-24 [1] RSPM (R 4.1.2)
#>  farver           2.1.0      2021-02-28 [1] RSPM (R 4.1.2)
#>  fastmap          1.1.0      2021-01-25 [1] RSPM (R 4.1.2)
#>  feasts         * 0.2.2      2021-06-03 [1] RSPM (R 4.1.2)
#>  fpp3           * 0.4.0      2021-02-06 [1] RSPM (R 4.1.2)
#>  fs               1.5.2      2021-12-08 [1] CRAN (R 4.1.2)
#>  generics         0.1.2      2022-01-31 [1] RSPM (R 4.1.2)
#>  ggplot2        * 3.3.6      2022-05-03 [1] CRAN (R 4.1.2)
#>  glue             1.6.2      2022-02-24 [1] CRAN (R 4.1.2)
#>  gtable           0.3.0      2019-03-25 [1] RSPM (R 4.1.2)
#>  highr            0.9        2021-04-16 [1] RSPM (R 4.1.2)
#>  htmltools        0.5.2      2021-08-25 [1] RSPM (R 4.1.2)
#>  knitr            1.39       2022-04-26 [1] RSPM (R 4.1.2)
#>  lattice          0.20-45    2021-09-22 [2] CRAN (R 4.1.2)
#>  lifecycle        1.0.1      2021-09-24 [1] CRAN (R 4.1.2)
#>  lubridate      * 1.8.0      2021-10-07 [1] RSPM (R 4.1.2)
#>  magrittr         2.0.3      2022-03-30 [1] RSPM (R 4.1.2)
#>  munsell          0.5.0      2018-06-12 [1] RSPM (R 4.1.2)
#>  nlme             3.1-153    2021-09-07 [2] CRAN (R 4.1.2)
#>  numDeriv         2016.8-1.1 2019-06-06 [1] RSPM (R 4.1.2)
#>  pillar           1.7.0      2022-02-01 [1] RSPM (R 4.1.2)
#>  pkgconfig        2.0.3      2019-09-22 [1] RSPM (R 4.1.2)
#>  progressr        0.10.1     2022-06-03 [1] CRAN (R 4.1.2)
#>  purrr            0.3.4      2020-04-17 [1] CRAN (R 4.1.2)
#>  R6               2.5.1      2021-08-19 [1] CRAN (R 4.1.2)
#>  rappdirs         0.3.3      2021-01-31 [1] CRAN (R 4.1.2)
#>  Rcpp             1.0.8.3    2022-03-17 [1] RSPM (R 4.1.2)
#>  reprex           2.0.1      2021-08-05 [1] RSPM (R 4.1.2)
#>  rlang            1.0.2      2022-03-04 [1] RSPM (R 4.1.2)
#>  rmarkdown        2.14       2022-04-25 [1] RSPM (R 4.1.2)
#>  rstudioapi       0.13       2020-11-12 [1] CRAN (R 4.1.2)
#>  scales           1.2.0      2022-04-13 [1] RSPM (R 4.1.2)
#>  sessioninfo      1.2.2      2021-12-06 [1] RSPM (R 4.1.2)
#>  stringi          1.7.6      2021-11-29 [1] RSPM (R 4.1.2)
#>  stringr          1.4.0      2019-02-10 [1] RSPM (R 4.1.2)
#>  tibble         * 3.1.7      2022-05-03 [1] CRAN (R 4.1.2)
#>  tidyr          * 1.2.0      2022-02-01 [1] RSPM (R 4.1.2)
#>  tidyselect       1.1.2      2022-02-21 [1] RSPM (R 4.1.2)
#>  tsibble        * 1.1.1      2021-12-03 [1] RSPM (R 4.1.2)
#>  tsibbledata    * 0.4.0      2022-01-07 [1] RSPM (R 4.1.2)
#>  urca             1.3-0      2016-09-06 [1] RSPM (R 4.1.2)
#>  utf8             1.2.2      2021-07-24 [1] RSPM (R 4.1.2)
#>  vctrs            0.4.1      2022-04-13 [1] RSPM (R 4.1.2)
#>  withr            2.5.0      2022-03-03 [1] RSPM (R 4.1.2)
#>  xfun             0.31       2022-05-10 [1] CRAN (R 4.1.2)
#>  yaml             2.3.5      2022-02-21 [1] CRAN (R 4.1.2)
#> 
#>  [1] /home/mitchell/R/x86_64-pc-linux-gnu-library/4.1
#>  [2] /opt/R/4.1.2/lib/R/library
#> 
#> ──────────────────────────────────────────────────────────────────────────────

Error when passing new_data back to transformation in ARIMA model to generate forecast

2 Answers