I am running a linear mixed effects model with a dataset consisting of ~500K observations. I have a continuous response (i.e. Y), 24 categorical variables (i.e. X1,..., X24), 8 continuous variables (i.e. Z1,..., Z8), and one variable for ID (i.e. ID with 1160 levels) that is used for specifying random effects. The linear mixed model formula is Y ~ X1 + ... + X24 + Z1 + ... + Z8 + (0+X1|ID) + ... + (0+X24|ID)
In Windows, to fit the model in Julia 0.3, I use the MixedModels package (version 0.3.22) using the following:
- Pre-fit step:
fit1 = lmm(my_formula, my_DATA), and - Fit step:
fit(fit1).
The pre-fit step takes ~ 38 seconds and the fitting step ~14 seconds; 52 seconds in total
Again, in Windows, the model was fitted in Julia 1.2 using the MixedModels (version 2.2.0):
- Pre-fit step:
fit2 = LinearMixedModel(@formula(my_formula),my_DATA), and - Fit step:
fit!(fit2).
In this case, the pre-fit step takes ~20 seconds, while the fitting step ~301 seconds (321 seconds in total)
When setting up a Linux Docker in the same server and using Julia 1.2 with MixedModels (version 2.2.0):
- Pre-fit step:
fit3 = LinearMixedModel(@formula(my_formula),my_DATA), and - Fit step:
fit!(fit3).
The pre-fit step takes ~21 seconds, while the fitting step ~108 seconds (129 seconds in total). In this case, if I consider setting up the number of threads to 1, the pre-fit step takes ~25 seconds, while the fitting step ~55 seconds (80 seconds in total).
See below, the summarized results. Please note, that more configurations have been investigated. With regard to the columns, the model was fit in either Windows, or Dockerized Linux (column Windows/Linux). I have examined setting the number of threads to 1 or allowing it to take its default value prior to model fitting (column: BLAS.set_num_threads(1)).
+---------------+---------------+-------------------------+--------------+----------+------------+
| Windows/Linux | Julia version | BLAS.set_num_threads(1) | Pre-fit time | Fit time | Total time |
| | (MixedModels) | | | | |
+---------------+---------------+-------------------------+--------------+----------+------------+
| | 0.3.12(0.3.22)| No | 38 sec | 14 sec | 52 sec |
| +---------------+-------------------------+--------------+----------+------------+
| Windows | 1.2 (2.2.0) | No | 20 sec | 301 sec | 321 sec |
+---------------+---------------+-------------------------+--------------+----------+------------+
| | 1.2 (2.2.0) | No | 21 sec | 108 sec | 129 sec |
| +---------------+-------------------------+--------------+----------+------------+
| | 1.2 (2.2.0) | Yes | 25 sec | 55 sec | 80 sec |
| Linux Docker +---------------+-------------------------+--------------+----------+------------+
| | 1.4 (2.2.0) | No | 25 sec | 108 sec | 133 sec |
| +---------------+-------------------------+--------------+----------+------------+
| | 1.4 (2.2.0) | Yes | 24 sec | 57 sec | 81 sec |
+---------------+---------------+-------------------------+--------------+----------+------------+
Given that I run the models multiple times, I end up with hours of difference between Julia 0.3.12 and the rest of the methods. Even by running the model in Linux using Julia 1.2 and setting the number of threads to 1, the process takes 50% more time. Does anyone know where this difference stems from? Is there a way to speed up the fitting process in Julia 1.2 or make it somewhat comparable to Julia 0.3.12?
This is a Windows 2012 server with CPU: Intel Xeon @3.5GHz E5-2637 v3 (4 cores; 8 threads) and 64 GB RAM. Both Windows and Linux runs were executed on the same server.