How to create a sys image that has multiple packages with their precompiled functions cached in julia?

user_9508751 0
Viewed 402

Let's say we have a symbol array of packages packages::Vector{Symbol} = [...] and we want to create a sys image using PackageCompiler.jl. We could simply use

using PackageCompiler

create_sysimage(packages; incremental = false, sysimage_path = "custom_sys.dll"

but without a precompile_execution_file, this isn't going to be worth it.

Note: sysimage_path = "custom_sys.so" on Linux and "custom_sys.dylib" on macOS...

For the precompile_execution_file, I thought running the test for each package might do it so I did something like this:

precompilation.jl

packages = [...]

@assert typeof(packages) == Vector{Symbol}

import Pkg

m = Module()
try Pkg.test.(Base.require.(m, packages)) catch ; end

The try catch is for when some tests give an error and we don't want it to fail.

Then, executing the following in a shell,

using PackageCompiler

packages = [...]

Pkg.add.(String.(packages))
Pkg.update()
Pkg.build.(String.(packages))

create_sysimage(packages; incremental = false,
                          sysimage_path = "custom_sys.dll",
                          precompile_execution_file = "precompilation.jl")

produced a sys image dynamic library which loaded without a problem. When I did using Makie, there was no delay so that part's fine but when I did some plotting with Makie, there still was the first time plot delay so I am guessing the precompilation script didn't do what I thought it would do.

Also when using tab to get suggestions in the repl, it would freeze the first time but I am guessing this is an expected side effect.

julia
1 Answers

There are a few problems with your precompilation.jl script, that make tests throw errors which you don't see because of the try...catch.

But, although running the tests for each package might be a good idea to exercise precompilation, there are deeper reasons why I don't think it can work so simply:

  • Pkg.test spawns a new process in which tests actually run. I don't think that PackageCompiler can see what happens in this separate process.
  • To circumvent that, you might want to simply include() every package's test/runtests.jl file. But this is likely to fail too, because of missing test-specific dependencies.

So I would say that, for this to work reliably and systematically for all packages, you'd have to re-implement (or re-use, if you can) some of the internal logic of Pkg.test in order to add all test-specific dependencies to the current environment.


That being said, some packages have ready-to-use precompilation scripts helping to do just this. This is the case for Makie, which suggests in its documentation to use the following file to build system images:

joinpath(pkgdir(Makie), "test", "test_for_precompile.jl")
user_12256070
Related