Identifying unusual memory allocation in julia

user_4770853 0
Viewed 594

I am a new julia user about to write my first script. I use R, Matlab and python for data analysis and I have yet to have to worry about concerns people in c or java would have with memory allocation and more advacned programming I geuss. I want to use julia to simulate some biological data and in python these types of programs can get really slow. So as I was reading the performance tips in the part on memory allocation caught my eye. As some one who kind of lacks a background in more advanced languages like C what should I be looking for? I read this stack post here but he had python code he had compiled and then run in Julia so I don't feel it is pertinent.

How will I know when I have unusual memory allocation, and conversely what does "good" memory allocation in Julia look like? Is there a way to identify what an unacceptable level of memory demand is based on what I am running?

In addition to the @time macro what other proofing tips do you have?

julia memory-management
1 Answers

To understand Julia performance issues very carefully read https://docs.julialang.org/en/v1/manual/performance-tips/ Running yourself all examples in that page is a must.

Usually, memory allocation is not the major bottleneck. From my experience when people are starting to use Julia the type stability and using global variables are a major source of all performance problems.

The above document has also a section on memory allocation:

"Unexpected memory allocation is almost always a sign of some problem with your code, usually a problem with type-stability or creating many small temporary arrays."

You will find there detailed discussion about addressing such issues.

Regarding code profiling you have the following tools:

  • @time macro (note that the first time you run it for a given methods you are actually mostly measuring compile time rather than run time. Hence usually BenchmarkTools.jl is recommended
  • BenchmarkTools.jl with its @btime macro (among others - the first choice to understand what is going on
  • inbuilt profiling (https://docs.julialang.org/en/v1/manual/profile/) together with ProfileView.jl and ProfileSVG.jl packages to visualize profiling data in standalone and Jupyter environments respectively
  • in most complicated cases you can view the code at various compiling stages with @code_lowered, @code_typed and @code_native

These are the basic and most useful tools for a great start to performant Julia code.

user_99577100
Related