Max Array Length in Julia

user_7210250 0
Viewed 880

I can create an array of a million elements like this:

Array(1:1_000_000)
Vector{Int64} with 1000000 elements

but if I try to create an array of a billion elements I get this:

Array(1:1_000_000_000)
Julia has exited.
Press Enter to start a new session.

Is Julia not able to handle a billion elements in an array or what am I doing wrong here?

julia
2 Answers

You are creating an Array of Int64, each of which needs to be stored in memory:

julia> sizeof(3)
8

So at some point you're bound to run out of memory - this is not due to some inherent limit on the number of elements in an array, but rather the size of the overall array, which in turn depends on the size of each element. Consider:

julia> sizeof(Int8(3))
1

julia> [Int8(1) for _ in 1:1_000_000_000]
1000000000-element Array{Int8,1}:
 1
 1
 1
 ⋮
 1
 1
 1

so filling the array with a smaller data type (8-bit rather than 64-bit Integer) allows me to create an array with more elements.

user_24998920

While there is no limit how big an Array can be in Julia there is obviously the available RAM memory limit (mentioned in the other answer). Basically, you can assume that all your available system memory can be allocated for a Julia process. sizeof is a good way to calculate how much RAM you need.

However, if you actually do big array computing in Julia the above limit can be circumvented in many ways:

  1. Use massive memory machines from a major cloud computing provider. I use Julia on AWS Linux and it walks like a charm - you can have a machine up to 4TB RAM on a virtual machine and 24TB RAM on a bare metal machine. While it is not a Julia solution, sometimes it is the easiest and cheapest way to go.

  2. Sometimes your data is sparse - you do not actually use all of those memory cells. In such cases consider SparseArrays. In other cases your sparse data is formatted in some specific way (e.g. non-zero values only on diagonal) in that case use BanndedMatrices.jl. It is worth noting that there is even a Julia package for infinite algebra. Basically whatever you find at the Julia Matrices project is worth looking at.

  3. You can use memory mapping - that means that most of your array is on disk and only some part is hold in RAM. In this way you are limited by your disk space rather than the RAM.

  4. You can use DistributedArrays.jl and have a single huge Array hosted on several machines.

Hope it will be useful for you or other people trying to do big data algebra in Julia.

user_99577100
Related