How can I create a Numpy Array that is much bigger than my RAM from 1000s of CSV files?

When working with numpy and large arrays, there are several approaches depending on what you need to do with that data.

The simplest answer is to use less data. If your data has lots of repeating elements, it is often possible to use a sparse array from scipy because the two libraries are heavily integrated.

Another answer (IMO: the correct solution to your problem) is to use a memory mapped array. This will let numpy only load the necessary parts to ram when needed, and leave the rest on disk. The files containing the data can be simple binary files created using any number of methods, but the built-in python module that would handle this is struct. Appending more data would be as simple as opening the file in append mode, and writing more bytes of data. Make sure that any references to the memory mapped array are re-created any time more data is written to the file so the information is fresh.

Finally is something like compression. Numpy can compress arrays with savez_compressed which can then be opened with numpy.load. Importantly, compressed numpy files cannot be memory-mapped, and must be loaded into memory entirely. Loading one column at a time may be able to get you under the threshold, but this could similarly be applied to other methods to reduce memory usage. Numpy's built in compression techniques will only save disk space not memory. There may exist other libraries that perform some sorts of streaming compression, but that is beyond the scope of my answer.

Here is an example of putting binary data into a file then opening it as a memory-mapped array:

import numpy as np

#open a file for data of a single column
with open('column_data.dat', 'wb') as f:
    #for 1024 "csv files"
    for _ in range(1024):
        csv_data = np.random.rand(1024).astype(np.float) #represents one column of data
        f.write(csv_data.tobytes())

#open the array as a memory-mapped file
column_mmap = np.memmap('column_data.dat', dtype=np.float)

#read some data
print(np.mean(column_mmap[0:1024]))

#write some data
column_mmap[0:512] = .5

#deletion closes the memory-mapped file and flush changes to disk.
#  del isn't specifically needed as python will garbage collect objects no
#  longer accessable. If for example you intend to read the entire array,
#  you will need to periodically make sure the array gets deleted and re-created
#  or the entire thing will end up in memory again. This could be done with a
#  function that loads and operates on part of the array, then when the function
#  returns and the memory-mapped array local to the function goes out of scope,
#  it will be garbage collected. Calling such a function would not cause a
#  build-up of memory usage.
del column_mmap

#write some more data to the array (not while the mmap is open)
with open('column_data.dat', 'ab') as f:
    #for 1024 "csv files"
    for _ in range(1024):
        csv_data = np.random.rand(1024).astype(np.float) #represents one column of data
        f.write(csv_data.tobytes())