Currently, I am creating .npz compressed files for storing large NumPy arrays. So every time I need to load the array from file and since it's a frequent process, I was thinking to store the NumPy array in the database. I am using the PostgreSQL database.
1 Answers
You could do this with a bytea column (it can store arbitrary binary). You can use pickle.dumps to turn your numpy array into a binary string and insert into postgres however you like. You can then go select that data and use pickle.loads to get your array back. Here's an answer to a similar question: https://stackoverflow.com/a/57644761/7386185
Depending on how large the array is, you might want to consider some kind of blob storage like amazon S3.
If you're frequently accessing this data and this is a production environment, you might want to consider keeping this data in-memory. If your array is large enough that you can't keep it around in memory too long, then you should consider if your application will allow for streaming of your data in batches or a buffer.
Related