How to save the results of an np.array for future use when using Google Colab

user_13498967 0
Viewed 9322

I am working on a project of Information Retrieval. For that I am using Google Colab. I am in the phase where I have computed some features ("input_features") and I have the labels ("labels") by doing a for loop, which took me about 4 hours to finish.

So at the end I have appended the results to an array:

input_features = np.array(input_features)
labels = np.array(labels)

So my question would be: Is it possible to save those results in order to use them future purposes when using google colab?

I have found 2 options that maybe could be applied but I don't know where these files are created.

1) To save them as csv files. And my code would be:

from numpy import savetxt
# save to csv file
savetxt('input_features.csv', input_features, delimiter=',')
savetxt('labels.csv', labels, delimiter=',')

And in order to load them:

from numpy import loadtxt
# load array
input_features = loadtxt('input_features.csv', delimiter=',')
labels = loadtxt('labels.csv', delimiter=',')
# print the array
print(input_features)
print(labels)

But still I don't get something back when I print.

2) Save the results of an array by using pickle where I followed these instructions from here: https://colab.research.google.com/drive/1EAFQxQ68FfsThpVcNU7m8vqt4UZL0Le1#scrollTo=gZ7OTLo3pw8M

from google.colab import files
import pickle
def features_pickeled(input_features, results):
  input_features = input_features + '.txt'
  pickle.dump(results, open(input_features, 'wb'))
  files.download(input_features)
def labels_pickeled(labels, results):
  labels = labels + '.txt'
  pickle.dump(results, open(labels, 'wb'))
  files.download(labels)

And to load them back:

def load_from_local():
  loaded_features = {}
  uploaded = files.upload()
  for input_features in uploaded.keys():
      unpickeled_features = uploaded[input_features]
      loaded[input_features] = pickle.load(BytesIO(data)) 
  return loaded_features 
def load_from_local():
  loaded_labels = {}
  uploaded = files.upload()
  for labels in uploaded.keys():
      unpickeled_labels = uploaded[labels]
      loaded[labels] = pickle.load(BytesIO(data))
  return loaded_labes

#How do I print the pickled files to see if I have them ready for use???

When using python I would do something like this for pickle:

#Create pickle file
with open("name.pickle", "wb") as pickle_file:
     pickle.dump(name, pickle_file)
#Load the pickle file
with open("name.pickle", "rb") as name_pickled:
     name_b = pickle.load(name_pickled)

But the thing is that I don't see any files to be created in my google drive.

Is my code correct or do I miss some part of the code?

Long description in order to hopefully have explained in detail what I want to do and what I have done for this issue.

Thank you in advance for your help.

google-colaboratory pickle arrays
2 Answers

Google Colaboratory notebook instances are never guaranteed to have access to the same resources when you disconnect and reconnect because they are run on virtual machines. Therefore, you can't "save" your data in Colab. Here are a few solutions:

  1. Colab saves your code. If the for loop operation you referenced doesn't take an extreme amount of time to run, just leave the code and run it every time you connect your notebook.
  2. Check out np.save. This function allows you to save an array to a binary file. Then, you could re-upload your binary file when you reconnect your notebook. Better yet, you could store the binary file on Google Drive, mount your drive to your notebook, and reference it like that.
user_127328650 
# Mount driver to authenticate yourself to gdrive
from google.colab import drive
drive.mount('/content/gdrive')

#---

# Import necessary libraries
import numpy as np
from numpy import savetxt
import pandas as pd

#---

# Create array
arr = np.array([1, 2, 3, 4, 5])

# save to csv file
savetxt('arr.csv', arr, delimiter=',')  # You will see the results if you press in the File icon (left panel)

And then you can load it again by:

# You can copy the path when you find your file in the file icon
arr = pd.read_csv('/content/arr.csv', sep=',', header=None) # You can also save your result as a txt file
arr
user_134989670
Related