Permanently saving train data in google colab

user_13844105 0
Viewed 2365

I have train data for 50GB.

My google drive capacity was 15GB so I upgraded it to 200GB and I uploaded my train data to my google drive

I connected to colab, but I can not find my train data in colab session, So I manually uploaded to colab which has 150GB capacity.

It says, it will be deleted when my colab connection is off.

It is impossible to save train data for colab permanently? And colab is free for 150GB?

And I see colab support nvidia P4 that is almost 5000$. can I use it 100% or it is shared to some portion(like 0.1%) to me? (When P4 is assigned to me)

python
2 Answers

The way you can do this is to mount your google drive into colab environment. Assume your files are kept under a folder named myfolder in your google drive. This is what I would suggest, do this before you read/write any file:

import os
from google.colab import drive
MOUNTPOINT = '/content/gdrive'
DATADIR = os.path.join(MOUNTPOINT, 'My Drive', 'myfolder')
drive.mount(MOUNTPOINT)

then, for example, your file bigthing.zip reside under myfolder in your google drive will be available in colab as path=os.path.join(DATADIR, 'bigthing.zip')

Similarly, when you save a file to a path like the above, you can find your file in Google Drive under the same directory.

user_92145170

In regards to the final questions, you are able to use it 100%, however, there are very inconsistent restrictions. Generally, you only get about 8 hours straight before you get kicked off, must be running code to keep the connection, and you can only use a GPU a few times in a row before you lose access for a day or so. You can pay for colab pro which would give you more access, and better GPUs in general for $10/month.

In my experience, before colab pro you could get a top GPU (Tesla P100) about 50% of the time. Now that they started the pro version I rarely get a P100 and get kicked off more often. So it can be a bit of a game to get regular use.

Another site that lets you do basically the same thing is https://console.paperspace.com/ They give you only 6 hour shifts on "notebook" but you wont get kicked off before then, and I can usually get a P5000 which is generally better than colab gives me.

https://www.kaggle.com/ will also give you 30 hours per week, so you really could get up to near 2 GPU hours for every hour of the day if you planned your life around it.

user_122883380
Related