My Google Colab Notebook is not using my 2TB Google Drive space

user_9726037 0
Viewed 1655

I have 2TB storage space on my Google Drive account.

I'd like to use them with Cloud Colab (GPU Python 3).

I've already mounted the Drive space but I'm still stuck with the ~69GB of Colab Storage.

(refer to: https://stackoverflow.com/a/59513316/9726037)

(1) Mounted the Drive space

WORKSPACE_PATH = 'ml/'

drive.mount('/content/drive/', force_remount=True)
ROOT_DIR  = '/content/drive/My Drive/science'
BASE_DIR  = os.path.join(ROOT_DIR, WORKSPACE_PATH)

Output: Mounted at /content/drive/

(2) Downloaded approx 40GB of data from GCS:

bucker_name = 'my_bucket_name'
!gsutil -m cp gs://{bucket_name}/* 'path_to_my_mounted_drive'

Output: 

Copying gs://xxx/test.h5...
Copying gs://xxx/train.h5...
[Errno 28] No space left on device
[Errno 28] No space left on device
[Errno 28] No space left on device
[Errno 28] No space left on device
[Errno 28] No space left on device
[Errno 28] No space left on device
[Errno 28] No space left on device
[Errno 28] No space left on device

When I check the colab storage free space : I only have 3GB/69GB, instead of 2 TB...

SORAGE SPACE

What I did wrong?

google-colaboratory google-drive-api
1 Answers

As I understand, when the drive is mounted, it need to cache the data from Google Drive. Even if you have 2TB on GDrive, you will still have the same limit as before, because the data need to be copied and cached.

So, you may need to copy a part, delete it, then copy the next parts. This may be a bit slower.

user_67290100
Related