Moving local machine docker-compose to GKE, which GCP services to use for persistence volume?

user_1573046 0
Viewed 133

Have a dockerized python script and using docker-compose for local development. Need to move it to GKE but unsure about which GCP services can be used for its persistent volume.

# docker-compose.yaml
version: "3"

services:
  python-obj-01:
    image: python-obj
    build:
      context: .
      dockerfile: dockerfile
    container_name: python-obj-01
    volumes:
      - ./data/inputs:/home/appuser/data/inputs
      - ./data/outputs:/home/appuser/data/outputs
      - ./data/model:/home/appuser/data/obj_detection

There are 3 mount points from my local machine:

  • ./data/input : Reads and process the input data
  • ./data/output: Writes the results
  • ./data/model : Reads the models of input data

In the python script, it will glob to find files from ./data/inputs, processes them, create sub-directory and files in ./data/output.

Besides Cloud Filestore, can Cloud Storage bucket able to fulfill the requirements? or other GCP services?

google-compute-engine google-kubernetes-engine kubernetes docker google-cloud-platform
1 Answers

can Cloud Storage bucket able to fulfill the requirements?

If you requirement is work with a volume like a filesystem the answer is NO

Why?

Cloud Storage is an object-storage and can't be used as a normal filesytem. It should be perfect to storage and read files, but in order to extract all the powerful and features from Cloud Storage, you need to rewrite your application to use cloud storage api, for example.

In GCP by default, the persistent storage is the compute engine persisten disks, you can also use some NFS service, like Firestore.

The best solution depends of how your application works and how the importance for this data for you.

Here you can read more about volumes in GCP.

user_119019580
Related