How to make Google Colab communicate with my local machine while running?

Viewed 284

Is this possible, I run my machine learning model on Google Colab using Hosted runtime (using google's GPUs), and make it communicate with my local machine where a process is running and generating new data (which will be fed into the above mentioned ML) and get training temporary results from Google Colab (which will be used to generate more new data)?

If not, how about I change my local machine with google cloud platform?

1 Answers

For the exchange or provision of data, I would suggest a storage that both your local machine and the Colab notebook can access. Either Google Drive or Google Cloud Storage would be a good choice (a notebook with different examples can be found here: https://colab.research.google.com/notebooks/io.ipynb). Since you mention the Google Cloud, PubSub would be a possible service that could be used to communicate between the local machine and the Colab notebook. For cloud storage, triggers can be created so that a PubSub message is published when an item is added (https://cloud.google.com/storage/docs/pubsub-notifications). In the Colab notebook, a pull subscription can now be used to wait for such a message in order to then load the new data from the storage (PubSub pull subscription: https://cloud.google.com/storage/docs/pubsub-notifications).

For the reverse way, i.e. the feedback of the model in Colab to the local machine could be made possible likewise over PubSub. Here, a message can be published from Colab in a second topic and then received on the local machine via pull subscription (sending a message in PubSub: https://cloud.google.com/pubsub/docs/publisher).

Related