GCP - AI Platform notebook schedule

user_18172561 0
Viewed 381

I am relatively new to GCP and am trying to schedule a notebook on GCP to run everyday. This notebook has dependencies in terms of libraries and other python modules/scripts. When I schedule this with the Cloud Scheduler (as shown in image), there are errors shown in logs at import statements of libraries and while importing other python modules.

schedule_on_gcp

I also created a requirements.txt file, but the scheduler doesn't seem to be reading it.
Am I doing something wrong?

Can anyone help or guide me with some possible solutions? Been stuck with this since a few days, any help would be highly appreciated.

PS- Cloud Functions would be by last option incase I'm not able to run this way.

google-cloud-scheduler gcp-ai-platform-notebook scheduler google-cloud-platform python
2 Answers

The problem is that we have 2 different environments:

  1. Notebook document itself
  2. Docker container that Notebook Executor uses when you click on Execute: a Docker container is passed to Executor backend (Notebooks API + Vertex Custom Job) and since you are installing the dependencies in the Notebook itself (Managed Notebook underlying infra), these are not included in the container, hence this fails. You need to pass a container that includes Selenium.

If you need to build a custom container I would do the following:

  1. Create a custom container
# Dockerfile.example
FROM gcr.io/deeplearning-platform-release/tf-gpu:latest
RUN pip install -y selenium

Then you’ll need to build and push it somewhere accessible.

PROJECT="my-gcp-project"
docker build . -f Dockerfile.example -t "gcr.io/${PROJECT}/tf-custom:latest"
gcloud auth configure-docker
docker push "gcr.io/${PROJECT}/tf-custom:latest"
  1. Specify the container when launching the Execution "Custom Container"

enter image description here

user_2608260

The error means that you are missing the selenium module, you need to install it. You can use the following commands to install it:

python -m pip install -U selenium (you need pip installed)

pip install selenium

or depending on your permissions:

sudo pip install selenium

For python3:

sudo pip3 install selenium

Edit 1:

If you have selenium installed, check where you have Python located and where the Python looks for libraries/packages, including the ones installed using pip. Sometimes Python runs from a location, but looks for libraries in a different location. Make sure Python is looking for the libraries in the right directory.

Here is an answer that you can use to check if Python is configured correctly.

user_175443120
Related