Deploy a Python Cloud Function with all package dependencies

user_10184250 0
Viewed 5891

I would like to deploy a cloud function that doesn't rely on using a requirements.txt to install packages. I want the packages to be available within storage or zipped up and upload as part of the function. Is this possible?

EDIT 6/14/2019

Basically I would like to send packages like numpy and pandas with my code to deploy a cloud function. I want to do this in the event that pypi.org is not available. I have tried following this piece of documentation. An example of what I am trying to do is below:

Folder Structure:

-> my_folder
    -> main.py
    -> libs
        -> numpy (the entire package)
        -> pandas (the entire package)
        -> __init__.py

main.py

import libs.numpy as np
import libs.pandas as pd

def function()
    do stuff with numpy and pandas

I then tried to deploy the function from gcloud command line and then gcp UI, both failed. If this is possible please help.

google-cloud-functions google-cloud-platform python
1 Answers

At the moment there is only two options:

  1. Using the requirements.txt
  2. Packaging the dependencies along with your function, link here

They cannot be zipped neither on storage, they will be treated as part of the source of the function.

If you choose to go with the second option, the parameter -t libs might help you. You can use it to install everything on a libs folder and then you can just move the content to the local directory. As a single command it would look like this:

pip install -t libs [your library name(s)] && rm -rf libs/*.dist-info && mv -r libs/* . && rm -rf libs

I added the rm -rf libs/*.dist-info portion in order to not pollute the source folder with tons of library version and distribution information that are useless to the function. Those are used by pip when freezing and planning updates.

EDIT 6/14/2019

You kept the libraries on the libs folder. That is the point before the mv -r libs/* . on the one-liner that I added above.

Using a libs folder keeps everything more organized, so if you want to keep the packages there you need to vendor that folder adding this to the top of your main.py, before all other imports:

# Vendoring packages from libs folder
import sys
import os
sys.path.insert(1, os.path.join(
    os.path.dirname(os.path.realpath(__file__)),
    "libs"
))
# All other imports go below this line

Explaining:

__file__ is a global variable present in every module that holds the path to the file from which the module was defined, that is the file where it is being used. In our case, the path to main.py.

Since we cannot be certain of the working directory at the moment main.py is imported we pass that to os.path.realpath to be certain of the path structure. Could be os.path.abspath to, I have seen and used both and haven't noticed any difference.

From the path of the file, we get the path of the directory of your source code with os.path.dirname and then to the libs folder inside it with os.path.join.

Now the most important part. When you try to import a package, python looks for them on the system/python path. So we add the libs full path that we built as the first lookup location on the system path after your working directory. New import statements will look on that folder first and the package is not there proceed normally with the rest of the lookup directories.
If you prefer to look for packages on libs only if they are not available in the system and the python environment, append the libs path instead of inserting it at index 1.

After that you don't need to prepend libs. on your imports, just use the normal import numpy.

On fully independent packages this might work, but not on packages with dependencies, since they expect their dependencies to be directly importable (from anywhere on sys.path).

user_98653930
Related