I'm deploying a dataflow template in python from my local virtual environment, which threw a bunch of unintelligible issues that end like this:
Discarding https://files.pythonhosted.org/packages/e0/e6/d14b4a2b54ef065b1a2c576537abe805c1af0c94caef70d365e2d78fc528/pyarrow-0.15.1.tar.gz#sha256=7ad074690ba38313067bf3bbda1258966d38e2037c035d08b9ffe3cce07747a5 (from https://pypi.org/simple/pyarrow/). Command errored out with exit status 1: \'C:\\Users\\PhuongAnhNguyenVenef\\AppData\\Local\\Programs\\Python\\Python37\\python.exe\' \'C:\\Users\\PhuongAnhNguyenVenef\\AppData\\Local\\Programs\\Python\\Python37\\lib\\site-packages\\pip\' install --ignore-installed --no-user --prefix \'C:\\Users\\PHUONG~1\\AppData\\Local\\Temp\\pip-build-env-kj19czll\\overlay\' --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- setuptools wheel setuptools_scm \'cython >= 0.29\' Check the logs for full command output.\r\nERROR: Could not find a version that satisfies the requirement pyarrow<3.0.0,>=0.15.1 (from apache-beam[gcp])\r\nERROR: No matching distribution found for pyarrow<3.0.0,>=0.15.1\r\n'
This is my requirements file:
apache-beam[gcp]==2.28.0
pandas
numpy
google-cloud==0.34.0
google-cloud-storage==1.33.0
google-cloud-bigquery==1.28.0
pyarrow==0.17.1
fsspec==0.8.4
geopy~=1.21.0
When I run the Dataflow job locally using DirectRunner, it ran successfully without any error. I can also install the entire requirements file without any issue as well. After the first fail, I commented out pyarrow and fsspec, reinstall CPython, but the issue still remains. I tried deploying the template on CloudShell, which succeeded, but the job failed with this error
"Error syncing pod d18cc4b816792b6af6e1c00dd0ced7fb ("dataflow-create-utilities-177a775e-04131228-8n1u-harness-2rxs_default(d18cc4b816792b6af6e1c00dd0ced7fb)"), skipping: failed to "StartContainer" for "python" with CrashLoopBackOff: "back-off 5m0s restarting failed container=python pod=dataflow-create-utilities-177a775e-04131228-8n1u-harness-2rxs_default(d18cc4b816792b6af6e1c00dd0ced7fb)""
which seems to be caused by dependencies installation as well:
ERROR: Could not find a version that satisfies the requirement wheel (from versions: none)
So my question is: how can I deploy this template with these packages? It is just bizzare that DirectRunner works and pip install works but not when I deploy the template.
EDIT:
I changed pyarrow version to 1.0.1 which is what I need. The issue remains.