Install Scrapy in apache airflow will cause INVALID_ARGUMENT

user_12991382 0
Viewed 697

I`m trying to install Scrapy from PyPi using below command.

gcloud composer environments update $(AIRFLOW_ENVIRONMENT_NAME) \
    --update-pypi-packages-from-file requirements.txt \
    --location $(AIRFLOW_LOCATION)

requirements.txt is like this.

google-api-python-client==1.7.*
google-cloud-datastore==1.7.*
Scrapy==2.0.0

After running gcloud command, It will cause an invalid argument but it runs successfully in the local environment.

gcloud composer environments update xxxx \
        --update-pypi-packages-from-file requirements.txt \
        --location asia-northeast1
ERROR: (gcloud.composer.environments.update) INVALID_ARGUMENT: Found 1 problem:
        1) Error validating key Scrapy. PyPi dependency name is not formatted properly. It must be lowercase and follow the format of 'identifier' specified in PEP-508.

Is there any way to install?

google-cloud-composer airflow scrapy google-cloud-platform
2 Answers

As the previous answer stated, the error that you are receiving now is quite clear and it's caused by the wrong formatting of the dependency. It should be scrapy==2.0.0 instead of Scrapy==2.0.0 inside the requirements.txt.

I would like to add that to avoid the installation error when you fix the formatting, you should add one more dependency to your list and that is attrs==19.2.0. I was able to install your requirements to my environment by specifying the following list:

google-api-python-client==1.7.*
google-cloud-datastore==1.7.*
scrapy==2.0.0
attrs==19.2.0
user_79031590

Even though you adjust package name in requirements.txt file according to PEP-508 document prerequisites, formatting certan package name in lowercase layout scrapy==2.0.0, the issue most probably will remain the same and updating process will stuck with the error:

Failed to install PyPI packages

Generally, this kind of error appears then the source PyPI package has some external dependencies or this package is sensitive on some system-level libraries that GCP Composer doesn't support.

In this case a vendor recommends two ways either using KubernetesPodOperator to build own custom image and use it in particular Kubernetes Pod or deploy PyPi package as a local Python library, uploading shared object libraries for the PyPI dependency to Airflow /plugins directory, find more info here.

user_99288090
Related