Airflow 1.10.9 - cannot import name '_check_google_client_version' from 'pandas_gbq.gbq'

user_1872639 0
Viewed 2073

I am currently using Airflow 1.10.9 on ECS. I explicitly specify Airflow version in my requirements.txt file as below.

apache-airflow[aws,celery,crypto,gcp,jdbc,mysql,password,postgres,slack,statsd]==1.10.9

After pandas_gbq.gbq 0.15.0 is released, our Airflow cluster has showed this error on Web UI.

cannot import name '_check_google_client_version' from 'pandas_gbq.gbq'

This is because bigquery_hook import _check_google_client_version from pandas_gbq but it was deleted since 0.15.0.

https://github.com/apache/airflow/blob/1.10.9/airflow/contrib/hooks/bigquery_hook.py#L40

https://github.com/pydata/pandas-gbq/blob/0.14.1/pandas_gbq/gbq.py#L37 https://github.com/pydata/pandas-gbq/blob/0.15.0/pandas_gbq/gbq.py

I tried to install pandas_gbq 0.14.1 by specifying the version of pandas-gbq as below but it does not work. pandas-gbq 0.15.0 was installed on the container.

pandas==0.25.3
pandas-gbq==0.14.1
apache-airflow==1.10.9

Would you help me install pandas-gbq 0.14.1 on my Airflow container?

airflow pandas google-bigquery python
1 Answers

In order to install airflow in repeatable way you need to follow the approach with constraints: http://airflow.apache.org/docs/apache-airflow/stable/installation.html#installation-script - note that you are using rather old version of Airflow so 1.10.9 constraints will be rather old, I'd recommend you to upgrade to later version of Airflow.

You can also prepare such a constraints file yourself from your installation pip freeze > constraints.txt and then you can modify the file and set the pandas_gbq to 0.14.1 (and then use that constraint file with --constraint flag). This will give you the exact versions of the dependencies you already have and force installation of 0.14.1 for pandas-gbq

user_5167010
Related