I have a virtualenv python3.6 inside a VM at /tmp/dataflow-venvtflya9ij, with apache-beam==2.27.0 installed,
and a script /home/airflow/gcs/dags/advisor/create_dataset/beam_pipelines/dummy.py consisting of only:
import apache_beam
if __name__ == '__main__': print('MAIN')
Running the script I get an error when evaluating the import statement:
airflow@airflow-worker-7fb797d459-nf8gh:~$ /tmp/dataflow-venvtflya9ij/bin/python /home/airflow/gcs/dags/advisor/create_dataset/beam_pipelines/dummy.py
Traceback (most recent call last):
File "/home/airflow/gcs/dags/advisor/create_dataset/beam_pipelines/dummy.py", line 1, in <module>
import apache_beam
File "/tmp/dataflow-venvtflya9ij/lib/python3.6/site-packages/apache_beam/__init__.py", line 95, in <module>
from apache_beam import coders
File "/tmp/dataflow-venvtflya9ij/lib/python3.6/site-packages/apache_beam/coders/__init__.py", line 19, in <module>
from apache_beam.coders.coders import *
File "/tmp/dataflow-venvtflya9ij/lib/python3.6/site-packages/apache_beam/coders/coders.py", line 43, in <module>
from future.moves import pickle
File "/tmp/dataflow-venvtflya9ij/lib/python3.6/site-packages/future/moves/__init__.py", line 8, in <module>
import_top_level_modules()
File "/tmp/dataflow-venvtflya9ij/lib/python3.6/site-packages/future/standard_library/__init__.py", line 810, in import_top_level_modules
with exclude_local_folder_imports(*TOP_LEVEL_MODULES):
File "/tmp/dataflow-venvtflya9ij/lib/python3.6/site-packages/future/standard_library/__init__.py", line 781, in __enter__
module = __import__(m, level=0)
File "/home/airflow/gcs/dags/advisor/create_dataset/beam_pipelines/test.py", line 3, in <module>
from apache_beam.options.pipeline_options import PipelineOptions
File "/tmp/dataflow-venvtflya9ij/lib/python3.6/site-packages/apache_beam/options/pipeline_options.py", line 41, in <module>
from apache_beam.transforms.display import HasDisplayData
File "/tmp/dataflow-venvtflya9ij/lib/python3.6/site-packages/apache_beam/transforms/__init__.py", line 23, in <module>
from apache_beam.transforms import combiners
File "/tmp/dataflow-venvtflya9ij/lib/python3.6/site-packages/apache_beam/transforms/combiners.py", line 45, in <module>
from apache_beam.transforms import core
File "/tmp/dataflow-venvtflya9ij/lib/python3.6/site-packages/apache_beam/transforms/core.py", line 40, in <module>
from apache_beam.coders import typecoders
File "/tmp/dataflow-venvtflya9ij/lib/python3.6/site-packages/apache_beam/coders/typecoders.py", line 198, in <module>
registry = CoderRegistry()
File "/tmp/dataflow-venvtflya9ij/lib/python3.6/site-packages/apache_beam/coders/typecoders.py", line 91, in __init__
self.register_standard_coders(fallback_coder)
File "/tmp/dataflow-venvtflya9ij/lib/python3.6/site-packages/apache_beam/coders/typecoders.py", line 95, in register_standard_coders
self._register_coder_internal(int, coders.VarIntCoder)
AttributeError: module 'apache_beam.coders.coders' has no attribute 'VarIntCoder'
But if I execute the virtualenv interactively and then run the import statement, it works OK:
airflow@airflow-worker-7fb797d459-nf8gh:~$ /tmp/dataflow-venvtflya9ij/bin/python
Python 3.6.10 (default, Feb 1 2021, 12:07:35)
[GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import apache_beam
>>>
In both scenarios, the sys.path is the same:
# --- [interactive run] sys.path
[
'',
'/opt/python3.6/lib/python36.zip',
'/opt/python3.6/lib/python3.6',
'/opt/python3.6/lib/python3.6/lib-dynload',
'/tmp/dataflow-venvtflya9ij/lib/python3.6/site-packages'
]
# --- [script run] sys.path
[
'/home/airflow/gcs/dags/advisor/create_dataset/beam_pipelines',
'/opt/python3.6/lib/python36.zip',
'/opt/python3.6/lib/python3.6',
'/opt/python3.6/lib/python3.6/lib-dynload',
'/tmp/dataflow-venvtflya9ij/lib/python3.6/site-packages'
]
So my question is why am I getting this inconsistent behaviour?
(Comment aside: the import error makes no sense, cause apache-beam is installed in the venv/lib/python3.6/site-packages and the coders class has the mentioned attribute.)