I recently installed python 3.7 and trying to play around with pyspark, I need to create
glueContext = GlueContext(SparkContext.getOrCreate())
sparkSession = glueContext.spark_session
for which when I import library its just failing to find dynamicframe
When I try to import from awsglue.context import GlueContext I get following error
from awsglue.context import GlueContext
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.7/site-packages/awsglue/__init__.py", line 13, in <module>
from dynamicframe import DynamicFrame
ModuleNotFoundError: No module named 'dynamicframe'
even this one is failing
from awsglue.transforms import *
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.7/site-packages/awsglue/__init__.py", line 13, in <module>
from dynamicframe import DynamicFrame
ModuleNotFoundError: No module named 'dynamicframe'
Pyspark module seems to have no problem importing from pyspark.context import SparkContext
There is no proper documentation on how to setup libraries needed.
Folder structure under /usr/local/lib/python3.7/site-packages/awsglue
├── __init__.py
├── __pycache__
│ ├── __init__.cpython-37.pyc
│ ├── context.cpython-37.pyc
│ ├── data_sink.cpython-37.pyc
│ ├── data_source.cpython-37.pyc
│ ├── functions.cpython-37.pyc
│ ├── gluetypes.cpython-37.pyc
│ ├── job.cpython-37.pyc
│ └── utils.cpython-37.pyc
├── context.py
├── data_sink.py
├── data_source.py
├── devutils.py
├── dynamicframe.py
├── functions.py
├── gluetypes.py
├── job.py
├── transforms
│ ├── __init__.py
│ ├── __pycache__
│ │ ├── __init__.cpython-37.pyc
│ │ ├── apply_mapping.cpython-37.pyc
│ │ ├── collection_transforms.cpython-37.pyc
│ │ ├── dynamicframe_filter.cpython-37.pyc
│ │ ├── dynamicframe_map.cpython-37.pyc
│ │ ├── errors_as_dynamicframe.cpython-37.pyc
│ │ ├── field_transforms.cpython-37.pyc
│ │ ├── relationalize.cpython-37.pyc
│ │ ├── resolve_choice.cpython-37.pyc
│ │ ├── transform.cpython-37.pyc
│ │ ├── unbox.cpython-37.pyc
│ │ └── unnest_frame.cpython-37.pyc
│ ├── apply_mapping.py
│ ├── collection_transforms.py
│ ├── drop_nulls.py
│ ├── dynamicframe_filter.py
│ ├── dynamicframe_map.py
│ ├── errors_as_dynamicframe.py
│ ├── field_transforms.py
│ ├── relationalize.py
│ ├── resolve_choice.py
│ ├── transform.py
│ ├── unbox.py
│ └── unnest_frame.py
└── utils.py