Spark SQL - Select yields AttributeError: 'module' object has no attribute 'api'

user_9010901 0
Viewed 1427

Good day,

I'm using a base install of Azure HDinsight cluster, configured for spark. I'm in Jupyter Notebook, PySpark.

Working in the provided 00 - [READ ME FIRST] PySpark Kernel Features.ipynb file, I've found the following error/bug when executing a spark sql 'SELECT':

AttributeError: 'module' object has no attribute 'api'

Executed code:

%%sql -o query1

SELECT clientid, querytime, deviceplatform, querydwelltime

FROM hivesampletable

WHERE state = 'Washington' AND devicemake = 'Microsoft'

I've had the same error show up when using SELECT in other code. Since the code noted here appears in the provided baseline 'tutorial', I expect its not a coding error. I've had the same error in PySpark and PySpark 3 kernel.

Does anyone have any experience/advice/suggestions they can share?

Trace:

AttributeErrorTraceback (most recent call last) /usr/bin/anaconda/lib/python2.7/site-packages/IPython/core/formatters.pyc in call(self, obj)

 902                 pass

 903             else:

 --> 904                 printer(obj)

 905                 return True

 906             # Finally look for special method names

/usr/bin/anaconda/lib/python2.7/site-packages/autovizwidget/widget/utils.pyc in display_dataframe(df)

114 

115 def display_dataframe(df):

 --> 116     selected_x = select_x(df)

117     selected_y = select_y(df, selected_x)

118     encoding = Encoding(chart_type=Encoding.chart_type_table, x=selected_x, y=selected_y,

/usr/bin/anaconda/lib/python2.7/site-packages/autovizwidget/widget/utils.pyc in select_x(data, order)

 70         _validate_custom_order(order)

 71 

 ---> 72     d = _classify_data_by_type(data, order)

 73 

 74     chosen_x = None

/usr/bin/anaconda/lib/python2.7/site-packages/autovizwidget/widget/utils.pyc in _classify_data_by_type(data, order, skip)

 48     for column_name in data:

 49         if column_name not in skip:

 ---> 50             typ = infer_vegalite_type(data[column_name])

 51             d[typ].append(column_name)

 52

/usr/bin/anaconda/lib/python2.7/site-packages/autovizwidget/widget/utils.pyc in infer_vegalite_type(data)

 14     """

 15 

 ---> 16     typ = pd.api.types.infer_dtype(data)

 17 

 18     if typ in ['floating', 'mixed-integer-float', 'integer',

AttributeError: 'module' object has no attribute 'api'

pyspark-sql azure-hdinsight azure
3 Answers

The notebook is using version 0.17.1 of pandas but the autovizwidget depends on a later version of pandas that has the 'api' module. I've been told that this will be resolved in a subsequent release of HDInsight configs.

ssh into the cluster and run the following:

sudo -HE /usr/bin/anaconda/bin/conda install pandas

user_12386030

Had the same issue. I used:

pip install pandas --upgrade --user

via the terminal available in the jupyter notebook.

user_93383920
Related