How to .dot in pyspark (AttributeError: 'DataFrame' object has no attribute 'dot')

user_7585973 0
Viewed 55

In pandas we know df1.dot(df2.T) for dot product but when I run in pySpark

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-7-2219b97587ee> in <module>
----> 1 df1.dot(df2.T)

/opt/cloudera/parcels/CDH-7.1.3-1.cdh7.1.3.p0.4992530/lib/spark/python/pyspark/sql/dataframe.py in __getattr__(self, name)
   1302         if name not in self.columns:
   1303             raise AttributeError(
-> 1304                 "'%s' object has no attribute '%s'" % (self.__class__.__name__, name))
   1305         jc = self._jdf.apply(name)
   1306         return Column(jc)

AttributeError: 'DataFrame' object has no attribute 'dot'
pyspark pandas python
1 Answers

Have you tried pandas-on-spark?

import pyspark.pandas as ps
ps.set_option('compute.ops_on_diff_frames', True)

df = ps.DataFrame([[0, 1, -2, -1], [1, 1, 1, 1]])
s = ps.Series([1, 1, 2, 1])

print(df @ s)

0   -4
1    5
dtype: int64

Notes

  • pyspark.pandas requires pyspark >= 3.2
  • The dot product works only for operations between Dataframe (dot) Series. So you cannot use it for something like: df @ df
  • To convert from spark RDD to pandas-on-spark you can use something like: pdf = df.to_pandas_on_spark()
  • To go back from pandas-on-spark to spark RDD, use: pdf.to_spark()
user_175870020
Related