pyspark ML LabeledPoint not working with LinearRegression

user_1686236 0
Viewed 270

I'm studying Spark 3.0.1 with pyspark, and have setup some data for simple OLS regression using

data = results.select('OrderMonthYear', 'SaleAmount').rdd.map(lambda row: LabeledPoint(row[1], [row[0]])).toDF()

The OrderMonthYear is my feature column (int), and SaleAmount is the response (float). The LabeledPoint method was imported from pyspark.mllib.regression. I then try to fit the regression model with

from pyspark.ml.regression import LinearRegression
lr = LinearRegression()
modelA = lr.fit(data, {lr.regParam:0.0})

to get this exception

IllegalArgumentException: requirement failed: Column features must be of type struct<type:tinyint,size:int,indices:array<int>,values:array<double>> but was actually struct<type:tinyint,size:int,indices:array<int>,values:array<double>>.

This is clearly not very helpful, as the required and passed features seem to be the same structs. I've searched online, and only found answers to this problem for java, or for someone building the struct themselves. The exception was thrown from a util function that was just throwing a java exception (#Hide where the exception came from that shows a non-Pythonic JVM exception message.), so I can't debug further.

apache-spark-mllib apache-spark-ml pyspark apache-spark python
2 Answers

MLlib and RDD-based MLlib functions are deprecated. I suggest using vector assembler of ML:

from pyspark.ml.feature import VectorAssembler
from pyspark.ml.regression import LinearRegression

data = spark.createDataFrame([[0,1],[1,2],[2,3]]).toDF('OrderMonthYear', 'SaleAmount')

va = VectorAssembler(inputCols=['SaleAmount'], outputCol='features')
data2 = va.transform(data)

lr = LinearRegression(labelCol='OrderMonthYear')
model = lr.fit(data2)
user_141657300

For anyone else following the same LI Learning course, based on some modifications to the accepted answer above to align more with what I was seeing in the course, here's what Cmd 4 cell should look like:

# convenience for specifying schema
from pyspark.ml.feature import VectorAssembler
 
data = VectorAssembler(inputCols=['OrderMonthYear'], outputCol='features').transform(results.select("OrderMonthYear", "SaleAmount")).drop('OrderMonthYear').withColumnRenamed('SaleAmount', 'label')
  
display(data)

Alternatively, you can use the following which also works:

from pyspark.ml.linalg import Vectors
data = results.rdd.map(lambda r: (Vectors.dense(r[0]), r[1])).toDF(["features","label"])
display(data)

Then you should be good to go. Note that you'll want to make the same changes to Cmd 4 in notebooks 4.4 and 4.5 as well. Hope this helps!

user_86052250
Related