How to get sum of product from columns in 2 data frame using PySpark

user_1254579 0
Viewed 132

There are 2 Data Frames one contain the price information as follows (day 1 to Day 100 as 100 rows),

 StoreId,ItemID, Date,Price
  HH-101,item1, d_1,      €9
  HH-101,item1, d_2,      €7
  ……………………………
  DH-101,item1, d_90,     €4
  ……………………………

  HH-101,item1, d_100,    €3

The 2nd data frame is a sales Information as shown (day 1 to Day 100 as 100 columns but 1 row)

Stored_ID, ItemID,   d-1, d-2, d-3,……. d-90,d-100
HH-101     , item1  , 2 , 4  , 0,………..,12  ,22
HH-101     , item2  , 1 , 0  , 3 ……………,3   ,3

What is the optimum PySpark script to produce another Data frame

with new column, which has the summation of

number of unit * Sales price , corresponding to each item

  example for store HH-101 and  item1
  
      2*9+ 4*7+........+.....+...12*4+22*3

Is there any single step instead of wrting sum of product for more than 100 column

pyspark apache-spark dataframe scala python
1 Answers

Here's a simpler example derived from your sample dataframes. I think it should also be scalable to your real data.

df1.show()
+-------+------+----+-----+
|StoreId|ItemID|Date|Price|
+-------+------+----+-----+
| HH-101| item1| d_1|   €9|
| HH-101| item1| d_2|   €7|
+-------+------+----+-----+

df2.show()
+-------+------+---+---+
|StoreId|ItemID|d_1|d_2|
+-------+------+---+---+
| HH-101| item1|  2|  4|
| HH-101| item2|  1|  0|
+-------+------+---+---+

You can unpivot df2 using stack with a query string generated from a list comprehension of the column names, then join to df1 using the first 3 columns, group by the store id and item id, and get the sum of price * number.

result = df2.selectExpr(
    'StoreId', 'ItemID',
    'stack(2, ' + ', '.join(["'%s', %s" % (c, c) for c in df2.columns[2:]]) + ') as (Date, Number)'
    # "stack(2, 'd_1', d_1, 'd_2', d_2) as (Date, Number)"
).join(
    df1, df1.columns[:3]
).groupBy(
    'StoreId', 'ItemID'
).agg(
    F.expr('sum(Number * float(substr(Price, 2))) as Total')
)

result.show()
+-------+------+-----+
|StoreId|ItemID|Total|
+-------+------+-----+
| HH-101| item1| 46.0|
+-------+------+-----+
user_141657300
Related