I have a PySpark dataframe with this schema:
|-- session_id: string (nullable = true)
|-- user_id: string (nullable = true)
|-- product: map (nullable = true)
| |-- key: string
| |-- value: struct (valueContainsNull = true)
| | |-- seen: long (nullable = true)
| | |-- bought: long (nullable = true)
The product map only records the product key(s) seen on a session_id.
I want to aggregate all keys in one record per user_id. When the same key is repeated on multiple session_id, then I want to sum the seen and bought counts.
For example:
user1|session1|{ham->{1,0}, beer->{1,1}}
user1|session2|{ham->{1,1}, fruit->{1,1}}
Would be aggregated to produce:
user1|{ham->{2,1}, beer->{1,1}, fruit->{1,1}}