How to input all the featues in one vector column

user_9576413 0
Viewed 62

Having a data frame like this :

|   X1  |   X2  |   X3  |
| ----- | ----- | ----- |
| 11    | 12    | 4     |
| 22    | 16    | 10    |
| 12    | 17    | 75    |
| 42    | 3     | 6     |
            .
            .
            .

How can I transform it to this :

|  Attributes   | label |
| ------------- | ----- |
| (11,22,12,42) | 0     |
| (12,16,17,3)  | 0     |
| (4,10,75,6 )  | 0     |
            .
            .
            .

Each column became a vector... I know how to do the opposite, which is putting all the values from each row in a vector using the VectorAssemblerbut I don't know how to put all the values from each column in a vector like that.

Any help would be highly appreciated.

pyspark pandas dataframe python
2 Answers

You could do this, however

In [33]: pd.DataFrame([[tuple(df[c]), 0] for c in df], columns=['Attributes', 'label'])
Out[33]: 
         Attributes  label
0  (11, 22, 12, 42)      0
1   (12, 16, 17, 3)      0
2    (4, 10, 75, 6)      0

However, it's not efficient to be storing tuples like this in a dataframe.

user_21372550

With pyspark , you can stack and then aggregate with groupby:

(df.selectExpr("stack(3,'X1',X1,'X2',X2,'X3',X3) as (cols,vals)")
.groupBy("cols").agg(F.collect_list("vals").alias("Attribute"))
.withColumn("label",F.lit(0))).show()

+----+----------------+-----+
|cols|       Attribute|label|
+----+----------------+-----+
|  X1|[11, 22, 12, 42]|    0|
|  X3|  [4, 10, 75, 6]|    0|
|  X2| [12, 16, 17, 3]|    0|
+----+----------------+-----+
user_98406370
Related