How to concatenate multiple columns into single column (with no prior knowledge on their number)?

user_7918575 0
Viewed 10206

Let say I have the following dataframe:

agentName|original_dt|parsed_dt|   user|text|
+----------+-----------+---------+-------+----+
|qwertyuiop|          0|        0|16102.0|   0|

I wish to create a new dataframe with one more column that has the concatenation of all the elements of the row:

agentName|original_dt|parsed_dt|   user|text| newCol
+----------+-----------+---------+-------+----+
|qwertyuiop|          0|        0|16102.0|   0| [qwertyuiop, 0,0, 16102, 0]

Note: This is a just an example. The number of columns and names of them is not known. It is dynamic.

apache-spark-sql apache-spark scala
5 Answers

There may be syntax errors in my answer. This is useful if you are using java<8 and spark<2.

String columns=null
For ( String columnName : dataframe.columns())
{
Columns = columns == null ? columnName : columns+"," + columnName;
}

SqlContext.sql(" select *, concat_ws('|', " +columns+ ") as complete_record " +
"from data frame ").show();
user_48735700
Related