How to read a text file as one string into Spark DataFrame with Java

user_7551211 0
Viewed 852

I want to create a DF of text files where each row represents a whole txt file in a column named text.

I've tried the following but I got a DF where the text is separated by lines.

Dataset<Row> df = spark.read()
            .textFile("resources/textfile.txt")
            .toDF("text");

Instead of DF of 1 row in the case of 1 file, I've got a DF of 70 rows for this file.

apache-spark-sql apache-spark java
1 Answers

You can collect the dataframe into an array and then join the array to a single string:

import static org.apache.spark.sql.functions.*;

df.agg(collect_list("text").alias("text"))
    .withColumn("text", concat_ws(" ", col("text")))
    .show();
user_21298010
Related