How to replace substrings in column of type String?

user_277434 0
Viewed 5028

I'm trying to convert the following Scala line (which extracts the digits from a string and which I used in the Scala shell):

val df2 = df.withColumn("only digits", regexp_replace(col("original"), "[^\\d]", ""))

to its Java equivalent and I am having problems with the conversion from org.apache.spark.sql.Column returned by col("original") to the String type required by String.replaceAll().

I tried the following, but it doesn't compile (cast still returns a Column).

import org.apache.spark.sql.Column;
import static org.apache.spark.sql.functions.*;
Dataset<Row> df2 = df.withColumn("new", col("original").cast("string").replaceAll("[^\\d]", ""));

I also looked at the Column javadocs and in the static functions imported above but haven't seen anything to help. Thanks.

apache-spark-sql apache-spark dataframe java
2 Answers

I have no words to say, @Jacek Laskowski was very explanatory and focus exactly at the point of your question. Nevertheless for the sake of a completeness i will provide an example of how this done with java, to be more understandable.

  SparkSession spark = SparkSession
                .builder()
                .appName("Java Spark SQL basic example")
                .config("spark.master", "local")
                .getOrCreate();
        JavaSparkContext jsc = new JavaSparkContext(spark.sparkContext());


        List<Row> data = Arrays.asList(
                RowFactory.create(1, "Hi I heard about *Spark"),
                RowFactory.create(2, "I wish *Java could use case classes"),
                RowFactory.create(3, "Logistic regression models are neat")
        );
        StructType schema = new StructType(new StructField[]{
                new StructField("label", DataTypes.IntegerType, false, Metadata.empty()),
                new StructField("sentence", DataTypes.StringType, false, Metadata.empty())
        });
        Dataset<Row> sentenceData = spark.createDataFrame(data, schema);
        Dataset<Row> transformedDS=sentenceData.withColumn("Replaced_Sentence", 
                functions.regexp_replace(sentenceData.col("sentence"),"\\*",""));
        transformedDS.show(false);

Output is:

+-----+-----------------------------------+-----------------------------------+
|label|sentence                           |Replaced_Sentence                  |
+-----+-----------------------------------+-----------------------------------+
|1    |Hi I heard about *Spark            |Hi I heard about Spark             |
|2    |I wish *Java could use case classes|I wish Java could use case classes |
|3    |Logistic regression models are neat|Logistic regression models are neat|
+-----+-----------------------------------+-----------------------------------+
user_31638240
Related