Spark java dataframe String cannot be converted to struct

user_3311616 0
Viewed 640

I have the below spark schema defined

        StructType state = DataTypes.createStructType(
                new StructField[] {
                        DataTypes.createStructField("version", DataTypes.IntegerType, false),
                        DataTypes.createStructField("value", DataTypes.StringType, false)
                        });

        ArrayType relationship = DataTypes.createArrayType(DataTypes.createStructType(
                new StructField[] {
                        DataTypes.createStructField("cid", DataTypes.StringType, false),
                        DataTypes.createStructField("state", state, false),
                        }));

        StructType cr = DataTypes.createStructType(
                new StructField[] {
                        DataTypes.createStructField("cmg", relationship, false)
                });

        StructType schema = DataTypes.createStructType(
            new StructField[] {
                    DataTypes.createStructField("cr", cr, false)
            });

If I create the dataframe as

Row r1 = RowFactory.create("{cr:{cmg:[{cid:\"B06XW5BXJZ\",state:{version:19,value:"approved"}}]}}");
List<Row> rowList = ImmutableList.of(r1);
Dataset<Row> df = spark.sqlContext().createDataFrame(rowList, schema);

The code gives below error

The value ({cr:{cmg:[{cid:"B06XW5BXJZ",state:{version:19,value:"approved"}}]}}) of the type (java.lang.String) cannot be converted to struct<cmg:array<struct<cid:string,state:struct<version:int,value:string>>>>

What am I missing?

apache-spark-sql apache-spark
1 Answers

When you execute createDataFrame(rowList, schema) Spark tries to interpret the content of each element in rowList using the provided schema. However, the values in rowList are strings, and not structured objects, so Spark is unable to apply the schema.

You have various options to load that object into a dataframe in structured form.

Load the data as json string and use spark to parse it

String jsonRow = "{cr:{cmg:[{cid:\"B06XW5BXJZ\",state:{version:19,value:\"approved\"}}]}}";
Dataset<Row> df = spark.createDataset(List.of(jsonRow), Encoders.STRING())
            .select(functions.from_json(functions.col("value"), schema, Map.of("allowUnquotedFieldNames", "true")));

in this case it first creates a Dataset<String> in which each row contains a single String column (value) and then uses the from_json spark sql function to parse the json using your schema.

Also note the use of the allowUnquotedFieldNames=true option, required because in the input string the field names are not quoted.

Manually create structured rows and load them in a Dataframe

Row structuredRow = RowFactory.create(RowFactory.create(List.of(RowFactory.create("B06XW5BXJZ", RowFactory.create(19, "approved")))));
Dataset<Row> df = spark.createDataFrame(List.of(structuredRow), schema);

This extends your initial attempt to use the RowFactory to manually create the rows. The rows must reflect the structure defined in the schema (or rather, the schema must respect the structure of the rows).

Use a custom Java bean class

Class definitions
public static class State implements Serializable {
    private Integer version;
    private String value;
    // getters, setters, constructors
}

public static class Relationship implements Serializable {
    private String cid;
    private State state;
    // getters, setters, constructors
}

public static class Cr implements Serializable {
    private List<Relationship> cmg;
    // getters, setters, constructors
}

public static class RowBean implements Serializable {
    private Cr cr;
    // getters, setters, constructors
}
Use the bean class to create a Dataset
RowBean row = new RowBean(new Cr(List.of(new Relationship("B06XW5BXJZ", new State(19, "approved")))));
Dataset<RowBean> ds = spark.createDataset(List.of(row), Encoders.bean(RowBean.class));

In this case, using a custom Java bean / Scala case class, the schema is extracted directly from the class structure using Encoders.bean()

user_38118950
Related