Spark row encoder: empty metadata

user_3737163 0
Viewed 250

I am using spark in Java and I am creating a Dataset of Row from an RDD OF row.

I create the schema using

Metadata meta = new MetadataBuilder().putString("type", "categorical").build();
StructField s = new StructField(name, IntegerType, true, meta);
StructType t = new StructType(new StructField[]{s});  
Encoder<Row> encoder = RowEncoder.apply(t);

and I use it in the dataset like this

ds.flatMap((FlatMapFunction<Row, Row>) this::customFlatMapRow, encoder);

for some reasons after I write the table and I inspect the fields of the schema and their metadata they are empty (despite the fact I created and set them like above). Somehow I am loosing them

apache-spark-sql apache-spark
1 Answers

If you inspect the ExpressionEncoder of the dataset, the metadata is available.

The code 

Metadata meta = new MetadataBuilder().putString("type", "categorical").build();
StructField s = new StructField("col", IntegerType, true, meta);
StructType t = new StructType(new StructField[]{s});
Encoder<Row> encoder = RowEncoder.apply(t);

Dataset<Row> df = spark.createDataset(Arrays.asList(1, 2, 3), Encoders.INT()).toDF("col");
Dataset<Row> df2 = df.flatMap((FlatMapFunction<Row, Row>) r -> Collections.singleton(r).iterator(), encoder);
System.out.println(df2.exprEnc().schema().fields()[0].metadata());

prints

{"type":"categorical"}
user_21298010
Related