Schema-less JSON to Apache Beam "Row" type?

Question

Is there a way to convert arbitrary schema-less JSON strings into Apache Beam "Row" types using the Java SDK? If not, is it possible to derive a Beam Schema type from an existing Object?

I've found the documentation for JsonToRow and ParseJsons, but they either require a Schema or POJO class to be provided in order to work. I also found that you can read JSON strings into a BigQuery TableRow but there doesn't seem to be a way to convert TableRow into Row that doesn't involve already having a schema.

user_582333 · Answer

No, this is not possible as Row (and the frameworks that use it) require knowing the schema at construction time. On option is, at construction time, to read a small portion of your data to infer your schema, and use this to invoke your JsonToRow transform.

user_477563 · Answer

Unfortunately, the generic answer appears to be "no", although there are some specific situations where the answer may be "yes".

The issue is that Schemas aren't 100% compatible with JSON data types, specifically because of the ARRAY field type.
In JSON, elements of a list may have different data types, but Schemas in Beam require each element to be of an ARRAY to be the same type. This type can be another ROW, or even a logical type, but all elements must be the same.
Unfortunately, using a ROW to replace an ARRAY doesn't entirely work. Although ROW fields are positional, they're also named, making them closer to a MAP. Furthermore, if your dataset contains JSON lists of differing lengths, you'll end up with each Row having a different Schema which will have undesirable consequences.

So if your JSON data doesn't use lists of arbitrary types, you should be ok. That said, Beam doesn't provide any utilities for deriving schemas from JSON, so you'll need to create that solution yourself.

user_2910265 · Answer

I'm running into this exact situation at work, and I second everything you said about limited JSON processing options. It turns out that Beam does provide something that can convert a BigQuery TableRow/TableSchema to Beam Row/Schema. If you choose that as your schema and proceed with JsonToRow, you'll need to bridge the data type impedance mismatch between JSON and Beam data types produced by BigQueryUtils.fromTableSchema (BYTES, DATETIME, logical types generated by BigQueryUtils).

Schema-less JSON to Apache Beam "Row" type?

3 Answers