are Beam Schemas relevant in Python?

user_783314 0
Viewed 534

I'm reading the Apache Beam programming guide, which starts off very excellent but becomes a bit harder to get through starting with the Schemas section.

My main question here is: Are schemas relevant if you are using Beam in Python? It seems like they might only be relevant if you are using a strongly typed language like Java, but I'm not sure. And while the programming guide is good about using different wording for Java vs Python early on in the guide, once you get to the Schemas section it is focused entirely on Java. So it's hard for me to tell if this is a topic I should know anything about if I am using Python.

Here is the section of the guide I am asking about: https://beam.apache.org/documentation/programming-guide/#schemas

apache-beam
1 Answers

There is a direct mapping between Python NamedTuple and Java's Schema. For Python it's especially important in the cross-language part of Beam (right now JDBC and Snowflake IOs) For example Java's

Schema javaSchema = Schema.builder().addInt32Field("f_int32").build()

Has its equivalent in Python:

PythonSchema = NamedTuple("PythonSchema", [ ("f_int32", int) ]

The same about Row and the PythonSchema instances.

When you send the tuples in your Python transform via cross-language, e.g.:

pipeline | GenerateSequence(start=0, end=3)  # return some int sequence
         | Map(lambda x: PythonSchema(f_int32=x)).with_output_types(PythonSchema)
         | WriteToJdbc(...)  # cross-language transform, accepts java Row elements

Then WriteToJdbc will receive in its Java pipeline PCollection with objects equal to

Row.withSchema(javaSchema).addValue(x).build()

Cross-language feature is quite new and very experimental and there are little transforms that use it. But it's considered to be Beam's future and the further Python/Go transforms are going to be the cross-language ones as they require just one (mostly) Java implementation and then just a mapping to the other languages. The native transforms usually use their own data types and don't bother with Schemas AFAIK.

I wrote it in a quite hurry so if something is not clear I will edit it. I hope I helped.

user_79464960
Related