Any time the boolean field is defined properly, all fields in the record become null. The is the exact opposite of the behaviour I would expect. Please, someone, help explain this insanity...
from pyspark.sql import SparkSession
from pyspark.sql.types import StructType, StructField, StringType, BooleanType
spark = SparkSession.builder.appName("Write parquet").master("local[*]").getOrCreate()
sc = spark.sparkContext
schema = StructType([
StructField("sometext", StringType(), True),
StructField("mybool", BooleanType(), True)
]
)
payload = [
{"sometext": "works when mybool isn't None, true or false", "mybool": "not_a_boolean"},
{"sometext": "returns empty when mybool is true", "mybool": True},
{"sometext": "returns empty when mybool is false", "mybool": False},
{"sometext": "returns empty when mybool is None", "mybool": None}]
events = sc.parallelize(payload)
df_with_schema = spark.read.schema(schema).json(events)
df_with_schema.show()
yields:
+--------------------+------+
| sometext|mybool|
+--------------------+------+
|works when mybool...| null|
| null| null|
| null| null|
| null| null|
+--------------------+------+