When creating a dataframe in pyspark, records with a proper boolean value cause entire row to be null

Viewed 126

Any time the boolean field is defined properly, all fields in the record become null. The is the exact opposite of the behaviour I would expect. Please, someone, help explain this insanity...

from pyspark.sql import SparkSession
from pyspark.sql.types import StructType, StructField, StringType, BooleanType

spark = SparkSession.builder.appName("Write parquet").master("local[*]").getOrCreate()
sc = spark.sparkContext

schema = StructType([
    StructField("sometext", StringType(), True),
    StructField("mybool", BooleanType(), True)
    ]
)

payload = [
    {"sometext": "works when mybool isn't None, true or false", "mybool": "not_a_boolean"},
    {"sometext": "returns empty when mybool is true", "mybool": True},
    {"sometext": "returns empty when mybool is false", "mybool": False},
    {"sometext": "returns empty when mybool is None", "mybool": None}]
events = sc.parallelize(payload)

df_with_schema = spark.read.schema(schema).json(events)
df_with_schema.show()

yields:

+--------------------+------+
|            sometext|mybool|
+--------------------+------+
|works when mybool...|  null|
|                null|  null|
|                null|  null|
|                null|  null|
+--------------------+------+
0 Answers
Related