Read JSON in pySpark with custom schema in GCP Dataproc

user_11247054 0
Viewed 823

In GCP Dataproc (with pySpark), I am doing a task i.e. to read JSON file as per custom schema and load it in a Dataframe.

I do have following sample testing JSON:

{"Transactions": [{"schema": "a",
"id": "1",
"app": "testing",
"description": "JSON schema for testing purpose"}]}

I have created following schema:

custom_schema = StructType([
                      StructField("Transactions",
                         StructType([
                             StructField("schema", StringType()),
                             StructField("id", StringType()),
                             StructField("app", StringType()),
                             StructField("description", StringType())
                                   ])
                            )])

Reading JSON as: df_2 = spark.read.json(json_path, schema = custom_schema)

Getting following results,

Getting following results,

Now, I need to check data in Dataframe, When I try to do df_2.show(), it is taking too much time and show as kernel Busy for hours.

I need help, that what I am missing here in code and how can I see the data in dataframe (Tabular format).

pyspark-dataframes google-cloud-dataproc apache-spark-sql google-cloud-platform json
1 Answers

I think the problem is with your custom schema definition and the JSON file. The following code and JSON file worked for me:

Code

from pyspark.sql import SparkSession
from pyspark.sql.types import StructType
from pyspark.sql.types import StructField
from pyspark.sql.types import StringType
from pyspark.sql.types import ArrayType

spark = SparkSession \
    .builder \
    .appName("JSON test") \
    .getOrCreate()

custom_schema = StructType([
    StructField("schema", StringType(), False),
    StructField("id", StringType(), True),
    StructField("app", StringType(), True),
    StructField("description", StringType(), True)])

df = spark.read.format("json") \
    .schema(custom_schema) \
    .load("gs://my-bucket/transactions.json")

df.show()

JSON file

The contents of gs://my-bucket/transactions.json is:

{"schema": "a", "id": "1", "app": "foo", "description": "test"}
{"schema": "b", "id": "2", "app": "bar", "description": "test2"}

Output

+------+---+---+-----------+
|schema| id|app|description|
+------+---+---+-----------+
|     a|  1|foo|       test|
|     b|  2|bar|      test2|
+------+---+---+-----------+
user_4042640
Related