I have the json file for which I have created a dataframe in pyspark.
Here is the json file content:
{"xyz": [ {"c1": "a", "c2": "b", "c3": "d"}]}
Here is the Structype schema that I have created.
schema = StructType([
StructField("abc",ArrayType(
StructType([
StructField("c1",StringType()),
StructField("c2",StringType()),
StructField("c3",StringType())])))])
```
rdd = sc.textFile(path).map(lambda x: x.encode("ascii", "ignore")).map(lambda line: json.loads(line))
df = rdd.toDF(schema=schema)`
df_colsexp = df.select(
col('cards.c1').alias('c1'),
col('cards.c2').alias('c2'),
col('cards.c3').alias('c3')
)
df_colsexp.show(5,False)`
df_colsexp.printSchema()
```
>>> df_colsexp.printSchema()
root
|-- c1: array (nullable = true)
| |-- element: string (containsNull = true)
|-- c2: array (nullable = true)
| |-- element: string (containsNull = true)
|-- c3: array (nullable = true)
| |-- element: string (containsNull = true)
df_colsexp.show(5,False)
Output:
c1 c2 c3
[a] [b] [c]
My question is why c1,c2,c3 columns are shown as array in the output? How can make these columns c1,c2,c3 as a Stringtype? So that in the output those square brackets will go away. ?