I have DynamoDb json files. which have data like following
text = sqlContext.read.text('dynamodb/*')
text.take(2)
[
Row(value='{"Item”:{“View1”:{“M”:{“1596385057313":{"B":"jSWdPakZ1cm91VWdNSWd5TmhiR1VZb2JUYi83b3U="},"1596385112207":{"B":"RNkJRMElXaWxDU2djSWdPakZ1cm9mNsTmhiR1VZaitIZS83b3U="}}},"blocking":{"BOOL":false},"date":{"S":"2020-09-13"},"id":{"S":"37816"}}}'),
Row(value='{"Item”:{“View1”:{“M”:{“1601064727986":{"B":"mRDbHdrb0hDSUM0byt2TExsb0RDTTRLWWdRSTZPb0JjZ2tLQjBadmNsTmhiR1Y2QlEwQUFJQS9HTExyazdmTUxnPT0="},"1601064741338":{"B":"nBCOUNRZ1VOTmRDbHdrb0hDSUM0byt2TExsb0RDTTRLWWdRSTZPQS9HTnJUbExmTUxnPT0=“}}},”View2”:{“M”:{“1601064618177":{"B":"gwSkNCUTAxMEtYQ1NnY0lnTGlqNjhzdVdnTUlWkdOdDh3dQ=="},"1601064728107":{"B":"SkNCUTAxMEtYQ1NnY0lnTGlqNjhzdVdnTUl6Z3BpQkFqbzZndQ=="},"1601064741411":{"B":"RFpta0gwSkNCUTAxMEtYQ1NnY0lnTGlqNjhzdVh3dQ=="}}},"blocking":{"BOOL":false},"date":{"S":"2020-09-25"},"id”:{“S”:”1123”}}}’)
]
As you can see data can have view1 and/or view2. Both are map which can have any number of values.Both View1 and View2 have timestamp and their byte value. now while reading these files as json, printschema looks like following
json_df = sqlContext.read.json("dynamodb/*")
json_df.printschema()
root
|-- Item: struct (nullable = true)
| |-- View: struct (nullable = true)
| | |-- M: struct (nullable = true)
| | | |-- 1601064618177: struct (nullable = true)
| | | | |-- B: string (nullable = true)
| | | |-- 1601064728107: struct (nullable = true)
| | | | |-- B: string (nullable = true)
| | | |-- 1601064741411: struct (nullable = true)
| | | | |-- B: string (nullable = true)
| |-- View1: struct (nullable = true)
| | |-- M: struct (nullable = true)
| | | |-- 1596385057313: struct (nullable = true)
| | | | |-- B: string (nullable = true)
| | | |-- 1596385112207: struct (nullable = true)
| | | | |-- B: string (nullable = true)
| | | |-- 1601064727986: struct (nullable = true)
| | | | |-- B: string (nullable = true)
| | | |-- 1601064741338: struct (nullable = true)
| | | | |-- B: string (nullable = true)
| |-- blocking: struct (nullable = true)
| | |-- BOOL: boolean (nullable = true)
| |-- date: struct (nullable = true)
| | |-- S: string (nullable = true)
| |-- id: struct (nullable = true)
| | |-- S: string (nullable = true)```
Now when I do df.show() like following, it will put `,` for timestamp which are not in view1 or view2 for that specific record. Imagine data is being 1 gb. I would see endless `,`
+--------------------+
| Item|
+--------------------+
|[, [[,,, [Q2dkSVp...|
|[[[[Q2doR1lYWnZjb...|
|[, [[[Q2dkSVpIQld...|
|[, [[,,,,,, [Q2dk...|
|[, [[,,,,, [Q2dkS...|
|[, [[,,, [Q2dkSVp...|
+--------------------+
How can I avoid that while reading json data.