How to properly read dynamodb json data (dynamodb exported data from s3) in spark where json files may have different size of data

Viewed 197

I have DynamoDb json files. which have data like following

text = sqlContext.read.text('dynamodb/*')
text.take(2)
[
Row(value='{"Item”:{“View1”:{“M”:{“1596385057313":{"B":"jSWdPakZ1cm91VWdNSWd5TmhiR1VZb2JUYi83b3U="},"1596385112207":{"B":"RNkJRMElXaWxDU2djSWdPakZ1cm9mNsTmhiR1VZaitIZS83b3U="}}},"blocking":{"BOOL":false},"date":{"S":"2020-09-13"},"id":{"S":"37816"}}}'),

 Row(value='{"Item”:{“View1”:{“M”:{“1601064727986":{"B":"mRDbHdrb0hDSUM0byt2TExsb0RDTTRLWWdRSTZPb0JjZ2tLQjBadmNsTmhiR1Y2QlEwQUFJQS9HTExyazdmTUxnPT0="},"1601064741338":{"B":"nBCOUNRZ1VOTmRDbHdrb0hDSUM0byt2TExsb0RDTTRLWWdRSTZPQS9HTnJUbExmTUxnPT0=“}}},”View2”:{“M”:{“1601064618177":{"B":"gwSkNCUTAxMEtYQ1NnY0lnTGlqNjhzdVdnTUlWkdOdDh3dQ=="},"1601064728107":{"B":"SkNCUTAxMEtYQ1NnY0lnTGlqNjhzdVdnTUl6Z3BpQkFqbzZndQ=="},"1601064741411":{"B":"RFpta0gwSkNCUTAxMEtYQ1NnY0lnTGlqNjhzdVh3dQ=="}}},"blocking":{"BOOL":false},"date":{"S":"2020-09-25"},"id”:{“S”:”1123”}}}’)
]

As you can see data can have view1 and/or view2. Both are map which can have any number of values.Both View1 and View2 have timestamp and their byte value. now while reading these files as json, printschema looks like following

json_df  = sqlContext.read.json("dynamodb/*")
json_df.printschema()
root
 |-- Item: struct (nullable = true)
 |    |-- View: struct (nullable = true)
 |    |    |-- M: struct (nullable = true)
 |    |    |    |-- 1601064618177: struct (nullable = true)
 |    |    |    |    |-- B: string (nullable = true)
 |    |    |    |-- 1601064728107: struct (nullable = true)
 |    |    |    |    |-- B: string (nullable = true)
 |    |    |    |-- 1601064741411: struct (nullable = true)
 |    |    |    |    |-- B: string (nullable = true)
 |    |-- View1: struct (nullable = true)
 |    |    |-- M: struct (nullable = true)
 |    |    |    |-- 1596385057313: struct (nullable = true)
 |    |    |    |    |-- B: string (nullable = true)
 |    |    |    |-- 1596385112207: struct (nullable = true)
 |    |    |    |    |-- B: string (nullable = true)
 |    |    |    |-- 1601064727986: struct (nullable = true)
 |    |    |    |    |-- B: string (nullable = true)
 |    |    |    |-- 1601064741338: struct (nullable = true)
 |    |    |    |    |-- B: string (nullable = true)
 |    |-- blocking: struct (nullable = true)
 |    |    |-- BOOL: boolean (nullable = true)
 |    |-- date: struct (nullable = true)
 |    |    |-- S: string (nullable = true)
 |    |-- id: struct (nullable = true)
 |    |    |-- S: string (nullable = true)```

Now when I do df.show() like following, it will put `,` for timestamp which are not in view1 or view2 for that specific record. Imagine data is being 1 gb. I would see endless `,` 


+--------------------+
|                Item|
+--------------------+
|[, [[,,, [Q2dkSVp...|
|[[[[Q2doR1lYWnZjb...|
|[, [[[Q2dkSVpIQld...|
|[, [[,,,,,, [Q2dk...|
|[, [[,,,,, [Q2dkS...|
|[, [[,,, [Q2dkSVp...|
+--------------------+

How can I avoid that while reading json data.

0 Answers
Related