JSONDecodeError : Extra data: line 2 column 1

Question

I'm trying to read json zipped file from S3 buckets and writing to a dynamo db table using aws lambda service and I chose python boto3 language. After I read the s3 data, while trying to run json.loads I'm getting this error.

My code looks something like -

import json
import gzip
import boto3
from io import BytesIO

s3 = boto3.resource('s3')
dynamodb = boto3.resource('dynamodb')

def lambda_handler(event, context):
    bucket = event['Records'][0]['s3']['bucket']['name']
    json_file_name = event['Records'][0]['s3']['object']['key']
    
    json_object = s3.Object(bucket, json_file_name)
    
    n = json_object.get()['Body'].read()
    gzipfile = BytesIO(n)
    gzipfile = gzip.GzipFile(fileobj=gzipfile)
    content = gzipfile.read().decode('utf-8')

    jsonDict = json.loads(content)
    
    # Write items to dynamo db table
    table = dynamodb.Table('mahbis01-AccountService-LedgerSummary-Duplicate')
    table.put_item(Item=jsonDict)
    
    return {
        'statusCode': 200,
        'body': json.dumps('Hello from Lambda!')
    }

When I printed content, I see the values like -

{
   "Item":{
      "SubsId":{
         "S":"255_0_908764"
      }
   }
}{
   "Item":{
      "SubsId":{
         "S":"255_0_908765"
      }
   }
}{
   "Item":{
      "SubsId":{
         "S":"255_0_908766"
      }
   }
}{
   "Item":{
      "SubsId":{
         "S":"255_0_908767"
      }
   }
}

How can I get rid of this and write the data to dynamo db?

user_248823 · Answer

Your content is obviously incorrect json. Assuming the content string has constant format, you can convert it to json using:

jsonDict = json.loads('['+content.replace('}{','},{')+']')

which will give you valid list of dicts:

[{'Item': {'SubsId': {'S': '255_0_908764'}}}, {'Item': {'SubsId': {'S': '255_0_908765'}}}, {'Item': {'SubsId': {'S': '255_0_908766'}}}, {'Item': {'SubsId': {'S': '255_0_908767'}}}]

Then you can iterate over it and process how you want, e.g.:

for item in jsonDict:
  print(item)
  # or upload to dynamodb

JSONDecodeError : Extra data: line 2 column 1

1 Answers