Parse Json without quotes in Python

The sad answer is: the contents of your "Records" field are simply not JSON. No amount of ad-hoc patching (= to :, adding quotes) will change that. You have to find out the language/format specification for what the producing system emits and write/find a proper parser for that particular format.

As a clutch, and only in the case that the above example already captures all the variability you might see in production data, a much simpler approach based on regular expressions (see package re or edd's pragmatic answer) might be sufficient.

If the producer of the data is consistent, you can start with something like the following, that aims to bridge the JSON gap.

import re
import json


source = {
  "Records": "{Output=[{_fields=[{Entity=ABC  , No=12345, LineNo=       1, EffDate=20200630}, {Entity=ABC  , No=567, LineNo=       1, EffDate=20200630}]}"
}

s = source["Records"]

# We'll start by removing any extraneous white spaces
s2 = re.sub('\s', '', s)

# Surrounding any word with "
s3 = re.sub('(\w+)', '"\g<1>"', s2)

# Replacing = with :
s4 = re.sub('=', ':', s3)

# Lastly, fixing missing closing ], }
## Note that }} is an escaped } for f-string.
s5 = f"{s4}]}}"

>>> json.loads(s5)
{'Output': [{'_fields': [{'Entity': 'ABC', 'No': '12345', 'LineNo': '1', 'EffDate': '20200630'}, {'Entity': 'ABC', 'No': '567', 'LineNo': '1', 'EffDate': '20200630'}]}]}

Follow up with some robust testing and have a nice polished ETL with your favorite tooling.

As i understand you are trying to parse the value of the Records item in the dictionary as JSON, unfortunately you cannot.

The string in that value is not JSON, and you must write a parser that will first parse the string into a JSON string according to the format that the string is written in by yourself. ( We don't know what "middleware" you are talking of unfortunately ).

tldr: Parse it into a JSON string, then parse the JSON into a python dictionary. Read this to find out more about JSON ( Javascript Object Notation ) rules.

There you go. This code will make it valid json:

notjson = """{
  "Records": "{Output=[{_fields=[{Entity=ABC  , No=12345, LineNo=       1, EffDate=20200630}, {Entity=ABC  , No=567, LineNo=       1, EffDate=20200630}]}"
}"""
notjson = notjson.replace("=","':") #adds a singlequote and makes it more valid
notjson = notjson.replace("{","{'")
notjson = notjson.replace(", ",", '")
notjson = notjson.replace("}, '{","}, {")
json = "{" + notjson[2:]
print(json)
print(notjson)