I have a dataframe I read from json, process, and save to csv. The df looks fine in terms of shape and df.head(), tail look fine. I write to a csv then read and get an error -
df = get_json(params)
df.to_csv(f'fname.csv')
testdf = pd.read_csv(f'fname.csv')
ParserError: Error tokenizing data. C error: Buffer overflow caught - possible malformed input file.
Is there some 'careful write' I should be doing?
Sometimes, there are some carriage returns '\r' in the data that pandas uses as a line terminator as if it was '\n'. So specifying lineterminator='\n' could resolve your issue:
testdf = pd.read_csv(f'fname.csv', lineterminator='\n')
You can also take a look at your data to see exactly where the error is occurring, please refer to this:
As per my experience with this issue, there was some problem with the default engine and doing this helped me:
df.read_csv('input.csv', engine = 'python')