How to read csv file via handling cases like newline character in a row?

Question

for row in codecs.getreader(self.encoding)(self.response[u'Body']).readlines():
            row_string = StringIO(row)
            print ("Row read from the data is: ")
            print (row_string.getvalue())
            df = pd.read_csv(row_string, sep=",")

I have written the above code to stream a csv file from S3 row by row. However, there is a row in a csv file with an enter in its one of the rows. Pandas is able to read it when file is downloaded in local but in the above code it is generating an error:

[2018-11-12 14:11:45,586] {models.py:1595} ERROR - Error tokenizing data. C error: EOF inside string starting at line 0

Ignore the line 0 remark above as you can see in my code, I read one row and form its data frame.

The complete error traceback is:

[2018-11-12 14:11:45,586]



{models.py:1595} ERROR - Error tokenizing data. C error: EOF inside
 string starting at line 0 Traceback (most recent call last):   File
 "/usr/local/lib/python3.5/dist-packages/airflow/models.py", line 1493,
 in _run_raw_task
     result = task_copy.execute(context=context)   File "/usr/local/lib/python3.5/dist-packages/airflow/operators/python_operator.py",
 line 89, in execute
     return_value = self.execute_callable()   File "/usr/local/lib/python3.5/dist-packages/airflow/operators/python_operator.py",
 line 94, in execute_callable
     return self.python_callable(*self.op_args, **self.op_kwargs)   File
 "/usr/local/lib/python3.5/dist-packages/pallet-0.0.0-py3.5.egg/pallet/tasks/versionator.py", line 228, in driver_de_versionator
     a.index_patch()   File "/usr/local/lib/python3.5/dist-packages/pallet-0.0.0-py3.5.egg/pallet/tasks/versionator.py", line 202, in index_patch
     DB.process(self.form_candidate_version, self.destination_of_kch_file_to_be_downloaded)   File
 "/usr/local/lib/python3.5/dist-packages/pallet-0.0.0-py3.5.egg/pallet/tasks/versionator.py", line 144, in form_candidate_version
     df = pd.read_csv(row_string, sep=",")   File "/usr/local/lib/python3.5/dist-packages/pandas/io/parsers.py", line
 678, in parser_f
     return _read(filepath_or_buffer, kwds)   File "/usr/local/lib/python3.5/dist-packages/pandas/io/parsers.py", line
 440, in _read
     parser = TextFileReader(filepath_or_buffer, **kwds)   File "/usr/local/lib/python3.5/dist-packages/pandas/io/parsers.py", line
 787, in __init__
     self._make_engine(self.engine)   File "/usr/local/lib/python3.5/dist-packages/pandas/io/parsers.py", line
 1014, in _make_engine
     self._engine = CParserWrapper(self.f, **self.options)   File "/usr/local/lib/python3.5/dist-packages/pandas/io/parsers.py", line
 1708, in __init__
     self._reader = parsers.TextReader(src, **kwds)   File "pandas/_libs/parsers.pyx", line 539, in
 pandas._libs.parsers.TextReader.__cinit__   File
 "pandas/_libs/parsers.pyx", line 737, in
 pandas._libs.parsers.TextReader._get_header   File
 "pandas/_libs/parsers.pyx", line 932, in
 pandas._libs.parsers.TextReader._tokenize_rows   File
 "pandas/_libs/parsers.pyx", line 2112, in
 pandas._libs.parsers.raise_parser_error pandas.errors.ParserError:
 Error tokenizing data. C error: EOF inside string starting at line 0

user_9183344 · Answer

A quick google of your woe turns up this link, which suggests:

The solution was to use the parameter engine='python' in the read_csv function call. The Pandas CSV parser can use two different “engines” to parse CSV file – Python or C (default).

pandas.read_csv(filepath_or_buffer, sep=', ', delimiter=None, 
                header='infer', names=None, 
                index_col=None, usecols=None, squeeze=False, 
                ..., engine=None, ...)

engine : {'c', 'python'}, optional

Parser engine to use. The C engine is faster while the python engine is currently more feature-complete.

How to read csv file via handling cases like newline character in a row?

1 Answers