I'm trying to use the relatively new option to pass a callable to the on_bad_lines argument while reading an unknown csv file, but it doesn't work.
Here is my code:
def foo(csv_file_data):
badlines_list = []
def badlines_collect(bad_line):
badlines_list.append(bad_line)
return None
df = pd.read_csv(StringIO(csv_file_data), engine="python", on_bad_lines=badlines_collect)
# do something
return df.to_csv(index=False), badlines_list
I'm testing the code with a file I know for certain has offending lines, but the badlines_list list is returned empty on each run.
I saw the PR about it here, and it seems like I'm using it correctly.
Is there something I'm doing wrong? Can someone please help me?
Note that I'm using the latest version of pandas (1.4.0).
Thanks!
Edit: added a screenshot of a local test that doesn't work for me