I am trying to call a function from a Jupyter Notebook (VSCode) that returns a pandas DataFrame df, however I have noticed that some rows of df are lost in the process. I have tried returning df itself and a deepcopy but both result in lost rows. When I run the code used in the function inside the notebook no data is lost, so I suspect the problem arises when passing df through the function. I should mention that the original size of df is 50558 and when returned the size goes down to 49997 (lost 561 elements).
A very crude sample of the code:
In the 'Reader.py' file:
from pathlib import Path
import pandas as pd
import os
import re
def getDF():
#reading files
(...)
#processing data
(...)
rows_list = []
for piece_of_data in data:
#adding piece_of_data to dictionary dict
(...)
rows_list.append(dict)
df = pd.DataFrame(rows_list)
return df
In the Jupyter Notebook:
import pandas as pd
from Reader import getDF
df = getDF()
I know I could simply include the function code in the notebook and start from there but I would rather not have a big chunk of code with the creation of the dataframe if possible.