Losing data when passing a pandas DataFrame through a function

Viewed 308

I am trying to call a function from a Jupyter Notebook (VSCode) that returns a pandas DataFrame df, however I have noticed that some rows of df are lost in the process. I have tried returning df itself and a deepcopy but both result in lost rows. When I run the code used in the function inside the notebook no data is lost, so I suspect the problem arises when passing df through the function. I should mention that the original size of df is 50558 and when returned the size goes down to 49997 (lost 561 elements).

A very crude sample of the code:

In the 'Reader.py' file:

from pathlib import Path
import pandas as pd
import os
import re

def getDF():
    #reading files
    (...)
    #processing data
    (...)

    rows_list = []
    for piece_of_data in data:
        #adding piece_of_data to dictionary dict  
        (...)  

        rows_list.append(dict)

    df = pd.DataFrame(rows_list)
    return df

In the Jupyter Notebook:

import pandas as pd
from Reader import getDF

df = getDF()

I know I could simply include the function code in the notebook and start from there but I would rather not have a big chunk of code with the creation of the dataframe if possible.

0 Answers
Related