Concat multiple Pandas dataframe in a for loop no matter the range

user19795989 0
Viewed 18

I am using PDF Plumber and I have a for loop which encompasses my extracted PDF data which I call PDF text. This for loop prints each page of the list as [1], [2], [3] etc.

Then each page is cleaned within this list before it is turned into a dataframe. To save me from creating multiple data frames I created 1 dataframe called temp dataframe and I'd like the for loop to create multiple data frames and combine them all together into one dataframe.

I have used the concat function before, but I'm unsure how to concat 1 dataframe to itself 3 times. If you look at the code I have below, the output of this only gives me the third dataframe which is self.pdf_text[3]. For context I'm unable to share the details of this list as I'm working with sensitive data, but not being able to see the list shouldn't matter as it's just a normal list of lists where each list represents a pdf page.

The for loop is working fine, I just need to be able to concat the dataframe 3 times as well. Please can anyone assist?

for i in range(len(self.pdf_text)):
                print(self.pdf_text[i])

                temp_pdf = pd.DataFrame(self.pdf_text[i])
                temp_pdf.drop([col for col in temp_pdf.columns if temp_pdf[col].apply(lambda x:'(' in str(x)).any()], axis=1,inplace=True)
                temp_pdf = temp_pdf.drop([col for col in temp_pdf.columns if temp_pdf[col].eq('sky').any()], axis=1)
                temp_pdf = temp_pdf.drop([col for col in temp_pdf.columns if temp_pdf[col].eq('high').any()], axis=1)
                temp_pdf = temp_pdf.drop([col for col in temp_pdf.columns if temp_pdf[col].eq('temp').any()], axis=1)
                temp_pdf = temp_pdf.drop([col for col in temp_pdf.columns if temp_pdf[col].eq('structure)').any()], axis=1)
                # temp_pdf = temp_pdf.drop(temp_pdf.iloc[:, 4:9], axis=1)
                temp_pdf.columns = range(temp_pdf.columns.size)

            combinedpdf = pd.concat([temp_pdf])
            print(combinedpdf)
range dataframe pandas loops python
1 Answers

I didn't bother to look into your logic about the pdfs, but I would create a function and use a list comprehension to store all the dataframes in a list to then concat them. I believe if you run the code below you will get what you are looking for.

def read_pdf_text(pdf_text):
    for i in range(len(pdf_text)):
        print(pdf_text[i])
        

        temp_pdf = pd.DataFrame(pdf_text[i])
        temp_pdf.drop([col for col in temp_pdf.columns if temp_pdf[col].apply(lambda x:'(' in str(x)).any()], axis=1,inplace=True)
        temp_pdf = temp_pdf.drop([col for col in temp_pdf.columns if temp_pdf[col].eq('sky').any()], axis=1)
        temp_pdf = temp_pdf.drop([col for col in temp_pdf.columns if temp_pdf[col].eq('high').any()], axis=1)
        temp_pdf = temp_pdf.drop([col for col in temp_pdf.columns if temp_pdf[col].eq('temp').any()], axis=1)
        temp_pdf = temp_pdf.drop([col for col in temp_pdf.columns if temp_pdf[col].eq('structure)').any()], axis=1)
        # temp_pdf = temp_pdf.drop(temp_pdf.iloc[:, 4:9], axis=1)
        temp_pdf.columns = range(temp_pdf.columns.size)
        
        return temp_pdf
    
pdf_list = [pdf1, pdf2, pdf3, pdf4, ....]

dfs = [read_pdf_text(x) for x in pdf_list]
dfs_concat = pd.concat(dfs)
ReinholdN0
Related