Python: Extract different tables and merge them into a single CSV file

user_10930017 0
Viewed 93

I want to find all the cold-storages in India and their capacity. The information is available in https://www.napanta.com/cold-storage/

Every state has its URL (example: https://www.napanta.com/cold-storage/andhra-pradesh/) ... and going to every state URL, we see that it has many districts, and each district has its own URL (https://www.napanta.com/cold-storage/andhra-pradesh/krishna) for the Krishna district in A.P.

My objective is to create One Single CSV files that have this information from all districts of every state. Additional columns in this One Big CSV file will be State and District name.

After we have this one state-district data frame, I want to append it to the larger dataframe for all states and districts. Finally export this large master dataframe into a single CSV file. The final result should be a single CSV file that contains all the data of every district of every state.

I tried the scraping part with Beautiful Soup and now, I'm stuck. Please, help me with the code.

pandas beautifulsoup csv python
2 Answers

You should get list of all links first. Can be done with selenium and beautiful-soup. Then you could try to loop over list of those links and read tables with pandas:

import pandas as pd
li = []
for link in list_of_links:
    dfi = pd.read_html(link)
    li.append(dfi)
df = pd.concat(li)
df.to_csv("Your_File.csv")
user_142089410

Once you have all the information, you can load it to a data frame and transform it using pandas, then export it to a csv using

DataFrame.to_csv('filename.csv')
user_97378970
Related