Pandas parallel URL downloads with pd.read_html

user_14708237 0
Viewed 50

I know I can download a csv file from a web page by doing:

import pandas as pd
import numpy as np
from io import StringIO    

URL = "http://www.something.com"
data = pd.read_html(URL)[0].to_csv(index=False, header=True)
file = pd.read_csv(StringIO(data), sep=',')

Now I would like to do the above for more URLs at the same time, like when you open different tabs in your browser. In other words, a way to parallelize this when you have different URLs, instead of looping through or doing it one at a time. So, I thought of having a series of URLs inside a dataframe, and then create a new column which contains the strings 'data', one for each URL.

list_URL = ["http://www.something.com", "http://www.something2.com", 
            "http://www.something3.com"]
df = pd.DataFrame(list_URL, columns =['URL'])    
df['data'] = pd.read_html(df['URL'])[0].to_csv(index=False, header=True)

But it gives me error: cannot parse from 'Series'

Is there a better syntax, or does this mean I cannot do this in parallel for more than one URL?

pandas web-scraping html
1 Answers

You could try like this:

import pandas as pd

URLS = [
    "https://en.wikipedia.org/wiki/Periodic_table#Presentation_forms",
    "https://en.wikipedia.org/wiki/Planet#Planetary_attributes",
]

df = pd.DataFrame(URLS, columns=["URL"])
df["data"] = df["URL"].map(
    lambda x: pd.read_html(x)[0].to_csv(index=False, header=True)
)

print(df)
# Output
                                           URL                                         data
0  https://en.wikipedia.org/wiki/Periodic_t...  0\r\nPart of a series on the\r\nPeriodic...
1  https://en.wikipedia.org/wiki/Planet#Pla...  0\r\n"The eight known planets of the Sol...
user_112460560
Related