Why does using str(df[col]) causing code to only relate to a single record?

user_13947916 0
Viewed 100

I have a dataframe containing leads (names). I am trying to search the web for relevant data regarding those leads.

I am using beautifulsoup and urllib to scrape the data. The url looks like this :

url = u'https://www.website.com/SearchResults?query=' + quote(str(df['name']))

The problem is that for each lead i get the exact same data, which is the data for the last lead in the dataframe of which data was retrieved.

whenever i use a string name instead of str(df['name']), i get for the specific lead the right data, and it looks like this :

url = u'https://www.website.com/SearchResults?query=' + quote('this+is+a+leads+name')

The reason i think the problem is specifically related to str(df['name']) is because whenever i remove it, i successfuly aquire data, otherwise, i get for 100,000 leads the same data. Only problem is, in order to use the leads from the dataframe i need to use str.

data-science pandas web-scraping beautifulsoup python
2 Answers

str(df) will return the text representation of dataframe. For example

>>> pd.DataFrame({"col1":[1,2,3,4], "name":["row1","row2","row3","row4"]})
    col1  name
0     1  row1
1     2  row2
2     3  row3
3     4  row4

the text representation for above dataframe will return

col1  name
1  row1
2  row2
3  row3
4  row4

So, to do row-wise operations it is best to use apply function from dataframe.

>>> def search_url(search_lead):
...     url = u'https://www.website.com/SearchResults?query=' + search_lead
...     print(url) #You can do your operation here

By using the above function, use apply

>>> df.name.apply(lambda x: search_url(x))
https://www.website.com/SearchResults?query=row1
https://www.website.com/SearchResults?query=row2
https://www.website.com/SearchResults?query=row3
https://www.website.com/SearchResults?query=row4
0    None
1    None
2    None
3    None
Name: name, dtype: object

You can do your web scraping inside the function. Hope this answer will help you.

user_41208060

Thank you everyone.

I tried what you guys suggested. What worked for me was indeed creating a function that will get the data from the web, and then using a for loop to call the function for each lead's name in a row.

In a nutshell this is what i did :

Function:

def getdata(name):

    url = u'https://www.website.com/search?q=' + quote(str(name))
    .
    .
    .
    return (data)

Loop :

for i, row in df.iterrows():

    leaddata = getdata(df.name[i])

    list1.append(leaddata)

I then proceeded to insert the list into the dataframe.

user_139479160
Related