Unable to process list of sets having multiple element in list when list of set can be empty

Question

When I scrape websites for all the emails on each website and try to output it, I can get a given data frame which is a list of sets of multiple elements for each website :

URL_WITH_EMAILS_DF = pd.DataFrame(data=[{'main_url': 'http://keilstruplund.dk', 'emails': [{'ole.norlin@mail.dk', 'ole.gregersen@hk.dk'}, set(),{'ole.norlin@mail.dk', 'ole.gregersen@hk.dk'}, {'ole.norlin@mail.dk', 'ole.gregersen@hk.dk'}, {'ole.norlin@mail.dk', 'ole.gregersen@hk.dk'}, {'ole.norlin@mail.dk', 'ole.gregersen@hk.dk'}, {'ole.norlin@mail.dk', 'ole.gregersens@hk.dk', 'ole.gregersen@hk.dk'}, {'ole.norlin@mail.dk', 'ole.gregersen@hk.dk'}, {'ole.norlin@mail.dk', 'ole.gregersen@hk.dk'}, {'ole.norlin@mail.dk', 'ole.gregersen@hk.dk'}, {'ole.norlin@mail.dk', 'ole.gregersen@hk.dk'}, {'ole.norlin@mail.dk', 'ole.gregersen@hk.dk'}, {'ole.norlin@mail.dk', 'ole.gregersen@hk.dk'}, {'ole.norlin@mail.dk', 'ole.gregersen@hk.dk'}, {'ole.norlin@mail.dk', 'ole.gregersen@hk.dk'}, {'ole.norlin@mail.dk', 'ole.gregersen@hk.dk'}, {'ole.norlin@mail.dk', 'ole.gregersen@hk.dk'}, {'prima-rent@youseepost.dk', 'jb@rentind.dk', 'frisoren01@gmail.com', 'ole.norlin@mail.dk', 'ole.gregersen@hk.dk'}, {'ole.norlin@mail.dk', 'ole.gregersen@hk.dk'}, {'stigterndrup@gmail.com', 'psn@psn.dk', 'samuel@malerfirmaet-lykkebo.dk', 'jan@mundt-reklame.dk', 'ole.norlin@mail.dk', 'nordsjalland@phonixtag.dk', 'jp@rudersdalmaleren.dk', 'vvs@hestetangen.dk', 'steenkragelund@mail.tele.dk', 'ole.gregersen@hk.dk'}, {'ole.norlin@mail.dk', 'ole.gregersen@hk.dk'}, {'ole.norlin@mail.dk', 'ole.gregersen@hk.dk'}, {'ole.norlin@mail.dk', 'kasserer@keilstruplund.dk', 'ole.gregersen@hk.dk'}, {'ole.norlin@mail.dk', 'ole.gregersen@hk.dk'}, {'ole.norlin@mail.dk', 'ole.gregersen@hk.dk'}, {'ole.norlin@mail.dk', 'ole.gregersen@hk.dk'}, {'ole.norlin@mail.dk', 'ole.gregersen@hk.dk'}, {'ole.norlin@mail.dk', 'ole.gregersen@hk.dk'}, {'ole.norlin@mail.dk', 'ole.gregersen@hk.dk'}, {'ole.norlin@mail.dk', 'ole.gregersen@hk.dk'}, {'ole.norlin@mail.dk', 'ole.gregersen@hk.dk'}]}, 
                                    {'main_url': 'http://kirsebaergaarden.com', 'emails': [{'info@kirsebaergaarden.com'}, {'info@kirsebaergaarden.com'}, {'info@kirsebaergaarden.com'}, {'info@kirsebaergaarden.com'}, {'info@kirsebaergaarden.com'}, {'info@kirsebaergaarden.com'}, {'info@kirsebaergaarden.com'}, {'info@kirsebaergaarden.com'}, {'info@kirsebaergaarden.com'}, {'soesterb@gmail.com', 'info@kirsebaergaarden.com'}, {'info@kirsebaergaarden.com'}, {'info@kirsebaergaarden.com'}, {'info@kirsebaergaarden.com'}, {'info@kirsebaergaarden.com'}, {'info@kirsebaergaarden.com'}]},
                                     {'main_url': 'http://koglernes.dk', 'emails': [{'info@koglernes.dk'}, {'info@koglernes.dk'}, {'info@koglernes.dk'}, {'info@koglernes.dk'},set(), set(), {'info@koglernes.dk'}, {'info@koglernes.dk'}]},
                                      {'main_url': 'http://kongehojensbornehave.dk', 'emails': [set()]}
                                   ])

However, I want to process the data frame to look like the following:

URL_WITH_EMAILS_DF = pd.DataFrame(data=[{'main_url': 'http://keilstruplund.dk', 'emails': ['ole.norlin@mail.dk', 'ole.gregersen@hk.dk', 'prima-rent@youseepost.dk', 'jb@rentind.dk', 'frisoren01@gmail.com','stigterndrup@gmail.com', 'psn@psn.dk', 'samuel@malerfirmaet-lykkebo.dk', 'jan@mundt-reklame.dk',  'nordsjalland@phonixtag.dk', 'jp@rudersdalmaleren.dk', 'vvs@hestetangen.dk', 'steenkragelund@mail.tele.dk', 'kasserer@keilstruplund.dk']},                                        
                                     {'main_url': 'http://kirsebaergaarden.com', 'emails': ['info@kirsebaergaarden.com']},
                                     {'main_url': 'http://koglernes.dk', 'emails': ['info@koglernes.dk']},
                                      {'main_url': 'http://kongehojensbornehave.dk', 'emails': []}
                                   ])

How can it be achieve?

I have tried the following code but it only manage to return first element of first set only while running to error when there is no element in the email list for a given website :

URL_WITH_EMAILS_DF['emails'] = [', '.join(x.pop()) if not None else "" for x in URL_WITH_EMAILS_DF['emails'].values]

P.S:

As per the first data frame, I needed to get a set of multiple emails to be inserted because there can be multiple webpages for a single website and I do not want to take duplicate emails from each web page
If a list has [set(), set()] or [], it should be consider as empty. Also if set() is available as a value on 'emails' it just throw "TypeError: 'NoneType' object is not iterable".
Thanks to Chris, he provided a solution here. However, it shows error mentioned in point#2. The solution is as follows:

from itertools import chain
        URL_WITH_EMAILS_DF['emails'] = URL_WITH_EMAILS_DF.emails.apply(lambda x: list(set(chain.from_iterable(x))))

user_16343464 · Answer

Use a simple list comprehension with a set union, this should be the fastest:

URL_WITH_EMAILS_DF['emails'] = [list(set.union(*s))
                                for s in URL_WITH_EMAILS_DF['emails']]

output:

                         main_url                                                                                                                                                                                                                                                                                                                                                   emails
0         http://keilstruplund.dk  [ole.gregersen@hk.dk, kasserer@keilstruplund.dk, jan@mundt-reklame.dk, nordsjalland@phonixtag.dk, vvs@hestetangen.dk, ole.norlin@mail.dk, frisoren01@gmail.com, stigterndrup@gmail.com, jb@rentind.dk, psn@psn.dk, prima-rent@youseepost.dk, samuel@malerfirmaet-lykkebo.dk, steenkragelund@mail.tele.dk, jp@rudersdalmaleren.dk, ole.gregersens@hk.dk]
1     http://kirsebaergaarden.com                                                                                                                                                                                                                                                                                                          [soesterb@gmail.com, info@kirsebaergaarden.com]
2             http://koglernes.dk                                                                                                                                                                                                                                                                                                                                      [info@koglernes.dk]
3  http://kongehojensbornehave.dk                                                                                                                                                                                                                                                                                                                                                       []

From a string that represents a list of sets:

from ast import literal_eval

URL_WITH_EMAILS_DF['emails'] = [list(set.union(*literal_eval(s)))
                                for s in URL_WITH_EMAILS_DF['emails']]

or, if you have improperly formed strings, and assuming you don't have single quotes in your email addresses, you can use a regex:

import re
URL_WITH_EMAILS_DF['emails'] = [list(set(re.findall("'([^']+@[^']+)'", s)))
                                for s in URL_WITH_EMAILS_DF['emails']]

user_10179777 · Answer

You can write a function to combine the sets through unions and then perform your filtering when casting it into a list:

def combine_sets(list_of_sets):
    output = set()    
    for one_set in list_of_sets:
        output = output.union(one_set)

    # Do whatever filtering you need here
    return list(email if email is not None else '' for email in output)

Then your code will just be:

URL_WITH_EMAILS_DF['emails'] = URL_WITH_EMAILS_DF['emails'].map(combine_sets)

Also, just a side note regarding your:

URL_WITH_EMAILS_DF['emails'] = [', '.join(x.pop()) if not None else "" for x in URL_WITH_EMAILS_DF['emails'].values]

you are checking if not None which will always evaluate to True, you should be typing if x is not None instead.

Unable to process list of sets having multiple element in list when list of set can be empty

2 Answers