Why is giving me this error :TypeError: cannot pickle '_io.TextIOWrapper' object?

user_9986262 0
Viewed 11067

I am trying to use multiprocessing and the idea is to get the links from the result of the Bing Search, but changing one of the configurations (cep configuration) using selenium. I have all the cep in a list (filecep) and I wanto to write all of the results on a csv file. This is my getUrlCleans function :

def getUrlCleans(search):


driver = webdriver.Firefox()

f = open('out/'+str(date.today())+'.csv','w')
f.write('url,cep')
f.write('\n')

url_cleans=[] 

pool=mp.Pool(mp.cpu_count())
pool.starmap(getUrlbyCEP,[(cep,driver,search,f) for cep in filecep])
pool.close()
f.close()

This is my getUrlbyCEP function :

def getUrlbyCEP(cep,driver,search,f):

driver.get('https://www.bing.com/account/general?ru=https%3a%2f%2fwww.bing.com%2f%3fFORM%3dZ9FD1&FORM=O2HV65#location')
                
    
cepInput = driver.find_element_by_id('geoname')
cepInput.clear()
cepInput.send_keys(cep)
time.sleep(0.5)
driver.execute_script("window.scrollTo(0,document.body.scrollHeight)")


saveButon=driver.find_element_by_id('sv_btn')
saveButon.click()




try:
    driver.find_element_by_id('geoname')        
    # continue
except:
    pass

searchInput=driver.find_element_by_id('sb_form_q')
searchInput.send_keys(search)

driver.find_element_by_id('sb_form_q').send_keys(Keys.ENTER)
time.sleep(0.5)

url_cleans=[]

for i in range(2):
    
    url_cleans=getLinks(driver,url_cleans)
    time.sleep(2)
    driver.find_element_by_xpath('//*[@title="Próxima página"]').click()
    url_cleans=getLinks(driver,url_cleans)
    for u in url_cleans:
        f.write(u+','+cep)
        f.write('\n')

Finally I call

getUrlCleans('sulamerica')

ang It gave me the error.... and I do not know why?

pool selenium multiprocessing python
1 Answers

So instead of using multiprocessing I used threads and it worked. This is what I change, instead of:

pool=mp.Pool(mp.cpu_count())
results = pool.starmap(getUrlbyCEP,[(cep,driver,search,f) for cep in filecep])

from multiprocecessing library (mp), I used this:

from multiprocessing.dummy import Pool as ThreadPool    
pool = ThreadPool(4)
results = pool.map(f_partial, filecep)
user_99862620
Related