"AttributeError: 'str' object has no attribute 'descendants' error with automation scraping with bs4 and selenium

user_16281418 0
Viewed 88

My objective with this code is to scrap the allocation of brazilian funds.

PATH = (r"C:\Users\gustavo.vieira\Desktop\python\drivers\msedgedriver.exe")
funds_list = ['VINLAND MACRO MASTER FUNDO DE INVESTIMENTO MULTIMERCADO']
url = 'https://cvmweb.cvm.gov.br/SWB/Sistemas/SCW/CPublica/CConsolFdo/FormBuscaParticFdo.aspx'


for i in funds_lista:
    driver = webdriver.Edge(PATH)
    driver.get(url)
    search = driver.find_element_by_name("txtCNPJNome")
    search.send_keys(i)
    search.send_keys(Keys.RETURN)
    driver.find_element_by_link_text(i).click()
    font = driver.find_element_by_link_text('Composição da Carteira').click()
    soup = BeautifulSoup(font)
    rows = soup.find_all("tr")
    print(rows)

My problem starts with bs4, I already tried to scrap the table using soup.find_all('table', id ='dlAplics'), or using (class_, all of them return an error. I already saw a lot of videos but I'm feeling unable to solve that by myself right now.

web-scraping selenium beautifulsoup python-3.x python
2 Answers

What happens?

You assign the .click() to variable font and try to process it with BeautifulSoup what won't work.

How to fix?

Instead provide driver.page_source to BeautifulSoup to operate on the html.

Change:

font = driver.find_element_by_link_text('Composição da Carteira').click()
soup = BeautifulSoup(font)

to:

driver.find_element_by_link_text('Composição da Carteira').click()
soup = BeautifulSoup(driver.page_source)

Note * Id's in general are unique so you could use soup.find('table', id ='dlAplics') instead of find_all() to directly operate on this table.*

Output

<table border="1" cellpadding="4" cellspacing="0" class="BodyPP" id="dlAplics" rules="all" style="border-width:1px;border-style:solid;width:100%;border-collapse:collapse;">
<tbody><tr style="background-color:#ACC2D8;">
<td align="center" colspan="11"><b>Lista de Aplicações<br/>Clique sobre o ativo para mais informações.</b></td>
</tr><tr style="background-color:#ACC2D8;">
<td align="center" rowspan="3" style="width:30%;"><b>Ativo</b></td><td align="center" rowspan="3"><b>Classificação</b></td><td align="center" rowspan="3"><b>Empresa Ligada</b></td><td align="center" colspan="4"><b>Negócios Realizados no Mês</b></td><td align="center" colspan="4"><b>Posição Final</b></td>
</tr><tr style="background-color:#ACC2D8;">
<td align="center" colspan="2"><b>Vendas</b></td><td align="center" colspan="2"><b>Aquisições</b></td><td align="center" rowspan="2" style="width:7%;"><b>Quant.</b></td><td align="center" colspan="2"><b>Valores</b></td><td align="center" rowspan="2" style="width:7%;"><b>% Patr. Líq.</b></td>
</tr><tr style="background-color:#ACC2D8;">
<td align="center" style="width:7%;"><b>Quant.</b></td><td align="center" style="width:7%;"><b>Valor<br/></b></td><td align="center" style="width:7%;"><b>Quant.</b></td><td align="center" style="width:7%;"><b>Valor<br/></b></td><td align="center" style="width:7%;"><b>Custo<br/></b></td><td align="center" style="width:7%;"><b>Mercado<br/></b></td>
</tr><tr style="background-color:Cornsilk;">
<td align="left"><span onclick="AbrirDet('CDADetAplic.aspx?PkCDAAplic=72375627')" style="cursor: hand; COLOR: olivedrab"><b></b> Ações<br/><b>Cod. Ativo:</b> RDOR3<br/><b>Dt. Ini. Vigen.:</b> 10/12/2020<br/></span></td><td align="center"><span id="dlAplics__ctl1_lbNConfTpNegoc">Para<br/>negociação</span></td><td align="center"><span id="dlAplics__ctl1_lbNConfEmpLigada">Não</span></td><td align="center">38.219</td><td align="center"><span id="dlAplics__ctl1_lbNConfVlVendasNegoc">2.139.131,81</span></td><td align="center">37.866</td><td align="center"><span id="dlAplics__ctl1_lbNConfVlAquisNegoc">2.239.218,96</span></td><td align="center">1.502</td><td align="center"><span id="dlAplics__ctl1_lbNConfVlCustoCorrecPosFim"></span></td><td align="center"><span id="dlAplics__ctl1_lbNConfVlPosFim">75.355,34</span></td><td align="center">0,005</td>
</tr><tr style="background-color:#FAEFCA;">
<td align="left"><span onclick="AbrirDet('CDADetAplic.aspx?PkCDAAplic=72375615')" style="cursor: hand; COLOR: olivedrab"><b></b> Ações<br/><b>Cod. Ativo:</b> RDOR3<br/><b>Dt. Ini. Vigen.:</b> 25/06/2021<br/></span></td><td align="center"><span id="dlAplics__ctl2_lbNConfAltTpNegoc">Para<br/>negociação</span></td><td align="center"><span id="dlAplics__ctl2_lbNConfAltEmpLigada">Não</span></td><td align="center">38.219</td><td align="center"><span id="dlAplics__ctl2_lbNConfAltVlVendasNegoc">2.139.131,81</span></td><td align="center">37.866</td><td align="center"><span id="dlAplics__ctl2_lbNConfAltVlAquisNegoc">2.239.218,96</span></td><td align="center">1.502</td><td align="center"><span id="dlAplics__ctl2_lbNConfAltVlCustoCorrecPosFim"></span></td><td align="center"><span id="dlAplics__ctl2_lbNConfAltVlPosFim">75.355,34</span></td><td align="center">0,005</td>
</tr><tr style="background-color:Cornsilk;">
<td align="left"><span onclick="AbrirDet('CDADetAplic.aspx?PkCDAAplic=72375728')" style="cursor: hand; COLOR: olivedrab"><b></b> Certificado ou recibo de depósito de valores mobiliários<br/><b>Descrição:</b> TAEE11 - TRANS. ALIANCA DE ENERGIA ELETRICA S/A - TAEE11<br/><b>CNPJ do emissor:</b> 07.859.971/0001-30<br/><b>Denominação Social do emissor: </b> TRANS. ALIANCA DE ENERGIA ELETRICA S/A<br/></span></td><td align="center"><span id="dlAplics__ctl3_lbNConfTpNegoc">Para<br/>negociação</span></td><td align="center"><span id="dlAplics__ctl3_lbNConfEmpLigada">Não</span></td><td align="center">0</td><td align="center"><span id="dlAplics__ctl3_lbNConfVlVendasNegoc">0,00</span></td><td align="center">0</td><td align="center"><span id="dlAplics__ctl3_lbNConfVlAquisNegoc">0,00</span></td><td align="center">979</td><td align="center"><span id="dlAplics__ctl3_lbNConfVlCustoCorrecPosFim"></span></td><td align="center"><span id="dlAplics__ctl3_lbNConfVlPosFim">35.273,37</span></td><td align="center">0,002</td>
</tr><tr style="background-color:#FAEFCA;">
<td align="left"><span onclick="AbrirDet('CDADetAplic.aspx?PkCDAAplic=72375488')" style="cursor: hand; COLOR: olivedrab"><b></b> Cotas de Fundos<br/><b></b> HASHDEX NASDAQ ETHEREUM REFERENCE PRICE FUNDO DE ÍNDICE<br/></span></td><td align="center"><span id="dlAplics__ctl4_lbNConfAltTpNegoc"></span></td><td align="center"><span id="dlAplics__ctl4_lbNConfAltEmpLigada">Não</span></td><td align="center">23.967</td><td align="center"><span id="dlAplics__ctl4_lbNConfAltVlVendasNegoc">1.667.612,78</span></td><td align="center">209.290</td><td align="center"><span id="dlAplics__ctl4_lbNConfAltVlAquisNegoc">15.823.222,00</span></td><td align="center">142.956</td><td align="center"><span id="dlAplics__ctl4_lbNConfAltVlCustoCorrecPosFim"></span></td><td align="center"><span id="dlAplics__ctl4_lbNConfAltVlPosFim">11.436.480,00</span></td><td align="center">0,787</td>
</tr><tr style="background-color:Cornsilk;">
<td align="left"><span onclick="AbrirDet('CDADetAplic.aspx?PkCDAAplic=72375486')" style="cursor: hand; COLOR: olivedrab"><b></b> Cotas de Fundos<br/><b></b> HASHDEX NASDAQ CRYPTO INDEX FUNDO DE INDICE<br/></span></td><td align="center"><span id="dlAplics__ctl5_lbNConfTpNegoc"></span></td><td align="center"><span id="dlAplics__ctl5_lbNConfEmpLigada">Não</span></td><td align="center">628.668</td><td align="center"><span id="dlAplics__ctl5_lbNConfVlVendasNegoc">38.177.765,73</span></td><td align="center">634.819</td><td align="center"><span id="dlAplics__ctl5_lbNConfVlAquisNegoc">39.878.659,26</span></td><td align="center">87.530</td><td align="center"><span id="dlAplics__ctl5_lbNConfVlCustoCorrecPosFim"></span></td><td align="center"><span id="dlAplics__ctl5_lbNConfVlPosFim">5.487.255,70</span></td><td align="center">0,378</td>
</tr><tr style="background-color:#FAEFCA;">
<td align="left"><span onclick="AbrirDet('CDADetAplic.aspx?PkCDAAplic=72375484')" style="cursor: hand; COLOR: olivedrab"><b></b> Cotas de Fundos<br/><b></b> ISHARES S&amp;P 500 FUNDO DE INVESTIMENTO EM COTAS DE FUNDO DE ÍNDICE - INVESTIMENTO NO EXTERIOR<br/></span></td><td align="center"><span id="dlAplics__ctl6_lbNConfAltTpNegoc"></span></td><td align="center"><span id="dlAplics__ctl6_lbNConfAltEmpLigada">Não</span></td><td align="center">26.057</td><td align="center"><span id="dlAplics__ctl6_lbNConfAltVlVendasNegoc">7.285.144,24</span></td><td align="center">31.466</td><td align="center"><span id="dlAplics__ctl6_lbNConfAltVlAquisNegoc">8.871.406,46</span></td><td align="center">5.983</td><td align="center"><span id="dlAplics__ctl6_lbNConfAltVlCustoCorrecPosFim"></span></td><td align="center"><span id="dlAplics__ctl6_lbNConfAltVlPosFim">1.678.530,65</span></td><td align="center">0,116</td>
</tr><tr style="background-color:Cornsilk;">
<td align="left"><span onclick="AbrirDet('CDADetAplic.aspx?PkCDAAplic=72375640')" style="cursor: hand; COLOR: olivedrab"><b></b> Investimento no Exterior<br/><b>Nome do emissor: </b> CITCO - 376420,897566<br/><b>Venc.:</b> 31/12/2050<br/></span></td><td align="center"><span id="dlAplics__ctl7_lbNConfTpNegoc">Para<br/>negociação</span></td><td align="center"><span id="dlAplics__ctl7_lbNConfEmpLigada">Não</span></td><td align="center">0</td><td align="center"><span id="dlAplics__ctl7_lbNConfVlVendasNegoc">0,00</span></td><td align="center">0</td><td align="center"><span id="dlAplics__ctl7_lbNConfVlAquisNegoc">0,00</span></td><td align="center">376.421</td><td align="center"><span id="dlAplics__ctl7_lbNConfVlCustoCorrecPosFim"></span></td><td align="center"><span id="dlAplics__ctl7_lbNConfVlPosFim">257.652.646,82</span></td><td align="center">17,731</td>
</tr><tr style="background-color:#FAEFCA;">
<td align="left"><span onclick="AbrirDet('CDADetAplic.aspx?PkCDAAplic=72375755')" style="cursor: hand; COLOR: olivedrab"><b></b> Valores a pagar<br/><b>Descrição:</b> AURA33<br/><b>CNPJ do emissor:</b> 07.857.093/0001-14<br/><b>Denominação Social do emissor: </b> AURA MINERALS INC.<br/></span></td><td align="center"><span id="dlAplics__ctl8_lbNConfAltTpNegoc">Para<br/>negociação</span></td><td align="center"><span id="dlAplics__ctl8_lbNConfAltEmpLigada"></span></td><td align="center"></td><td align="center"><span id="dlAplics__ctl8_lbNConfAltVlVendasNegoc"></span></td><td align="center"></td><td align="center"><span id="dlAplics__ctl8_lbNConfAltVlAquisNegoc"></span></td><td align="center"></td><td align="center"><span id="dlAplics__ctl8_lbNConfAltVlCustoCorrecPosFim"></span></td><td align="center"><span id="dlAplics__ctl8_lbNConfAltVlPosFim">458.916,75</span></td><td align="center">-0,032</td>
</tr>
</tbody></table>

user_144608240

To scrape all the text from the first table on Consulta a Carteiras de Fundos page you need to induce WebDriverWait for the visibility_of_element_located() and you can use either of the following Locator Strategies:

  • Code block:

    driver.get("https://cvmweb.cvm.gov.br/SWB/Sistemas/SCW/CPublica/CConsolFdo/FormBuscaParticFdo.aspx")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input#txtCNPJNome"))).send_keys("VINLAND MACRO MASTER FUNDO DE INVESTIMENTO MULTIMERCADO" + Keys.RETURN)
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a#ddlFundos__ctl0_lnkbtn1"))).click()
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.PARTIAL_LINK_TEXT, "Composição da Carteira"))).click()
print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//table[@id='tabAtivos']//table[@id='dlAplics']"))).text)

  • Console output:

    Lista de Aplicações
Clique sobre o ativo para mais informações.
Ativo Classificação Empresa Ligada Negócios Realizados no Mês Posição Final
Vendas Aquisições Quant. Valores % Patr. Líq.
Quant. Valor
Quant. Valor
Custo
Mercado
Ações
Cod. Ativo: RDOR3
Dt. Ini. Vigen.: 10/12/2020
Para
negociação Não 38.219 2.139.131,81 37.866 2.239.218,96 1.502 75.355,34 0,005
Ações
Cod. Ativo: RDOR3
Dt. Ini. Vigen.: 25/06/2021
Para
negociação Não 38.219 2.139.131,81 37.866 2.239.218,96 1.502 75.355,34 0,005
Certificado ou recibo de depósito de valores mobiliários
Descrição: TAEE11 - TRANS. ALIANCA DE ENERGIA ELETRICA S/A - TAEE11
CNPJ do emissor: 07.859.971/0001-30
Denominação Social do emissor: TRANS. ALIANCA DE ENERGIA ELETRICA S/A
Para
negociação Não 0 0,00 0 0,00 979 35.273,37 0,002
Cotas de Fundos
HASHDEX NASDAQ ETHEREUM REFERENCE PRICE FUNDO DE ÍNDICE
Não 23.967 1.667.612,78 209.290 15.823.222,00 142.956 11.436.480,00 0,787
Cotas de Fundos
HASHDEX NASDAQ CRYPTO INDEX FUNDO DE INDICE
Não 628.668 38.177.765,73 634.819 39.878.659,26 87.530 5.487.255,70 0,378
Cotas de Fundos
ISHARES S&P 500 FUNDO DE INVESTIMENTO EM COTAS DE FUNDO DE ÍNDICE - INVESTIMENTO NO EXTERIOR
Não 26.057 7.285.144,24 31.466 8.871.406,46 5.983 1.678.530,65 0,116
Investimento no Exterior
Nome do emissor: CITCO - 376420,897566
Venc.: 31/12/2050
Para
negociação Não 0 0,00 0 0,00 376.421 257.652.646,82 17,731
Valores a pagar
Descrição: AURA33
CNPJ do emissor: 07.857.093/0001-14
Denominação Social do emissor: AURA MINERALS INC.
Para
negociação 458.916,75 -0,032
user_74294470
Related