Beautifulsoup scraping .text and splitting them automatically

user_18728772 0
Viewed 49

A total BeautifulSoup newbie here. I need to scrape the publication tab's content from a certain URL (listed in the code sample below). I need to scrape the publications and split them into 'authors', 'title', and 'journal', which I can then convert to pandas DataFrame. I tried scraping the content using the code below:

page_url = 'https://www.vet.upenn.edu/research/centers-laboratories/research-laboratory/research-laboratory/anguera-laboratory'

def get_soup(url):
    r = requests.get('http://localhost:8050/render.html', params={'url': url, 'wait': 3})
    soup = BeautifulSoup(r.text, 'html.parser')

url_soup = get_soup(page_url)

publis = url_soup.find_all('div',{'class':'tab-content pad'})

for item in publis:
    publi = {
        'items': item.find('div',{'id':'tabs-publications'}).text.split('\n\n')
    }

publication_df = pd.DataFrame(publi)
new_publication_df = publication_df['items'].str.split(':', 1, expand=True)
new_publication_df[[1,2]] = new_publication_df[1].str.split(']',expand=True)

This returns what I need, but it is really prone to typos (e.g. there is one publication that uses 'J' rather than ']'). Is there any way that BeautifulSoup can automatically separate the text into three columns?

pandas web-scraping dataframe beautifulsoup python
1 Answers

Think you should use available structure instead of splitting the strings.

Select all <em> that holds the title, its previous_sibling, that holds the authors and also its next_sibling, that holds the journal.

Example
import requests
from bs4 import BeautifulSoup


r = requests.get('https://www.vet.upenn.edu/research/centers-laboratories/research-laboratory/research-laboratory/anguera-laboratory')
soup = BeautifulSoup(r.text)
data = []
for e in soup.select('#tabs-publications em'):
    data.append({
        'author':e.previous.get_text(strip=True)[:-1],
        'title':e.get_text(strip=True),
        'journal':e.next_sibling.get_text(strip=True)
    })

data
Output data
[{'author': 'Hantsoo Liisa, Kornfield Sara, Anguera Montserrat C, Epperson C Neill',
  'title': 'Inflammation: A Proposed Intermediary Between\xa0Maternal Stress and Offspring Neuropsychiatric Risk. [PMID30314641]',
  'journal': 'Biological psychiatry 85(2): 97-106, Jan 2019.'},
 {'author': 'Sierra Isabel, Anguera Montserrat C',
  'title': 'Enjoy the silence: X-chromosome inactivation diversity in somatic cells.[PMID31108425]',
  'journal': 'Current opinion in genetics & development 55: 26-31, May 2019.'},
 {'author': 'Syrett Camille M, Anguera Montserrat C',
  'title': 'When the balance is broken: X-linked gene dosage from two X chromosomes and female-biased autoimmunity. [PMID31125996]',
  'journal': 'Journal of leukocyte biology May 2019.'},
 {'author': 'Kotzin Jonathan J, Iseka Fany, Wright Jasmine, Basavappa Megha G, Clark Megan L, Ali Mohammed-Alkhatim, Abdel-Hakeem Mohamed S, Robertson Tanner F, Mowel Walter K, Joannas Leonel, Neal Vanessa D, Spencer Sean P, Syrett Camille M, Anguera Montserrat C, Williams Adam, Wherry E John, Henao-Mejia Jorge',
  'title': 'The long noncoding RNA regulates CD8 T cells in response to viral infection.[PMID31138702]',
  'journal': 'Proceedings of the National Academy of Sciences of the United States of America 116(24): 11916-11925, Jun 2019.'},
 {'author': 'Syrett Camille M, Paneru Bam, Sandoval-Heglund Donavon, Wang Jianle, Banerjee Sarmistha, Sindhava Vishal, Behrens Edward M, Atchison Michael, Anguera Montserrat C',
  'title': 'Altered X-chromosome inactivation in T cells may promote sex-biased autoimmune diseases. [PMID30944248',
  'journal': 'JCI insight 4(7), Apr 2019.'},...]
Output DataFrame
pd.DataFrame(data)
author title journal
Hantsoo Liisa, Kornfield Sara, Anguera Montserrat C, Epperson C Neill Inflammation: A Proposed Intermediary Between Maternal Stress and Offspring Neuropsychiatric Risk. [PMID30314641] Biological psychiatry 85(2): 97-106, Jan 2019.
Sierra Isabel, Anguera Montserrat C Enjoy the silence: X-chromosome inactivation diversity in somatic cells.[PMID31108425] Current opinion in genetics & development 55: 26-31, May 2019.
Syrett Camille M, Anguera Montserrat C When the balance is broken: X-linked gene dosage from two X chromosomes and female-biased autoimmunity. [PMID31125996] Journal of leukocyte biology May 2019.
Kotzin Jonathan J, Iseka Fany, Wright Jasmine, Basavappa Megha G, Clark Megan L, Ali Mohammed-Alkhatim, Abdel-Hakeem Mohamed S, Robertson Tanner F, Mowel Walter K, Joannas Leonel, Neal Vanessa D, Spencer Sean P, Syrett Camille M, Anguera Montserrat C, Williams Adam, Wherry E John, Henao-Mejia Jorge The long noncoding RNA regulates CD8 T cells in response to viral infection.[PMID31138702] Proceedings of the National Academy of Sciences of the United States of America 116(24): 11916-11925, Jun 2019.
Syrett Camille M, Paneru Bam, Sandoval-Heglund Donavon, Wang Jianle, Banerjee Sarmistha, Sindhava Vishal, Behrens Edward M, Atchison Michael, Anguera Montserrat C Altered X-chromosome inactivation in T cells may promote sex-biased autoimmune diseases. [PMID30944248 JCI insight 4(7), Apr 2019.
Syrett Camille M, Sindhava Vishal, Sierra Isabel, Dubin Aimee H, Atchison Michael, Anguera Montserrat C Diversity of Epigenetic Features of the Inactive X-Chromosome in NK Cells, Dendritic Cells, and Macrophages. [PMID30671059] Frontiers in immunology 9: 3087, 2018.
Le Coz Carole, Trofa Melissa, Syrett Camille M, Martin Anna, Jyonouchi Harumi, Jyonouchi Soma, Anguera Montserrat C, Romberg Neil CD40LG duplication-associated autoimmune disease is silenced by nonrandom X-chromosome inactivation.[PMID29499223] The Journal of allergy and clinical immunology 141(6): 2308-2311.e7, Jun 2018.
Syrett Camille M, Sierra Isabel, Berry Corbett L, Beiting Daniel, Anguera Montserrat C Sex-Specific Gene Expression Differences Are Evident in Human Embryonic Stem Cells and During In Vitro Differentiation of Human Placental Progenitor Cells. [PMID29993333] Stem cells and development 27(19): 1360-1375, Oct 2018.
Wang Jianle, Anguera Montserrat C In Vitro Differentiation of Human Pluripotent Stem Cells into Trophoblastic Cells. [PMID28362386] Journal of visualized experiments : JoVE(121), Mar 2017.
Syrett Camille M, Sindhava Vishal, Hodawadekar Suchita, Myles Arpita, Liang Guanxiang, Zhang Yue, Nandi Satabdi, Cancro Michael, Atchison Michael, Anguera Montserrat C Loss of Xist RNA from the inactive X during B cell development is restored in a dynamic YY1-dependent two-step process in activated B cells. [PMID28991910] PLoS genetics 13(10): e1007050, Oct 2017.
Penkala Ian, Wang Jianle, Syrett Camille M, Goetzl Laura, López Carolina B, Anguera Montserrat C lncRHOXF1, a Long Noncoding RNA from the X Chromosome That Suppresses Viral Response Genes during Development of the Early Human Placenta. [PMID27066803] Molecular and cellular biology 36(12): 1764-75, Jun 2016.
Wang Jianle, Syrett Camille M, Kramer Marianne C, Basu Arindam, Atchison Michael L, Anguera Montserrat C Unusual maintenance of X chromosome inactivation predisposes female lymphocytes for increased expression from the inactive X. [PMID27001848] Proceedings of the National Academy of Sciences of the United States of America 113(14): E2029-38, Apr 2016.

...

user_144608240
Related