I am trying to write a web scraping code for this link, which has around 300 pages. There are some links (records) on each page that the data needs to be taken from the page of that record. For example, the DRAMP00005, has different fields, that I need to extract the required data and save it to the panda data frame.
I wrote the code:
from bs4 import BeautifulSoup
from lxml import html
import requests as rq
import re
import pandas as pd
import logging
import matplotlib.pyplot as plt
import pdb
import json
base_url='http://dramp.cpu-bioinfor.org/browse/All_Information.php?id='
url='{}DRAMP00005'.format(base_url)
page = rq.get(url)
htmlSoup = BeautifulSoup(page.content, "lxml")
divs_section=htmlSoup.select("div.bs-docs-section") # get the contents of all sections such as General Information, Activity Information,
#print(len(divs_section))
print(divs_section[1])
new_table = pd.DataFrame(columns=range(0,14), index = [url])
new_table.columns=['DRAMP ID','Peptide Name','Source','Family','Gene', 'Sequence','Sequence Length','UniProt Entry','Protein Existence', 'Biological Activity','Target Organism','Hemolytic Activity','Cytotoxicity','Binding Target']
#Add all text from html elements td width=61%
row_marker = 0
for dv in range(0,1): #range(len(divs_section)):
container=divs_section[dv].find_all("ul", {'class':'list-inline'}) # the content inside each section
#print(container)
for row in range(len(container)):
column_marker = 0
columns =container[row].find_all('li')
# print(len(columns))
for col in columns:
print(col.get_text())
new_table.iat[row_marker,column_marker] = col.get_text()
column_marker += 1
#print(len(container[row].find_all('li')))
new_table
and the output looked like this:
6
DRAMP ID
DRAMP00005
Peptide Name
Epicidin 280 (Bacteriocin)
Source
Staphylococcus epidermidis BN 280 (Gram-positive bacteria)
Family
Belongs to the lantibiotic family (Class I bacteriocin)
Gene
eciA
Sequence
SLGPAIKATRQVCPKATRFVTVSCKKSDCQ
Sequence Length
30
UniProt Entry
O54220
Protein Existence
Protein level
DRAMP ID Peptide Name Source Family Gene Sequence Sequence Length UniProt Entry Protein Existence Biological Activity Target Organism Hemolytic Activity Cytotoxicity Binding Target
http://dramp.cpu-bioinfor.org/browse/All_Information.php?id=DRAMP00005 Protein Existence Protein level NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
I am new to web-scraping and practicing to learn. I want to get the values in the columns of the table:
for example, DRAMP00005 in column with title DRAMP ID and etc.
How to correct this?
My other questions is how to repeat web scraping:
- on the same page and extracting from the 20 records (each record has a link directing to the information that needs to be extracted),
- Moving from the pages one by one until it reaches the last page of records (i.e., page 285)