how to get the exact text that include in multiple class

user_13012011 0
Viewed 86

I am new on python. I try to scrape the data from the websites. but I failed to extract that data which I needed. here I share my python code

import requests
from bs4 import BeautifulSoup
url = 'https://v2.sherpa.ac.uk/view/publisher_list/1.html'
r = requests.get(url)
htmlContent = r.content
soup = BeautifulSoup(htmlContent, 'html.parser')
title = soup.title
print(soup.find_all("div", {"class" :["ep_view_page ep_view_page_view_publisher_list", "row"]}))

what I face is I need the data that is in div class = row but here there are two div class with row name. and one more thing that what should I write to get the data from the multiple URL and pages if you see that there is the tag having class col span-6 and col span-3; on href tag when I link on that it opens one new page.

<div class="row">
  <div class="col span-6">
    <a href="http://v2.sherpa.ac.uk/id/publisher/50974?template=romeo">'Grigore Antipa' National Museum of Natural History</a>
  </div>
  <div class="**col span-3**">
    <strong>Romania</strong>
    <span class="label">Country</span>
  </div>
  <div class="**col span-3**">
    <strong>1 [<a href="/view/publication_by_publisher/50974.html">view</a> ]</strong>
    <span class="label">Publication Count</span>
  </div>
</div>

and here I share the sitemap

<div class="row">
            <h1 class="h1_like_h2">Publishers</h1>
            <div class="ep_view_page ep_view_page_view_publisher_list">

</p><div class="row">
  <div class="**col span-6">
    <a href="http://v2.sherpa.ac.uk/id/publisher/50974?template=romeo">'Grigore Antipa' National Museum of Natural History</a>
  </div>
  <div class="col span-3">
    <strong>Romania</strong>
    <span class="label">Country</span>
  </div>
  <div class="col span-3">
    <strong>1 [<a href="/view/publication_by_publisher/50974.html">view</a> ]</strong>
    <span class="label">Publication Count</span>
  </div>
</div>

<p></p><a name="group_=28"></a><h2>(</h2><p>


</p><div class="row">
  <div class="col span-6">
    <a href="http://v2.sherpa.ac.uk/id/publisher/13937?template=romeo">(ISC)²</a>
  </div>
  <div class="col span-3">
    <strong>United States of America</strong>
    <span class="label">Country</span>
  </div>
  <div class="col span-3">
    <strong>1 [<a href="/view/publication_by_publisher/13937.html">view</a> ]</strong>
    <span class="label">Publication Count</span>
  </div>
</div>

<p></p><a name="group_1"></a><h2>1</h2><p>


</p><div class="row">
  <div class="col span-6">
    <a href="http://v2.sherpa.ac.uk/id/publisher/1939?template=romeo">1066 Tidsskrift for historie</a>
  </div>
  <div class="col span-3">
    <strong>Denmark</strong>
    <span class="label">Country</span>
  </div>
  <div class="col span-3">
    <strong>1 [<a href="/view/publication_by_publisher/1939.html">view</a> ]</strong>
    <span class="label">Publication Count</span>
  </div>
</div>

<p></p><a name="group_A"></a><h2>A</h2><p>

**so on.......**
web-scraping python-3.x
1 Answers

I'm not really sure if that's what you wanted, but I've had some fun getting this stuff.

Basically, the code below scrapes the entire page - 4487 entries - for:

Note: This is just a sample of data.

  • The name of the entity - ANZAMEMS (Australian and New Zealand Association for Medieval and Early Modern Studies)
  • The URL to its sub-page - http://v2.sherpa.ac.uk/id/publisher/1853?template=romeo
  • The country - Australia
  • The view count - 1
  • The so called "publisher url" - https://v2.sherpa.ac.uk//view/publication_by_publisher/1853.html

and spits all of this out to a .csv file that looks like this:

enter image description here

Here's the code:

import csv
import requests
from bs4 import BeautifulSoup


def make_soup():
    p = requests.get("https://v2.sherpa.ac.uk/view/publisher_list/1.html").text
    return BeautifulSoup(p, "html.parser")


main_soup = make_soup()
col_span_6_soup = main_soup.find_all("div", {"class": "col span-6"})
col_span_3_soup = main_soup.find_all("div", {"class": "col span-3"})


def get_names_and_urls():
    data = [a.find("a") for a in col_span_6_soup if a is not None]
    return [[i.text, i.get("href")] for i in data if "romeo" in i.get("href")]


def get_countries():
    return [c.find("strong").text for c in col_span_3_soup[::2]]


def get_views_and_publisher():
    return [
        [
            i.find("strong").text.replace(" [view ]", ""),
            f"https://v2.sherpa.ac.uk/{i.find('a').get('href')}",
        ] for i in col_span_3_soup[1::2]
    ]


table = zip(get_names_and_urls(), get_countries(), get_views_and_publisher())
with open("loads_of_data.csv", "w") as output:
    w = csv.writer(output)
    w.writerow(["NAME", "URL", "COUNTRY", "VIEWS", "PUBLISHER_URL"])
    for col1, col2, col3 in table:
        w.writerow([*col1, col2, *col3])
    print("You've got all the data!")
user_61067910
Related