This is a good website. When you go in depth on how the website loads the data and when you see the source code of the website, all the data is held inside the script tag in a Javascript Object form. It's not a JSON. If you pull the data inside the script then you will get all the article links, images etc...
Because it's a Javascript Object, you need 3rd party libraries to convert to a json. I used demjson library to do this job - https://github.com/dmeranda/demjson
The below script saves the data to a json file. Once you have the json getting all the links shouldn't be difficult.
import requests, demjson, json
from bs4 import BeautifulSoup
res = requests.get("https://edition.cnn.com/politics")
soup = BeautifulSoup(res.text, "html.parser")
script = None
for i in soup.find_all("script"):
if "window.CNN" in i.text:
script = i.get_text(strip=True)
if script is None: print("No data found")
else:
data = script.partition("CNN.contentModel")[-1].partition("FAVE.settings")[0]
json_data = demjson.decode(data[data.index('{'):-1])
with open("data.json", "w") as f:
json.dump(json_data, f)
Output:
{
"hasVideo": false,
"layout": "no-rail",
"vertical": "politics",
"sectionName": "politics",
"pageType": "section",
"env": "prod",
"type": "page",
"analytics": {
"pageTop": {},
"headline": "",
"author": "",
"showName": "",
"subSectionName": "",
"isArticleVideoCollection": false,
"publishDate": "2014-02-27T01:35:32Z",
"lastUpdatedDate": "2020-08-06T09:31:15Z",
"pageBranding": "10-minute-preview",
"cep_topics": {
"brsf": [],
"buzz": [],
"iabt": [],
"sent": [
"16B6"
],
"tags": [],
"shortSource": "se_politics",
"source": "section_politics"
},
"chartbeat": {
"sections": ""
},
"branding_content_page": "10-minute-preview",
"branding_content_zone": [
"default"
],
"branding_content_container": [
"default"
],
"branding_content_card": [
""
]
},
"edition": "international",
"sourceId": "section_politics",
"title": "CNNPolitics - Political News, Analysis and Opinion",
"siblings": {
"articleList": [
{
"uri": "/2020/08/06/politics/donald-trump-mail-in-voting-election/index.html",
"headline": "Trump's mail-in voting falsehoods are part of a wide campaign to discredit the election",
"thumbnail": "//cdn.cnn.com/cnnnext/dam/assets/200805203446-02-donald-trump-0805-small-11.jpg",
"duration": "",
"description": "<a href=\"http://www.cnn.com/specials/politics/president-donald-trump-45\" target=\"_blank\">President Donald Trump's</a> barrage of <a href=\"http://www.cnn.com/2020/08/05/politics/fact-check-trump-fox-friends-pandemic-biden-protests/index.html\" target=\"_blank\">challenges to the reputation, structures and traditions</a> of elections is conjuring up a contentious and potentially constitutionally critical three-month period for America's democracy.",
"layout": ""
},
{
"uri": "/2020/08/05/politics/donald-trump-press-briefing-beirut-coronavirus-voting-fact-check/index.html",
"headline": "Fact check: At briefing, Trump continues to mislead on coronavirus, mail-in voting and Beirut",
"thumbnail": "//cdn.cnn.com/cnnnext/dam/assets/200805203446-02-donald-trump-0805-small-11.jpg",
"duration": "",
"description": "President Donald Trump ended his Wednesday much like he began it, by repeating falsehood after falsehood.",
"layout": ""
},
{
"uri": "/2020/08/05/politics/state-department-russian-disinformation-report/index.html",
"headline": "US accuses Russia of conducting sophisticated disinformation and propaganda campaign ",
"thumbnail": "//cdn.cnn.com/cnnnext/dam/assets/170626163907-russia-dnc-hacking-ron-2-00000808-small-11.jpg",
"duration": "",
"description": "A <a href=\"https://content.govdelivery.com/attachments/USSTATEBPA/2020/08/05/file_attachments/1512230/Pillars%20of%20Russias%20Disinformation%20and%20Propaganda%20Ecosystem_08-04-20%20%281%29.pdf\" target=\"_blank\">new report</a> from the US State Department accuses Russia of conducting a sophisticated disinformation and propaganda campaign that uses a variety of approaches including Kremlin-aligned news sites to promote their agenda.",
"layout": ""
},
{
"uri": "/2020/08/05/politics/fact-check-trump-ad-biden-basement-delaware-photos-iowa/index.html",
"headline": "<strong>Fact check: </strong>Trump ad edits out microphone and trees from Biden photo to make him seem alone in basement",
"thumbnail": "//cdn.cnn.com/cnnnext/dam/assets/200803235935-01-joe-biden-campaign-0720-small-11.jpg",
"duration": "",
"description": "A new <a href=\"https://www.youtube.com/watch?v=9PUfxZQa7WQ&feature=emb_title\" target=\"_blank\">ad</a> from President Donald Trump's campaign deceptively alters a photo of former Vice President Joe Biden campaigning outdoors in Iowa to make it seem as if Biden is \"hiding\" in his Delaware basement.",
"layout": ""
},
{
"uri": "/2020/08/05/politics/mark-meadows-unemployment-benefits-extension-coronavirus-relief-cnntv/index.html",
"headline": "White House chief of staff floats executive action on unemployment and evictions if Congress can't strike deal",
"thumbnail": "//cdn.cnn.com/cnnnext/dam/assets/191219132522-03-mark-meadows-lead-image-small-11.jpg",
"duration": "",
"description": "White House chief of staff Mark Meadows said Wednesday that <a href=\"https://www.cnn.com/specials/politics/president-donald-trump-45\" target=\"_blank\">President Donald Trump</a> is prepared to take executive action on eviction protection and extending enhanced unemployment benefits if Congress isn't close to <a href=\"https://www.cnn.com/2020/08/05/politics/congress-stimulus-negotiations/index.html\" target=\"_blank\">a coronavirus recovery package</a> by Friday. ",
"layout": ""
},
{
"uri": "/2020/08/05/politics/trump-campaign-four-debates/index.html",
"headline": "Trump campaign calls for a fourth presidential debate, citing early voting",
"thumbnail": "//cdn.cnn.com/cnnnext/dam/assets/200709094609-trump-biden-split-small-1-1.jpg",
"duration": "",
"description": "<a href=\"https://www.cnn.com/election/2020/candidate/trump\" target=\"_blank\">Donald Trump's</a> presidential campaign called for an additional presidential debate in a letter to the Commission on Presidential Debates on Wednesday. ",
"layout": ""
},
{
"uri": "/2020/08/05/politics/schlapp-mail-voting-expansion-nevada-fact-check/index.html",
"headline": "<strong>Fact Check: </strong>With vote by mail expansion, can Nevada voters cast ballots after Election Day?",
"thumbnail": "//cdn.cnn.com/cnnnext/dam/assets/200610082429-voting-north-las-vegas-small-11.jpg",
"duration": "",
"description": "President Donald Trump reversed his stance on voting by mail Tuesday when he <a href=\"https://www.cnn.com/2020/08/04/politics/donald-trump-mail-in-voting-florida/index.html\" target=\"_blank\">tweeted</a> that doing so in Florida is \"safe and secure.\" When asked about the reversal later Tuesday afternoon, Trump seemed to imply that Republican-run states with existing mail-in voting programs were up to par, but Democratic states establishing or expanding mail-in voting during the pandemic were not.",
"layout": ""
},
...
...
...
