Extract URL from Mixcloud playlist Python Selenium

user_14461403 0
Viewed 86

I need some help extracting the URL from the anchor href tags from the mixcloud.com user page. I understand the page is generated using javascript and i am using selenium to get around this, i have had success using similar method with Youtube playlist but i can not get this to work. Here is the url of the mixes i am trying to extract urls for.

https://www.mixcloud.com/caimanblack/


<div class="AudioCard__DetailsContainer-sc-1ltw4p1-6 euvMwc">
<div class="AudioCardTitle__Container-sc-1kxsru9-1 hGblkL">
<div class="AudioCardPlayButton__PlayButtonContainer-sc-1iib1iv-0 diYcBm AudioCardTitle__PlayButton-sc-1kxsru9-2 dDAfgc" title="Play">
<div class="AudioCardPlayButton__PlayButtonIconContainer-sc-1iib1iv-3 izFLOx">
<svg width="24" height="24" viewBox="0 0 24 24" xmlns="http://www.w3.org/2000/svg">
<title>Icon / 24 / Play Solid</title>
<path fill="#1E2337" d="M20 10.67L7.9 4 6 4.9v14.42l1.9.68L20 13.33z" fill-rule="evenodd"></path></svg></div>
<svg version="1.1" xmlns="http://www.w3.org/2000/svg" class="AudioCardPlayButton__PlayButtonRings-sc-1iib1iv-2 iIGcCU">
<circle class="ring-listened" cx="50%" cy="50%" r="22" style="stroke-dashoffset: 34.5575px; stroke-dasharray: 0px, 138.23px; stroke: rgb(243, 178, 166);"></circle>
<circle class="ring-remaining" cx="50%" cy="50%" r="22" style="stroke-dashoffset: 172.788px; stroke-dasharray: 138.23px, 0px;"></circle></svg></div>
<div class="AudioCardTitle__DetailsContainer-sc-1kxsru9-3 cTqEgM">
<a class="AudioCardTitle__PlainLink-sc-1kxsru9-0 AudioCardTitle__TitleLink-sc-1kxsru9-4 jKwuem" href="/caimanblack/93-94-dark-jungle-mix-5/">93-94 Dark Jungle Mix 5</a>
<div class="AudioCardTitle__OwnerText-sc-1kxsru9-5 gxeIb">by&nbsp;
<span class="hovercard-anchor AudioCardTitle__OwnerHovercard-sc-1kxsru9-7 cicNsQ">
<a class="AudioCardTitle__PlainLink-sc-1kxsru9-0 AudioCardTitle__OwnerLink-sc-1kxsru9-6 YOGda" href="/caimanblack/">Caiman Black</a>

here is what i have tried and i get nothing.

for item in range(20): 
    WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.TAG_NAME, "body"))).send_keys(Keys.END)


mixes = driver.find_elements_by_class_name('styles__SectionContainer-obazx4-0 fKqoOc')


for mix in mixes:
    link = mix.find_element_by_xpath('.//*[@class="AudioCardTitle__Container-sc-1kxsru9-1 hGblkL"]')
    print(link)
webdriverwait selenium xpath css python
2 Answers

Could you try this ? :

mixes = driver.find_elements_by_xpath("//a[contains(@class,'AudioCardTitle')]")
for mix in mixes:
    print(mix.get_attribute("href"))
user_148643080

To get all the links Use WebDriverWait() and wait for visibility_of_all_elements_located() and the following CSS selector.

driver.get("https://www.mixcloud.com/caimanblack/")
AllLinks=[item.get_attribute("href") for item in WebDriverWait(driver,10).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR,"a[class^='AudioCardTitle__PlainLink'][class$='jKwuem']")))]
print(AllLinks)
print("Total Links : {}".format(len(AllLinks)))

Console Output:

['https://www.mixcloud.com/caimanblack/93-94-dark-jungle-mix-5/', 'https://www.mixcloud.com/caimanblack/93-95-intelligent-db-mix-2/', 'https://www.mixcloud.com/caimanblack/94-95-jungle-db-rollers-mix-2/', 'https://www.mixcloud.com/caimanblack/94-95-jungle-db-rollers-mix-1/', 'https://www.mixcloud.com/caimanblack/94-95-jungle-db-mix-1/', 'https://www.mixcloud.com/caimanblack/94-95-jungle-mix-1/', 'https://www.mixcloud.com/caimanblack/93-95-intelligent-db-mix-1/', 'https://www.mixcloud.com/caimanblack/94-96-intelligent-db-mix-4/', 'https://www.mixcloud.com/caimanblack/94-96-intelligent-db-mix-3/', 'https://www.mixcloud.com/caimanblack/dub-7-king-tubby-others/', 'https://www.mixcloud.com/caimanblack/94-jungle-mix-2/', 'https://www.mixcloud.com/caimanblack/94-jungle-mix-1/', 'https://www.mixcloud.com/caimanblack/94-jungle-death/', 'https://www.mixcloud.com/caimanblack/94-94-jungle-mix-4/', 'https://www.mixcloud.com/caimanblack/93-94-jungle-mix-3/', 'https://www.mixcloud.com/caimanblack/96-99-dark-tech-db-3/', 'https://www.mixcloud.com/caimanblack/96-99-dark-tech-db-2/', 'https://www.mixcloud.com/caimanblack/96-99-dark-tech-db-1/', 'https://www.mixcloud.com/caimanblack/dub-6-king-tubby-the-upsetters-augustus-pablo-others/', 'https://www.mixcloud.com/caimanblack/dub-5-augustus-pablo-revolutionaries-aggrovators-others/', 'https://www.mixcloud.com/caimanblack/dub-4-king-tubby-the-upsetters-linval-thompson-others/', 'https://www.mixcloud.com/caimanblack/94-96-intelligent-db-mix-2/', 'https://www.mixcloud.com/caimanblack/94-96-intelligent-db-mix-1/', 'https://www.mixcloud.com/caimanblack/93-96-deep-jungle-mix-3/', 'https://www.mixcloud.com/caimanblack/93-96-deep-jungle-mix-2/', 'https://www.mixcloud.com/caimanblack/93-96-deep-jungle-mix-1/', 'https://www.mixcloud.com/caimanblack/96-99-jazz-funk-drum-bass-mix-1/', 'https://www.mixcloud.com/caimanblack/dub-90s-00s-mix-3/', 'https://www.mixcloud.com/caimanblack/dub-90s-00s-mix-2/', 'https://www.mixcloud.com/caimanblack/dub-90s-00s-mix-1/']
Total Links : 30

You could use below xpath as well result would be same.

driver.get("https://www.mixcloud.com/caimanblack/")
AllLinks=[item.get_attribute("href") for item in WebDriverWait(driver,10).until(EC.visibility_of_all_elements_located((By.XPATH,"//a[starts-with(@class,'AudioCardTitle__PlainLink') and contains(@class,'jKwuem')]")))]
print(AllLinks)
print("Total Links : {}".format(len(AllLinks)))

Now If you want to iterate you could use this.

for item in AllLinks:
    print(item)
user_108856840
Related