Scraping Bandcamp fan collections via POST

user_14474566 0
Viewed 464

I've been trying to scrape Bandcamp fan pages to get a list of the albums they have purchased and I'm having trouble efficiently doing it. I wrote something with Selenium but it's mildly slow so I'd like to learn a solution that'd maybe send a POST request to the site and parse the JSON from there.

Here's a sample collection page: https://bandcamp.com/nhoward

Here's the Selenium code:

def scrapeFanCollection(url):
    browser = getBrowser()
    setattr(threadLocal, 'browser', browser)
    #Go to url
    browser.get(url)
    
    try:
        #Click show more button
        browser.find_element_by_class_name('show-more').click()
        
        #Wait two seconds
        time.sleep(2)
        #Scroll to the bottom loading full collection
        scroll(browser, 2)
    except Exception:
        pass
    
    #Return full album collection
    soup_a = BeautifulSoup(browser.page_source, 'lxml', parse_only=SoupStrainer('a', {"class": "item-link"}))
        
    #Empty array
    urls = []
    
    # Looping through all the a elements in the page source
    for item in soup_a.find_all('a', {"class": "item-link"}):
        url = item.get('href')
        if(url != None):
            urls.append(url)
    
    return urls
web-scraping python
3 Answers

The API can be accessed as follows:

$ curl -X POST -H "Content-Type: Application/JSON" -d \
'{"fan_id":82985,"older_than_token":"1586531374:1498564527:a::","count":10000}' \
https://bandcamp.com/api/fancollection/1/collection_items

I didn't encounter a scenario where a "older_than_token" was stale, so the problem boils down to getting the "fan_id" given a URL.

This information is located in a blob in the id="pagedata" element.

>>> import json
>>> import requests
>>> from bs4 import BeautifulSoup
>>> res = requests.get("https://www.bandcamp.com/ggorlen")
>>> soup = BeautifulSoup(res.text, "lxml")
>>> user = json.loads(soup.find(id="pagedata")["data-blob"])
>>> user["fan_data"]["fan_id"]
82985

Putting it all together (building upon this answer):

import json
import requests
from bs4 import BeautifulSoup

fan_page_url = "https://www.bandcamp.com/ggorlen"
collection_items_url = "https://bandcamp.com/api/fancollection/1/collection_items"
res = requests.get(fan_page_url)
soup = BeautifulSoup(res.text, "lxml")
user = json.loads(soup.find(id="pagedata")["data-blob"])

data = {
    "fan_id": user["fan_data"]["fan_id"],
    "older_than_token": user["wishlist_data"]["last_token"],
    "count": 10000,
}
res = requests.post(collection_items_url, json=data)
collection = res.json()

for item in collection["items"][:10]:
    print(item["album_title"], item["item_url"])

I'm using user["wishlist_data"]["last_token"] which has the same format as the "older_than_token" just in case this matters.

user_62433520

In order to get the entire collection i changed the previous code from

"older_than_token": user["wishlist_data"]["last_token"] to

user["collection_data"]["last_token"]

which contained the right token

user_194592550

Unfortunately for you, this particular Bandcamp site doesn't seem to make any HTTP API call to fetch the list of albums. You can check that by using your browser developer tools, Network tab, click on XHR filter. The only call being made seems to be fetching your collection details.

user_61583070
Related