I am working on a web scraping projecting using python. I am supposed to scrape this website(https://www.fec.gov/data/receipts/?committee_id=C00703975&two_year_transaction_period=2020&data_type=processed). The issue I am having is that when you go through the data, the url never changes. I am unsure of how to proceed. From my experience when scraping a website the url changes when switching pages but this one doesn't change pages it just goes through the data on the same page. Thank you!
1 Answers
This website deals with a lot of data and it is dynamic. Within the same layout, the data displayed will keep changing. Another way to approach your problem (may not be the best) is to export the data and save it to a staging area and work on that data. You can regularly refresh your staging area from https://www.fec.gov/data/browse-data/?tab=bulk-data
This significantly reduces the need for a scraping program and lets you focus on actual data analysis.
Related