I know this question has come up a billion times and that what I want to do isn't Selenium's intended purpose, but I don't know of anything else that can achieve this purpose. I've read those answers to the best of my abilities, and lots of documentation, but I could use some pointers.
I'm trying to download a number of files from CDC Compressed Mortality which requires one to 1) press 'I Agree', 2) Navigate through a bunch of menus, check boxes and dropdown boxes and 3) press 'Send' and wait for the file to automatically start downloading.
There are a few very troublesome limitations of the webpage which has led me to look for ways to automate this.
- Exporting the resulting dataset using the 'send' button is inconsistent with some settings, omitting data points, namely in some cases the file produced doesn't reflect the settings for suppressed/omitted values
- The page limits the number of lines of data
I've found that by exporting data by individual states, the two points above are no longer a problem however this is super labor intensive, and not much fun. I should note that I have no experience with Python (or real programming), but the documentation seemed good enough for me to get this to somewhat work. This is what I would like to do:
- Navigate to page, press 'I accept'
- Select a State
- Fill in some options
- Hit send
- Wait for file to finish downloading
Since setting the Firefox profile skips the download box, the file automatically begins downloading. I am able to determine whether a file has finished downloading by finding the newest file and waiting until the .part extension is gone.
The code works up until it tries to select 12 Florida, then, everything just stops. Firefox freezes and no file ever begins to download. Repeating this manually, it goes without a problem.
from selenium import webdriver
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.support.ui import Select, WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import os, unittest, time, re
basedir = os.getcwd()
savedir = os.path.join(basedir, 'download')
# Check download status
def checkdownload():
os.chdir(savedir)
files = filter(os.path.isfile, os.listdir(os.getcwd()))
files = [os.path.join(os.getcwd(), f) for f in files] # add path to each file
files.sort(key=lambda x: os.path.getmtime(x))
if not files :
newest_file = "no"
else :
newest_file = files[-1]
os.chdir(basedir)
return newest_file
# Set user profile
fp = webdriver.FirefoxProfile()
fp.set_preference("browser.download.folderList",2)
fp.set_preference("browser.download.manager.showWhenStarting",False)
fp.set_preference("browser.download.dir",basedir+'\\download')
fp.set_preference("browser.helperApps.neverAsk.saveToDisk","text/plain")
# Before anything downloads
previousnew = checkdownload()
# Create a new instance of the Firefox driver
b = webdriver.Firefox(firefox_profile=fp)
b.get("http://wonder.cdc.gov/cmf-icd9.html")
b.implicitly_wait(1)
### Find states
b.find_element_by_xpath("/html/body/div/form/div[3]/div/center/input").click() # Press 'I agree'
# print [o.text for o in Select(b.find_element_by_id("SD16.V1")).options]
# Make a list of all the states available
options = Select(b.find_element_by_id("codes-D16.V9")).options
optionsList = []
for option in options:
optionsList.append(option.get_attribute("value"))
if option.get_attribute("value") == "*All*":
optionsList.remove(option.get_attribute("value")) # Remove the *All* option
# Loop over states individually
for optionValue in optionsList:
print "\nRunning on %s" % optionValue
b.get("http://wonder.cdc.gov/cmf-icd9.html")
b.implicitly_wait(1)
b.find_element_by_xpath("/html/body/div/form/div[3]/div/center/input").click() # Press 'I agree'
print "Add Selections"
# 1. Table layout, id = SB_1 ... SB_5
Select(b.find_element_by_id("SB_1")).select_by_visible_text("Age Group")
Select(b.find_element_by_id("SB_2")).select_by_visible_text("Race")
Select(b.find_element_by_id("SB_3")).select_by_visible_text("Gender")
Select(b.find_element_by_id("SB_4")).select_by_visible_text("County")
Select(b.find_element_by_id("SB_5")).select_by_visible_text("Year")
# 2. Location, id = codes-D16.V9
Select(b.find_element_by_id("codes-D16.V9")).deselect_by_index(0) # remove *All* option
Select(b.find_element_by_id("codes-D16.V9")).select_by_value(optionValue) # selection
# Age Group, id = SD16.V5
Select(b.find_element_by_id("SD16.V5")).deselect_by_index(0) # remove *All* option
Select(b.find_element_by_id("SD16.V5")).select_by_value('20-24')
Select(b.find_element_by_id("SD16.V5")).select_by_value('25-34')
Select(b.find_element_by_id("SD16.V5")).select_by_value('35-44')
Select(b.find_element_by_id("SD16.V5")).select_by_value('45-54')
Select(b.find_element_by_id("SD16.V5")).select_by_value('55-64')
# Gender, id = SD16.V7
# Race, id = SD16.V8
# Hisp, Does not exist in this file
# Year, id = SD16.V1
yr = 1997, 1998
Select(b.find_element_by_id("SD16.V1")).deselect_by_index(0) # remove *All* option
select = Select(b.find_element_by_id("SD16.V1"))
for o in yr:
select.select_by_value("%s" % o)
# ICD-9 Codes, id = codes-D16.V2
# Rate per, id = SO_rate_per
# Other options
b.find_element_by_id("export-option").click()
b.find_element_by_id("CO_show_totals").click()
b.find_element_by_id("CO_show_zeros").click()
b.find_element_by_id("CO_show_suppressed").click()
# Submit
print "Submit"
b.find_element_by_xpath("/html/body/div/form/table/tbody/tr/td/div[2]/div[2]/center/input[1]").click()
# Check if file has begun downloading
print "Waiting for new file"
new = checkdownload()
while previousnew == new:
print "... waiting"
new = checkdownload()
continue
print "Waiting for download to finish"
# New file found, wait until it doesn't have .part extension
new = checkdownload()
while os.path.splitext(new)[1] == ".part":
print "... downloading"
new = checkdownload()
continue
print "Downloaded"
continue
b.quit()
I'm unable to determine why this happens, as no error is produced. Any ideas on what I am doing wrong?
PS. I realize my code is horrendous, and that one honest answer would be 'You're doing everything wrong'. But, I'm truly at a loss why this simple script is behaving like this.