I am trying to pull descriptions of a few listings, and my code is grabbing other details like address or price, but its not getting description. Any ideas what's wrong here?
Here's an example: http://www.redfin.com/IL/Chicago/1515-S-Prairie-Ave-60605/unit-P-273/home/12677887
description = WebDriverWait(driver, 0.01).until(EC.presence_of_element_located((By.XPATH, '/html/body/div[1]/div[10]/div[2]/div[7]/section/div/div/div/div/div/div[1]/div/div[1]/div/div/p/span/text()'))).text
It is not recommended to use absolute XPath strings, as these can easily be outdated by any update to the website itself. I recommended going to the website and copying the relative XPATH instead. This way you can drill into the HTML and attempt to figure out whether your WebElement actually exists or not.
Update: Your wait time is also rather small. Try changing this to a higher value (say 10) to make sure that this isn't the issue as well.
To get description use css selector and explicit wait:
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By as By
wait = WebDriverWait(driver, 20)
wait.until(EC.visibility_of_element_located(
(By.CSS_SELECTOR, "#marketing-remarks-scroll>.text-base>span")))
description = driver.find_element_by_css_selector("#marketing-remarks-scroll>.text-base>span").text
The wait you are using is not the same as time.sleep(), as you probably think. It waits until element is found and after that the program continues to execute. Unlike time.sleep() which will wait for a given period of time and will not let a program to execute until wait given time is finished.
The approach should be - using reliable selectors (xpath and css) and wait until they are found.