Trouble Parsing Text Using Beautifulsoup And Python
I am trying to retrieve the comment section on regulations.gov pages. An example is the paragraph 'Restrictions on Proprietary Trading... with free market driven valuations.' on ht
Solution 1:
You need to let PhantomJS
explicitly wait for the element to become present before reading the page_source
. Worked for me:
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.PhantomJS()
driver.get("http://www.regulations.gov/#!documentDetail;D=OCC-2011-0014-0032")
wait = WebDriverWait(driver, 10)
wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, "div.GGAAYMKDGNE")))
Post a Comment for "Trouble Parsing Text Using Beautifulsoup And Python"