Scrape Data Through Xpath From Div That Contains Javascript In Scrapy Python

I am working on scrapy , i am scraping a site and using xpath to scrape items. But some of the div contains javascript, so when i used xpath until the div id that contains javascri

Solution 1:

<divclass="subContent2"><divid="contentDetails"><divclass="eventDetails"><h2><ahref="javascript:;"onclick="jdevents.getEvent(117032)">Some data</a></h2></div></div></div>

The problem is not the javascript code in this case to get 'Some data' string.

You need either to get the subnode:

required_data = hxs.select('//div[@class="subContent2"]/div[@id="contentDetails"]/div[@class="eventDetails"]/h2/a/text()')

or use string function:

required_data = hxs.select('string(//div[@class="subContent2"]/div[@id="contentDetails"]/div[@class="eventDetails"])')