Skip to content Skip to sidebar Skip to footer

Scrapy Unable To Make Request() Callback

I am trying to make recursive parsing script with Scrapy, but Request() function doesn't call callback function suppose_to_parse(), nor any function provided in callback value. I t

Solution 1:

Move the yield outside of the if statement:

for link in hxs.select('//a/@href').extract():
    url = link
    ifnot link.startswith('http://') andnot link.startswith('#'):
        url = (self.start_urls[0] + link).replace('//','/')

    print url
    yield Request(url, callback=self.suppose_to_parse)

Solution 2:

I'm not an expert but i tried your code and i think that the problem is not on the Request, the generated url's seems to be broken, if you add some url's to a list and iterate thorough them and yield the Request with the callback, it works fine.

Post a Comment for "Scrapy Unable To Make Request() Callback"