Facebook Url Returning An Mobile Version Url Response In Scrapy

I had one question , may be its wierd , but wanna know it, I tried to access facebook with URl www.facebook.com through scrapy. I had given it in start_url. After running i got the

Solution 1:

There is a global setting for that: USER_AGENT

UPDATE :

You know, maybe dealing with the mobile version is an advantage after all. Other sites redirect browsers to other pages when no javascript can be executed:

<noscript><metahttp-equiv="refresh"content="0; URL=/homedepot?_fb_noscript=1" /></noscript>

Dealing with a no js version or a mobile version of the site means less sizes of the pages and less additional info on the page - so the html will not change much over times, and your xpath queries will still work.

In that case just disable JS in Firefox or set a different User-Agent in it to get the same pages scrapy gets. Here are more hints on how to use Firefox for testing scrapy: Using Firefox for scraping