Skip to content Skip to sidebar Skip to footer

Change Number Of Running Spiders Scrapyd

Hey so I have about 50 spiders in my project and I'm currently running them via scrapyd server. I'm running into an issue where some of the resources I use get locked and make my s

Solution 1:

This can be controlled by scrapyd settings. Set max_proc to 1:

max_proc

The maximum number of concurrent Scrapy process that will be started.

Post a Comment for "Change Number Of Running Spiders Scrapyd"