Running many instances of one spider about crawly

[oltarasenko]

Hey @serpent213,

Indeed, Crawly is built around the spider names, and there is no easy way to switch to something else right now.

However, it may be the case that you don't need it. Let me try to explain my points here:

1. Spider - is not a stateful process; it's just a set of callbacks. So there is no such thing as running 100 instances of the spider.
2. When we talk about the stateful part of it, it's all about Manager, RequestsStorage (scheduled requests), ItemsStorage (extracted items)

When I think about your case, as I understand it, you want a broad crawl that goes to multiple websites and extracts all the information from them. Probably there is some scheduler outside Crawly that just does something like:

Crawly.Engine.start_spider(BroadCrawler, start_urls: "google.com")
Crawly.Engine.start_spider(BroadCrawler, start_urls: "openai.com")

May it be the case, that you need an API to add extra requests to an already running spider?

[serpent213]

When I think about your case, as I understand it, you want a broad crawl that goes to multiple websites and extracts all the information from them. Probably there is some scheduler outside Crawly that just does something like:

Crawly.Engine.start_spider(BroadCrawler, start_urls: "google.com")
Crawly.Engine.start_spider(BroadCrawler, start_urls: "openai.com")

Exactly, that was my first attempt. Thank you for the inspiration, will look into it!