`.download()` scalability ? about wikiapi

[kanasimi]

1. I did not imagine this, but I reserved the possibility for this.
2. You know I have the apihighlimits permission in wiki commons, so I often use this... but it should work for users without apihighlimits permission too. There should no limit for downloading large categories.
3. I have processed for categories with 100K+ files.
4. Yes, the library will skip files existed.
5. No. I have not coding this yet.
6. I have never heard of wiki commons blocking peoples downloading files, so...

[kanasimi]

Well, it seems I need to do some works...

[kanasimi]

What categories do you want to synchronize?

[hugolpz]

Aim is to provide convenient dumps for each category in Category:Lingua_Libre_pronunciation.
The largest ones are 60k (ben) and 250k (fra) files strong. The whole 130 categories contain 700,000 files.

The point for WikiapiJS .download() is scalability, ability to handle such large categories with resilience and speed, their initial download and their later periodic update. Ideally weekly.

[hugolpz]

Nice !

[hugolpz]

This scale up question is handled in two related issues:

• #55
• #53
• #57