[Suggestion] Allow an integer parameter for 'randomize_start' in executor/base.py about datatrove

[guipenedo]

maybe seconds will be more flexible for "softer" use cases

[guipenedo]

We encounter them often, usually quite a lot of tasks fail (even with randomize start). We simply relaunch them afterwards.
I believe the requests from our cluster all come from the same IP, so I do not really have any recommendations on this

[guipenedo]

Super interesting! Were these network speed improvements (to s3 I assume?) only during the beginning of the job (when each worker is listing all the files) or throughout it?

Also, what logs did you have in mind? Maybe you could add them to your upcoming PR?

[guipenedo]

I think it's a good idea. Would you be interested in creating a PR for this change? Maybe randomize_start could become randomize_start_duration with a default value of 0 (disabled)

[justHungryMan]

I’d be happy to create a PR for this change. Regarding the unit for randomize_start_duration, do you think it would be more appropriate to use seconds or minutes as the default unit of time?

[justHungryMan]

Regarding this issue, did you also encounter Too Many Requests errors while working with Common Crawl?

I’m interested in determining the optimal number of multi-processing instances per IP to avoid hitting S3’s limit. Could you share any insights or recommendations on this?

[justHungryMan]

Thank you for sharing your experience!

[justHungryMan]

@guipenedo

With the implementation of this feature and increasing the wait time, we've noticed a corresponding improvement in network speeds. It seems likely that there were existing bottlenecks due to excessive API calls. (wait time 180 -> 2500, network speed 180MBit/s -> 1.4GBit/s)

In light of this, it might be beneficial to include log messages that reflect these changes or provide guidelines for adequately setting the randomize_start_duration to prevent such bottlenecks.

And also I will do additional PR to examples/{some codes} for fixing randomize_start parameter, this weekend.