Submitting bootstrap run in batches can leave lots of unused compute about bbr

[seth127]

Thanks for capturing this @kylebaron do you think we should move this to an internal Metworx ticket, or do you think it's worth looking into whether the way bbr is submitting models is playing into this?

[kylebaron]

I think this part is relevant to the way that bbr is doing it; you can get some really skewed run times so I think this batching strategy will have problems sooner than later

This set was appropriately scaled, but new jobs can get scheduled until every run of previous batch finishes

Not terrible for this example that runs fast and is easy; this won't work when you get a much more complicated model and the variability in run time is large, with some runs taking very long to finish

internal:~$ qstat -f |grep amd; qstat |grep -c Run
[email protected] BIP   0/0/16         5.35     lx-amd64
[email protected] BIP   0/0/16         5.45     lx-amd64
[email protected] BIP   0/0/16         5.44     lx-amd64
[email protected] BIP   0/0/16         6.06     lx-amd64
[email protected] BIP   0/0/16         5.71     lx-amd64
[email protected] BIP   0/1/16         5.60     lx-amd64
[email protected] BIP   0/1/16         5.29     lx-amd64
2