The fnc-adversarial from cjstadler

fnc-adversarial's People

Contributors

Watchers

fnc-adversarial's Issues

make adversarial sample generation parallel

three options

split input csv and process with multiple python processes; parallelize with shell script
add argument to process i-j items in the list; parallelize with shell script
native parallelization with python

currently have number 3 implemented with joblib, here's performance on a small subset.
Clearly, there's something happening (pickling results?) at the end (after 91 done) that causes it to take 2 min for the final join. It may not matter though because I think this time will be constant with larger subsets.

Replacing 4 words in each example
[Parallel(n_jobs=38)]: Using backend LokyBackend with 38 concurrent workers.
[Parallel(n_jobs=38)]: Done   9 tasks      | elapsed:   11.5s
[Parallel(n_jobs=38)]: Done  22 tasks      | elapsed:   12.6s
[Parallel(n_jobs=38)]: Done  36 out of 100 | elapsed:   13.4s remaining:   23.8s
[Parallel(n_jobs=38)]: Done  47 out of 100 | elapsed:   14.3s remaining:   16.1s
[Parallel(n_jobs=38)]: Done  58 out of 100 | elapsed:   15.8s remaining:   11.5s
[Parallel(n_jobs=38)]: Done  69 out of 100 | elapsed:   16.8s remaining:    7.6s
[Parallel(n_jobs=38)]: Done  80 out of 100 | elapsed:   19.2s remaining:    4.8s
[Parallel(n_jobs=38)]: Done  91 out of 100 | elapsed:   20.9s remaining:    2.1s
[Parallel(n_jobs=38)]: Done 100 out of 100 | elapsed:  2.2min finished

Recommend Projects