Create benchmark configs about mlprogram

[HiroakiMikami]

• NL2Prog
  • Dataset: Hearthstone
  • Training: about 20 epoch (CPU)
  • Evaluation timeout (66samples): 15sec (small), 180sec (large)
• PbE (w/o input)
  • Dataset: CSG
  • Training: 6000iter (CPU)
  • Evaluation timeout (30samples): 30sec (small), 360sec (large)
• auto repair

[HiroakiMikami]

TODO

• CSG (small) should increase the problem size (?)
  • baseline model -> small evaluation => generation-rate=50%
• Select CPU/GPU for benchmarking
• Benchmark script
• Disable multiprocess evaluation. Handle same as distributed training If want to do so.

[HiroakiMikami]

Notes

• Hearthstone training throughput, baseline LSTM model
  • w/ CPU (my local desktop, WSL2): about 3 min / epoch (300ms /sample @ 1 process, 250ms/sample @ 2process 225ms/sample @ 4process w/ OMP_NUM_THREADS=1)
  • w/ GPU (Colab): about 2.5~3 min / epoch