Speed up training? about coref

[mandarjoshi90]

Around 55K steps for SpanBERT base. Here's the final part of the log.

2019-06-27 00:47:50,244 - INFO - __main__ - [54000] evaL_f1=0.7766, max_f1=0.7766
2019-06-27 00:48:29,488 - INFO - __main__ - [54100] loss=2.11, steps/s=2.11
2019-06-27 00:49:05,383 - INFO - __main__ - [54200] loss=1.67, steps/s=2.11
2019-06-27 00:49:44,517 - INFO - __main__ - [54300] loss=1.57, steps/s=2.11
2019-06-27 00:50:22,661 - INFO - __main__ - [54400] loss=1.81, steps/s=2.11
2019-06-27 00:51:03,515 - INFO - __main__ - [54500] loss=2.94, steps/s=2.11
2019-06-27 00:51:43,155 - INFO - __main__ - [54600] loss=1.69, steps/s=2.11
2019-06-27 00:52:16,367 - INFO - __main__ - [54700] loss=2.86, steps/s=2.11
2019-06-27 00:52:52,681 - INFO - __main__ - [54800] loss=0.89, steps/s=2.11
2019-06-27 00:53:35,524 - INFO - __main__ - [54900] loss=2.33, steps/s=2.11
2019-06-27 00:54:16,111 - INFO - __main__ - [55000] loss=1.13, steps/s=2.12
2019-06-27 00:55:44,326 - INFO - __main__ - [55000] evaL_f1=0.7771, max_f1=0.7771

[mandarjoshi90]

The usual way would be increasing the batch size. The code has been adapted from e2e-coref which pretty much "hardcodes" a batch size of 1. You'll have to do a significant bit of rewriting to use bigger batches.

Just to make sure I'm not misdiagnosing the problem, how large are your sequences? This kind if utilization does not appear to be normal (the model itself should be pretty big); I suspect it's not using the GPU at all. Did you set GPU=0 during your call to train.py? Is TF detecting the GPUs?

[armancohan]

Thanks, yes there was an issue with Tensofrlow not using the gpu at all (just put some initial load on it). With reinstalling it and recompiling the coref_kernels the problem seems to be solved and I'm getting full gpu utilization.
Another question, do you remember roughly how many steps the model needs training for peak performance and the final loss values? Thanks!

[mandarjoshi90]

Hi Arman. Were you able to train this? Which base model are you using -- BERT or SpanBERT?

[armancohan]

Hi Mandar, thanks for checking. Yes, I was able to train this, but my eval F1 was maxing at around 67%. Actually, I started from a variant of SciBERT. Are there any important hyperparameters that I need to change? I kept everything same as what is in the default config files.

[mandarjoshi90]

I take it that you're training this on some other coref dataset (and hence SciBERT)? If so, I'm not sure what the baselines look like. If this is still OntoNotes, that number seems rather low. Might be a good idea to take the pretrained coref model an evaluate on your domain.

Generally, I've found a variance of a couple of points across different learning rates. Not sure if that's really helpful. Sorry.

[armancohan]

Yes, I think the problem could be domain mismatch.
This is helpful. Thanks!