Training Not Working about amr

[HoltSpalding]

Here's what is printed when it crashes:
Exception in thread "main" java.lang.OutOfMemoryError
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at java.util.concurrent.ForkJoinTask.getThrowableException(ForkJoinTask.java:598)
at java.util.concurrent.ForkJoinTask.reportException(ForkJoinTask.java:677)
at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:735)
at java.util.stream.ReduceOps$ReduceOp.evaluateParallel(ReduceOps.java:714)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
at edu.uw.cs.lil.amr.data.LabeledAmrSentenceCollection.(LabeledAmrSentenceCollection.java:83)
at edu.uw.cs.lil.amr.data.LabeledAmrSentenceCollection$Creator.create(LabeledAmrSentenceCollection.java:169)
at edu.uw.cs.lil.amr.data.LabeledAmrSentenceCollection$Creator.create(LabeledAmrSentenceCollection.java:97)
at edu.cornell.cs.nlp.spf.explat.ParameterizedExperiment.readResrouces(ParameterizedExperiment.java:204)
at edu.cornell.cs.nlp.spf.explat.DistributedExperiment.readResrouces(DistributedExperiment.java:206)
at edu.uw.cs.lil.amr.exp.AmrExp.(AmrExp.java:105)
at edu.uw.cs.lil.amr.exp.AmrExp.(AmrExp.java:117)
at edu.uw.cs.lil.amr.exp.AmrGenericExperiment.main(AmrGenericExperiment.java:28)
at edu.uw.cs.lil.amr.Main.main(Main.java:61)
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
at com.google.common.collect.Tables.immutableCell(Tables.java:67)
at com.google.common.collect.StandardTable$CellIterator.next(StandardTable.java:323)
at com.google.common.collect.StandardTable$CellIterator.next(StandardTable.java:306)
at com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48)
at uk.ac.ed.easyccg.syntax.ParserAStar.parseAstar(ParserAStar.java:339)
at uk.ac.ed.easyccg.syntax.ParserAStar.doParsing(ParserAStar.java:226)
at uk.ac.ed.easyccg.syntax.ParserAStar.parseTokens(ParserAStar.java:120)
at edu.uw.cs.lil.amr.ccgbank.easyccg.EasyCCGWrapper.getSpans(EasyCCGWrapper.java:54)
at edu.uw.cs.lil.amr.data.LabeledAmrSentence.(LabeledAmrSentence.java:45)
at edu.uw.cs.lil.amr.data.LabeledAmrSentenceCollection.lambda$new$0(LabeledAmrSentenceCollection.java:81)
at edu.uw.cs.lil.amr.data.LabeledAmrSentenceCollection$$Lambda$39/1560160481.apply(Unknown Source)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
at java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:747)
at java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:721)
at java.util.stream.AbstractTask.compute(AbstractTask.java:316)
at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)

[yoavartzi]

It looks like a memory problem. I don't remember how much memory we used for training, but our machines had quite a bit of memory, maybe 64GB or 128GB. The exception seems to be happening in one of the threads. We don't have a graceful way to bring down the entire system when a thread throws an exception. Whatever happens after this kind of exception is not to be relied on, and you should just kill the process.

[yoavartzi]

One thing you can do is limit to very short sentences. Memory consumption is tightly coupled with sentence length due to the use of a CKY chart. The model that you will get might not be good, but it will be a good way to test your setup.

[HoltSpalding]

I fixed my memory problems and was able to train a model on all the ldc data. However, an amr.pre.sp model was created and not all the files were present in the logs folder that were present when I trained a much smaller model. Do you know why this is? Is it cause of the use of split up data? Would putting all the data into one file fix this? My exp and inc files seems exactly the same across experiments, excpet when I trained the smaller model and got an amr.sp file, the data wasn't split up.

[yoavartzi]

If you use separate files, they have to be merged. I think the exp files we released do that. Once you merge the data resource, it should be used a single resource, so I am not sure why it will behave differently.