johnmm / jatetoolkit Goto Github PK

I am new can you provide me some basic steps to build this

What steps will reproduce the problem?

java -Xmx1024m -classpath 
/Users/sarnobat/trash/jatetoolkit-read-only/dist/:/Users/sarnobat/trash/jatetool
kit-read-only/libs/apache-log4j-1.2.15/log4j-1.2.15.jar:/Users/sarnobat/trash/ja
tetoolkit-read-only/libs/apache-opennlp-1.51/jwnl-1.3.3.jar:/Users/sarnobat/tras
h/jatetoolkit-read-only/libs/apache-opennlp-1.51/opennlp-maxent-3.0.1-incubating
.jar:/Users/sarnobat/trash/jatetoolkit-read-only/libs/apache-opennlp-1.51/opennl
p-tools-1.5.1-incubating.jar:/Users/sarnobat/trash/jatetoolkit-read-only/libs/dr
agon/dragontool.jar:/Users/sarnobat/trash/jatetoolkit-read-only/libs/hsqldb2.2.3
/hsqldb.jar:/Users/sarnobat/trash/jatetoolkit-read-only/libs/hsqldb2.2.3/sqltool
.jar:/Users/sarnobat/trash/jatetoolkit-read-only/libs/wit-commons/wit-commons.ja
r: uk.ac.shef.dcs.oak.jate.test.AlgorithmTester 
/Users/sarnobat/trash/jatetoolkit-read-only/nlp_resources/  test/example/ 
test/output

What is the expected output? What do you see instead?
don't know

What version of the product are you using? On what operating system?
trunk

Please provide any additional information below.


Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
        at java.util.Arrays.copyOf(Arrays.java:2734)
        at java.util.ArrayList.ensureCapacity(ArrayList.java:167)
        at java.util.ArrayList.add(ArrayList.java:351)
        at uk.ac.shef.dcs.oak.jate.core.npextractor.NGramExtractor.getNGram(NGramExtractor.java:123)
        at uk.ac.shef.dcs.oak.jate.core.npextractor.NGramExtractor.extract(NGramExtractor.java:67)
        at uk.ac.shef.dcs.oak.jate.core.npextractor.NGramExtractor.extract(NGramExtractor.java:49)
        at uk.ac.shef.dcs.oak.jate.core.feature.indexer.GlobalIndexBuilderMem.build(GlobalIndexBuilderMem.java:53)
        at uk.ac.shef.dcs.oak.jate.test.AlgorithmTester.main(AlgorithmTester.java:83)

What steps will reproduce the problem?

java -Xmx1024m -classpath 
/home/minev/jate/dist/jate.jar:/home/minev/jate/libs/apache-log4j-1.2.15/log4j-1
.2.15.jar:/home/minev/jate/libs/apache-opennlp-1.51/jwnl-1.3.3.jar:/home/minev/j
ate/libs/apache-opennlp-1.51/opennlp-maxent-3.0.1-incubating.jar:/home/minev/jat
e/libs/apache-opennlp-1.51/opennlp-tools-1.5.1-incubating.jar:/home/minev/jate/l
ibs/dragon/dragontool.jar:/home/minev/jate/libs/hsqldb2.2.3/hsqldb.jar:/home/min
ev/jate/libs/hsqldb2.2.3/sqltool.jar:/home/minev/jate/libs/wit-commons/wit-commo
ns.jar: uk.ac.shef.dcs.oak.jate.test.TestFrequency 
/home/minev/jate/nlp_resources/ test/example test/output

What is the expected output? What do you see instead?

Exception in thread "main" java.io.FileNotFoundException: 
D:/work/JATR_SDK/jate_googlecode/nlp_resources/stoplist.txt (No such file or 
directory)
    at java.io.FileInputStream.open(Native Method)
    at java.io.FileInputStream.<init>(FileInputStream.java:120)
    at java.io.FileReader.<init>(FileReader.java:55)
    at uk.ac.shef.dcs.oak.jate.util.control.StopList.loadStopList(StopList.java:44)
    at uk.ac.shef.dcs.oak.jate.util.control.StopList.<init>(StopList.java:30)
    at uk.ac.shef.dcs.oak.jate.test.TestFrequency.main(TestFrequency.java:70)


What version of the product are you using? On what operating system?

Product version: 1.1
OS: Mac OS 10.6 and Ubuntu Linux 11.04

Please provide any additional information below.

I've updated the config file jate_1.1.properties (once in root dir and in /src).

What steps will reproduce the problem?
1. in Algorithm Tester activate the option to write persistent HSQL database
2. check the database script
3.

What is the expected output? What do you see instead?
term2doc table only has a document for each term
doc2term table only has a term for each document
term2var only has a var for each term

What version of the product are you using? On what operating system?
1.1 / ubuntu 12.04

Please provide any additional information below.

In the GlobalIndexWriterHSQL.persist class there is a "=" instead a "+=" in the 
inner for loops of the code blocks: 
  //persist term2DocMap,  //persist doc2TermMap and //persist termId2VarIds map

i.e.

for (Map.Entry<Integer, Set<Integer>> e : index.getTerm2Docs().entrySet()) {
                pstmt.setInt(1, e.getKey());
                String docIdStr = "";
                for(Integer i: e.getValue()){
                    //docIdStr=HSQLDBUtil.VALUE_SEPARATOR+i;
                    docIdStr+=HSQLDBUtil.VALUE_SEPARATOR+i;
                }

What steps will reproduce the problem?
1. TestFrequency counter for verb phrases
2. I've created a class analogues to the NounPhraseExtractorOpenNLP and updated 
the strings to B-VP and I-VP
3. The algorithm counts/extracts further the noun phrases

What is the expected output? What do you see instead?
Verb Phrase + counter

What version of the product are you using? On what operating system?
1.11

Please provide any additional information below.

What properties must be further altered so that only verb phrases are counted? 
The OpenNLP parser supports this type of annotation.

When I ran this on 6000 small documents it ran for 6+ hours without completing.

With the attached performance fixes that same test runs in just a few minutes.

Hi,

could you provide some more information about the way you compiled the BNC 
reference list? Is there any reference to the source if its external?

mihail

This is a question/feature request. In my testing, it seems like the main 
bottleneck is the building of the GlobalIndex, as opposed to using the 
FeatureBuilder classes for doing the counts. However, while there is a 
multithread version of the FeatureBuilder classes, there is none for the 
GlobalIndex builders. Are there plans to implement parallel versions of these 
builders? I am not very experienced with Java, but I might try to implement 
them if it is feasible to do so.


Matt