pepperkit / corenlp-stop-words-annotator

Stop words annotator for Stanford's CoreNLP library.

CoreNLP Stop Words Annotator

Annotator for CoreNLP library, allows adding the set of rules or/and the word themselves, which should be filtered out in the CoreNLP pipeline processing.

Usage

Just add the annotator and CoreNLP library with models into the dependencies list like this:

        <dependency>
            <groupId>io.github.pepperkit</groupId>
            <artifactId>corenlp-stop-words-annotator</artifactId>
            <version>1.0.0</version>
        </dependency>

        <dependency>
            <groupId>edu.stanford.nlp</groupId>
            <artifactId>stanford-corenlp</artifactId>
            <version>4.2.2</version>
        </dependency>
        <dependency>
            <groupId>edu.stanford.nlp</groupId>
            <artifactId>stanford-corenlp</artifactId>
            <version>4.2.2</version>
            <classifier>models</classifier>
        </dependency>

The annotator is configured with Properties, it marks the words as stopped using one of the following rules:

• provided list of particular words (and/or its lemmas) using a string containing comma-separated words, or a file with newline-separated words (from any place in the file system or from a bundled resource) - stopwords.customList, stopwords.customListFilePath, and stopwords.customListResourcesFilePath properties (if all of the properties are provided, only one list of words will be initialized from a provided property, the order of precedence: string with words, from a file, from a bundled resource);
• POS (part-of-speech) categories (of words lemmas) as a string containing a comma-separated list of the categories - stopwords.withPosCategories property;
• the length of a word or its lemma - stopwords.shorterThan and stopwords.withLemmasShorterThan properties.

Description of the available POS categories can be found here (also see complex example below):

• https://nlp.stanford.edu/software/pos-tagger-faq.html
• https://catalog.ldc.upenn.edu/docs/LDC99T42/tagguid1.pdf

Requirements

• Java version should be 8 or higher;
• annotator should be added at the project's POM as a dependency;
• CoreNLP library should be present in the classpath;
• tokenize, ssplit, pos, and lemma annotators should be present in the pipeline before stopwords annotator.

Simple Example

If we just want to filter out the words from a list of stop words, we can easily do it like following:

class Example {
    public Set<String> getInterestingWords() {
        final String text = "Once upon a time there was a dear little girl who was loved by everyone who looked at her";
        
        final Properties props;
        props = new Properties();
        props.put("annotators", "tokenize, ssplit, pos, lemma, stopwords");
        props.setProperty("customAnnotatorClass.stopwords", "io.github.pepperkit.corenlp.stopwords.StopWordsAnnotator");
        props.setProperty("ssplit.isOneSentence", "true");
        
        // Filter out these words
        props.setProperty("stopwords.customList", "once,upon,a,little,girl");

        StanfordCoreNLP pipeline = new StanfordCoreNLP(props);
        Annotation document = new Annotation(text);
        pipeline.annotate(document);

        Set<String> result = new HashSet<>();
        List<CoreLabel> tokens = document.get(CoreAnnotations.TokensAnnotation.class);

        for (CoreLabel token : tokens) {
            // token.get(StopWordsAnnotator.class) will be TRUE if the word is stopped
            if (!token.get(StopWordsAnnotator.class)) {
                result.add(token.get(CoreAnnotations.LemmaAnnotation.class));
            }
        }
        return result;
    }
}

Complex Example

Let's use stopwords annotator for a particular complex scenario when we need to process a text and extract a set of lemmas of only "interesting" words, where the word is considered "interesting", if it is not a common word (more detailed definition is further in the text).

Scenario:

Given I have the text
  And the stop words are defined in the resources file (containing the most common English words)
When I launch text processing using StanfordCoreNLP pipeline with StopWordsAnnotator
  And set it to mark words as stopped if it is shorter than 3 letters (to remove all the punctuation and simple words like be, so, etc.)
  And is of POS category I am not interested in
  And is in the list of stop words I provided in the resources file
Then I should be able to filter out the common words from the text

class Example {
    public Set<String> getInterestingWords() {
        // I have the text:
        final String text = "Once upon a time there was a dear little girl who was loved by everyone who looked at " +
                "her, but most of all by her grandmother, and there was nothing that she would not have given to the " +
                "child. Once she gave her a little riding hood of red velvet, which suited her so well that she would" +
                " never wear anything else; so she was always called 'Little Red Riding Hood.'";

        // I want to get the list of lemmas created from the text, excluding words from the provided list and all the
        // common or simple words (like propositions, conjunctions, etc.), since I want to extract only the words
        // I could be interested to learn
        String[] expectedWords = {"dear", "look", "have", "give", "riding", "hood", "velvet", "suit", "wear", "call"};

        // And the stop words in resources (containing the most known English words)
        final String stopWordsResourcePath = "common-words-list-it.txt";

        final Properties props;
        props = new Properties();
        props.put("annotators", "tokenize, ssplit, pos, lemma, stopwords");
        props.setProperty("customAnnotatorClass.stopwords", "io.github.pepperkit.corenlp.stopwords.StopWordsAnnotator");
        props.setProperty("ssplit.isOneSentence", "true");

        // to filter out all the punctuation and simple words like be, so, etc.
        props.setProperty("stopwords.withLemmasShorterThan", "3");

        // to filter out all the common and simple words
        // Description of the available POS categories can be found here:
        // - https://nlp.stanford.edu/software/pos-tagger-faq.html
        // - https://catalog.ldc.upenn.edu/docs/LDC99T42/tagguid1.pdf
        props.setProperty("stopwords.withPosCategories",
                "NNP,NNPS," + // proper noun singular and plural
                        "PDT," + // predeterminer
                        "IN,CC," + // conjunction and coordinating conjunction (but, and etc.)
                        "DT," + // determiner - the, a, etc.
                        "UH," + // interjection - my, his, oh, uh etc.
                        "FW," + // foreign word
                        "MD," + // modal verb
                        "RP," + // particle
                        "PRP,PRP$," + // personal pronoun
                        "EX," + // existential there
                        "POS," + // possessive ending: 's
                        "SYM," + // symbol
                        "WDT,WP,WP$," + // wh-determiner (who), wh-pronoun (who, what, whom) and possessive wh-pronoun (whose)
                        "WRB" // wh-adverb
        );

        // provide the file with stop words list
        props.setProperty("stopwords.customListResourcesFilePath", stopWordsResourcePath);

        // Annotate the text using StanfordCoreNLP pipeline
        StanfordCoreNLP pipeline = new StanfordCoreNLP(props);
        Annotation document = new Annotation(text);
        pipeline.annotate(document);

        // Process returned tokens
        Set<String> result = new HashSet<>();
        List<CoreLabel> tokens = document.get(CoreAnnotations.TokensAnnotation.class);

        // Return only lemmas of only interesting words
        for (CoreLabel token : tokens) {
            if (!token.get(StopWordsAnnotator.class)) {
                result.add(token.get(CoreAnnotations.LemmaAnnotation.class));
            }
        }
        return result;
    }
}

Project's structure

└── src
    ├── main                # code of the annotator
    ├── test                # unit tests
    └── integration-test    # integration tests

corenlp-stop-words-annotator's Issues

Add simple example

Add default sets for stop words commonly used in NLP

Update Checkstyle version

corenlp-stop-words-annotator/pom.xml

Line 159 in 3b0de99

<version>8.39</version>

Publish to maven

Add info on how to add the dependency

Add custom filter to use (via inheritance)

Add license to source code

Release version 1.0.0

Add documentation

Add compatibility badge (with CoreNLP version)