- Introduction
- Main Features
- Setup
- Setup Using Docker
- Usage
- Models and Scenarios
- Important Notes
- Training Your Own Model
- Morpheme and Word Embeddings
- Evaluation
- Ben-Mordecai Corpus
- Citations
Code and models for neural modeling of Hebrew NER. Described in the TACL paper "Neural Modeling for Named Entities and Morphology (NEMO2)" along with extensive experiments on the different modeling scenarios provided in this repository.
- Trained on the Hebrew NER and Morphology NEMO corpus of gold annotated Modern Hebrew news articles.
- Multiple modeling options to go from raw Hebrew text to morpheme and/or token-level NER boundaries.
- Neural model implementation of NCRF++
- bclm is used for reading and transforming morpho-syntactic information layers.
- Clone this NEMO repo:
git clone https://github.com/OnlpLab/NEMO.git - Enter the repo directory:
cd NEMO - Preferably in a virtual env:
pip install -r requirements.txt - Unpack model files:
gunzip data/*.gz - Install
yap: https://github.com/OnlpLab/yap
- In YAP folder, run YAP API server
./yap api - In NEMO folder, run NEMO API server
uvicorn api_main:app --port 8090
- Change
YAP_PATHinconfig.pyto the path of your localyapexecutable.
docker-compose up(pulls, builds and/or startup will take a few minutes, depending on your bandwidth)- That's it. You now have NEMO API running and available at local port
8090.- YAP API docker is also running in the background, you can make it available by uncommenting the last two lines of
docker-compose.yml.
- YAP API docker is also running in the background, you can make it available by uncommenting the last two lines of
- Once the API server is up, check out the API documentation by opening (http://localhost:8090/docs) in your browser.
- You can find the available API endpoints and more usage examples in api_usage.ipynb.
- All you need to do is run
nemo.pywith a specific command (scenario), on a text file of Hebrew sentences separated by a line-break. - You can run a neural NER model directly, or choose a full end-to-end scenario that includes morphological segmentation and alignments (described fully in the next section). e.g.:
- the
run_ner_modelcommand with thetoken-singlemodel will tokenize sentences and run thetoken-singlemodel:python nemo.py run_ner_model token-single example.txt example_output.txt
- the
morph_hybridcommand runs the end-to-end segmentation and NER pipeline which provided our best performing morpheme-level NER boundaries:python nemo.py morph_yap morph example.txt example_output_MORPH.txt
- the
- You can find outputs of different commands on the input in example.txt in: morph_hybrid_align_tokens, morph_hybrid, morph_yap, multi_align_hybrid, single
- For a full list of the available commands please consult the next section and the inline documentation at the end of
nemo.py.
Models are all standard Bi-LSTM-CRF with char encoding (LSTM/CNN) of NCRFpp with pre-trained fastText embeddings. Differences between models lay in:
- Input units: morphemes
morphvs. tokenstoken-* - Output label set:
token-singlesingle sequence labels (e.g.B-ORG) vs.token-multimulti-labels (atomic labels, e.g.O-ORG^B-ORG^I-ORG) that predict, in order, the labels for the morphemes the token is made of.
| Token-based Models | Morpheme-based Model |
|---|---|
 |
 |
Morphemes must be predicted. This is done by performing morphological disambiguation (MD). We offer two options to do so:
- Standard pipeline: MD using YAP. This is used in the
morph_yapcommand, which runs ourmorphNER model on the output of YAP joint segmentation. - Hybrid pipeline: MD using our best performing Hybrid approach, which uses the output of the
token-multimodel to reduce the MD option space. This is used inmorph_hybrid,multi_align_hybridandmorph_hybrid_align_tokens. We will explain these scenarios next.
| MD Approach | Commands |
|---|---|
Standard  |
morph_yap |
Hybrid   |
morph_hybrid,multi_align_hybrid,morph_hybrid_align_tokens |
Finally, to get our desired output (tokens/morphemes), we can choose between different scenarios, some involving extra post-processing alignments:
- To get morpheme-level labels we have two options:
- Run our
morphNER model on predicted morphemes: Commands:morph_yapormorph_hybrid(better). token-multilabels can be aligned with predicted morphemes to get morpheme-level boundaries. Command:multi_align_hybrid.
- Run our
Run morph NER on Predicted Morphemes |
Multi Predictions Aligned with Predicted Morpheme |
|---|---|
 |
 |
morph_yap,morph_hybrid |
multi_align_hybrid |
- To get token-level labels we have three options:
run_ner_modelcommand withtoken-singlemodel.- the predicted labels of the
token-multican be mapped totoken-singlelabels to get standard token-single output. The commandmulti_to_singledoes this end-to-end. - Morpheme-level output can be aligned back to token-level boundaries. Command:
morph_hybrid_align_tokens(this achieved best token-level results in our experiments).
Run token-single |
Map token-multi to token-single |
Align morph NER with Tokens |
|---|---|---|
 |
 |
 |
run_ner_model token-single |
multi_to_single |
morph_hybrid_align_tokens |
- Note: while the
morph_hybrid*scenarios offer the best performance, they are slightly less efficient since they requires running bothmorphandtoken-multiNER models (yap calls take up most of the runtime anyway, so this is not extremely significant).
- NCRFpp was great for our experiments on the NEMO corpus (which is given constant data), but it holds some caveats for real life scenarios of arbitrary text:
- fastText is not used on the fly to obtain vectors for OOV words (i.e. those that were not seen in our Wikipedia corpus). Instead, it is used as a regular embedding matrix. Hence the full generalization capacities of fastText, as shown in our experiments, are not available in the currently provided models, which will perform slightly worse than they could on arbitrary text. In our experiments we created such a matrix in advance with all the words in the NEMO corpus and used it during training. Information regarding training your own model with your own vocabulary in the next section.
- If you do wish to replicate our reported results on the Hebrew treebank, download the
*oov*models from here and extract to thedata/folder (they already appear inconfig.py).
- In the near future we plan to publish a cleaner end-to-end implementation, including use of our new AlephBERT pre-trained Transformer models.
- For archiving and reproducibility purposes, our original code used for experiments and analysis can be found in the following repos: https://github.com/cjer/NCRFpp, https://github.com/cjer/NER (beware - 2 years of Jupyter notebooks).
We provide template NCRF++ config files. These files already contain the hyperparameters we used in our training. To train your own model:
- Copy the config for the variant (token-multi, token-single, morph) you wish to use from the ncrf_train_configs folder.
- Change the parameter
word_emb_dirto that of an embedding vectors file in standard word2vec textual format. You can use the fastText bin models we make available (in the next section) or any other embedding vectors of your choice. - Run the following in your shell:
python ncrf_main.py --config <path_to_config> --device <gpu_device_number>- For more information, please consult NCRF++ documentation.
- To evaluate your trained models, please consult the evaluation section.
The word embeddings we trained and used in our models are available:
- Space-delimited tokens (traditional word embeddings): fastText (bin, text), GloVe, word2vec
- Morphemes: fastText (bin, text), GloVe, word2vec
These were trained on a 2013 Wiki dump corpus by Yoav Goldberg, which we re-tokenized and then re-parsed using YAP:
- Space-delimited tokens
- Morphemes, automatic YAP segmentation (using the morpheme FORM as the unit for embedding)
- CONLL files of full morpho-syntactic output of YAP
To evaluate your predictions against gold use the ne_evaluate_mentions.py script. Evaluation looks for exact match of string and entity category, but is slightly different than the standard CoNLL2003 evaluation commonly used for NER. The reason is that predicted segmentation differs from gold, so positional indexes of sequence labels cannot be used. What we do instead, is extract multi-sets of entity mentions and use set operations to compute precision, recall and F1-score. You can find more detailed discussion of evaluation in the NEMO2 paper.
To evaluate an output prediction file against a gold file use:
python ne_evaluate_mentions.py <path_to_gold_ner> <path_to_predicted_ner>If you're within python, just call ne_evaluate_mentions.evaluate_files(...) with the same parameters.
In our NEMO2 paper we also evaluate our models on the Ben-Mordecai Hebrew NER Corpus (BMC). The 3 random splits we used can be found here.
If you use any of the NEMO2 code, models, embeddings or the NEMO corpus, please cite the NEMO2 paper:
@article{10.1162/tacl_a_00404,
author = {Bareket, Dan and Tsarfaty, Reut},
title = "{Neural Modeling for Named Entities and Morphology (NEMO2)}",
journal = {Transactions of the Association for Computational Linguistics},
volume = {9},
pages = {909-928},
year = {2021},
month = {09},
abstract = "{Named Entity Recognition (NER) is a fundamental NLP task, commonly formulated as classification over a sequence of tokens. Morphologically rich languages (MRLs) pose a challenge to this basic formulation, as the boundaries of named entities do not necessarily coincide with token boundaries, rather, they respect morphological boundaries. To address NER in MRLs we then need to answer two fundamental questions, namely, what are the basic units to be labeled, and how can these units be detected and classified in realistic settings (i.e., where no gold morphology is available). We empirically investigate these questions on a novel NER benchmark, with parallel token- level and morpheme-level NER annotations, which we develop for Modern Hebrew, a morphologically rich-and-ambiguous language. Our results show that explicitly modeling morphological boundaries leads to improved NER performance, and that a novel hybrid architecture, in which NER precedes and prunes morphological decomposition, greatly outperforms the standard pipeline, where morphological decomposition strictly precedes NER, setting a new performance bar for both Hebrew NER and Hebrew morphological decomposition tasks.}",
issn = {2307-387X},
doi = {10.1162/tacl_a_00404},
url = {https://doi.org/10.1162/tacl\_a\_00404},
eprint = {https://direct.mit.edu/tacl/article-pdf/doi/10.1162/tacl\_a\_00404/1962472/tacl\_a\_00404.pdf},
}If you use the NEMO2's NER models please also cite NCRF++:
@inproceedings{yang2018ncrf,
title={{NCRF}++: An Open-source Neural Sequence Labeling Toolkit},
author={Yang, Jie and Zhang, Yue},
booktitle={Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics},
Url = {http://aclweb.org/anthology/P18-4013},
year={2018}
}
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent