spgbarrett / gearlab_topobert

Source code for geoparser TopoBERT

Introduction

TopoBERT, a toponym recognition module based on a one-dimensional Convolutional Neural Network (CNN) and Bidirectional Encoder Representation from Transformers (BERT).

The struture of the model is shown in the figure below:

Performance

The model is trained using CoNLL-2003 and evaluated with Harvey2017.

Evaluated with Harvey2017 comparing with other popular models:

Model	Precision	Recall	F1-score
Stanford NER (broad location)	0.729	0.440	0.548
spaCy NER (board location)	0.461	0.304	0.366
BiLSTM-CRF	0.703	0.600	0.649
DM_NLP	0.729	0.680	0.703
NeuroTPR	0.787	0.678	0.728
TopoBERT	0.898	0.835	0.865

Requirements

geojson 2.5.0
matplotlib 3.4.3
nltk 3.6.5
numpy 1.21.2
pandas 1.3.3
regex 2021.9.30
scikit-learn 1.0
scipy 1.7.1
seaborn 0.11.2
seqeval 1.2.2
tokenizers 0.10.3
torch 1.9.1+cu102
torchvision 0.10.1+cu102
tqdm 4.62.3
transformers 4.11.2

How to deploy

1. Clone the source codes and place in a path that your project can access.
2. Download the pretrained models, unzip the file and place the folder with its original name in the pretrained_models folder.
3. Download the required dependencies
4. Ready to use.

Use case

from topo_bert import *  # Refer to the path you put your downloaded files

test_text = """HarveyStorm over Austin TX at 8: 00 AM CDT via Weather Underground"""
current_geoparser = TopoBERT()
result = current_geoparser.predict(test_text)
print(result)

The demo output results:

{
	'combined_addresses': ['Austin', 'TX'],
	'address_result': ['Austin', 'TX'],
    'full_address': 'Austin TX',
	'org_result': [{
		'word': 'HarveyStorm',
		'tag': 'B-ORG',
		'confidence': 0.9983394145965576
	}, {
		'word': 'over',
		'tag': 'O',
		'confidence': 0.9998631477355957
	}, {
		'word': 'Austin',
		'tag': 'B-LOC',
		'confidence': 0.9995130300521851
	}, {
		'word': 'TX',
		'tag': 'B-LOC',
		'confidence': 0.9928538203239441
	}, {
		'word': 'at',
		'tag': 'O',
		'confidence': 0.9999804496765137
	}, {
		'word': '8',
		'tag': 'O',
		'confidence': 0.9999505281448364
	}, {
		'word': ':',
		'tag': 'O',
		'confidence': 0.9999704360961914
	}, {
		'word': '00',
		'tag': 'O',
		'confidence': 0.99994957447052
	}, {
		'word': 'AM',
		'tag': 'O',
		'confidence': 0.9463351368904114
	}, {
		'word': 'CDT',
		'tag': 'B-MISC',
		'confidence': 0.5280879735946655
	}, {
		'word': 'via',
		'tag': 'O',
		'confidence': 0.9999630451202393
	}, {
		'word': 'Weather',
		'tag': 'B-ORG',
		'confidence': 0.9993113279342651
	}, {
		'word': 'Underground',
		'tag': 'I-ORG',
		'confidence': 0.9984622001647949
	}]
}

How to train your own model

You can train your own model with the code below:

model_args_used = {
       "--cuda": "use GPU",
       "--pretrained_model": "bert-large-cased",
       "--num_of_labels": 12,
       "--model_hidden_layer_size": 1024,
       "--no_hidden_layers": 24,
       "--dropout": 0.1,
       "--out-channel": 16,
       "--freeze-bert": False,
       "--verbose": "whether to output the test results"
   }

exp_train_config = {
           "--task_name": "bert_geoparsing",
           "--toponym_only": False,
           "--random_seed": 42,
           "--use_gpu": 1,
           "--train_data_type": "conll",
           "--validate_data_type": "conll",
           "--test_data_type": "conll",
           "--train_data_dir": "Put your own file absolute path here",
           "--validate_data_dir": "Put your own file absolute path here",
           "--test_data_dir": "Put your own file absolute path here",
           "--train_data_file": "train.txt",
           "--validate_data_file": "test.txt",
           "--test_data_file": "test.txt",
           "--is_validate": 1,
           "--is_test": 1,
           "--output_dir": "./outputs",
           "--cache_dir": "./cache",
           "--bert_model": "bert-large-cased",
           "--do_lower_case": False,
           "--max_seq_length": 128,
           "--training_epoch": 50,
           "--train_batch_size": 32,
           "--test_batch_size": 32,
           "--learning_rate": 5e-5,
           "--warm_up_proportion": 0.1,
           "--weight_decay": 0.01,
           "--adam_epsilon": 1e-8,
           "--max_grad_norm": 1.0,
           "--num_grad_accum_steps": 1,
           "--loss_scale": 0
       }

model = BertCNN1DNer(model_config=model_args_used)
current_trainer = TopoBertModelTrainer(model, train_config = exp_train_config)
current_trainer.train()

gearlab_topobert's Issues

About Replicating the Paper Results

Hi Bing,
Firstly, wonderful work! I really appreciate the effort to make your code open as well.

I've been trying to replicate your paper results (TopoBERT). While I can reproduce the numbers in Table 2 for testing on CoNLL, it seems the optimal model trained on CoNLL does not yield optimal results for Harvey in Table 3 (best case F-1 ~0.72 for LOC). I just wonder if you could let you the set of "model arguments" and "training configs" to reproduce the optimal results on Harvey? Thank you very much!

Specify python version

Hi Bing,

Thanks for this - this is really great work! To improve reproducability it would be useful to put which Python version you are using in the README file. It's taken me a while to set up a functioning environment since its not clear which Python version you used for the original project. I've had success replecating the results using Python 3.8, but some of my packages are different versions to those suggested in the README.

Again, this is a really nice project and it seems to work well once the dependencies are ironed out. Looking forward to using it in the future!

Joe