anotherdayattheraces / level-4-project

Entity Retrieval for Complex Answer Retrieval

level-4-project

Entity Retrieval for Complex Answer Retrieval

This system is intended to be an entity retrieval engine.

To use the system you will need:

JDK (Java development kit) get it here: http://www.oracle.com/technetwork/java/javase/downloads/jdk9-downloads-3848520.html Apache maven get it here: https://maven.apache.org/download.cgi

Once you have installed these and cloned this repository you are going to need to build both Galago and the entity linker individually, do this by typing mvn install -DskipTests on a terminal in each respective directory. (Do this for galago first).

Now you need the data - first download and unzip the pubmed article files from here (labelled pmc.blah.tar.gz: http://www.trec-cds.org/2016.html These files are very large in size so you may not be able to use all of them, but the more files you have the more complete your system will be. Now you need the files from Dbpedia, get them from here: http://wiki.dbpedia.org/downloads-2016-10 download the relevant .ttl files from under the "datasets" heading.

level-4-project's Issues

Enhance dictionary with more medical entities

At the moment the dictionary is only initialised with diseases from the icd9/10 databases, It needs to be enhanced to include as many medical entities as possible.

Web server for user interface

The current system uses java user input to get the user's query, this should be changed to add a web server to allow the user to enter their query there and receive results on the web server.

Create database for KB

At the moment the system is only able to show medical-related entities, however this limits the system greatly, as many searches may involve many non-medical entities.

Recognises only 3 word entities

Some entities are > 3 words, system should be changed to identify these

Speed up dictionary

At the moment the dictionary is inefficient and slow to search, it needs to be sped up, sorting it alphabetically would be a good place to start.

Formatting of qrels

The qrels files are formatted strangely with some characters being encoded with % this could be fixed so all correct entities are read in.

Pre-load dictionary

At the moment, when running the system you must wait 5 minutes to generate the dictionary, this should be changed so the dictionary is pre-loaded before run-time.

Change galago index

At the moment the documents in the index are badly formed and have poor use of tokens/fields, this must be changed so data is structured into a readable and helpful format.