opensemanticsearch / solr-ontology-tagger Goto Github PK

Disambiguate labels that exists more than one time by machine learning / classifiying with model from manual tagged documents and/or opendata like wikipedia page(s).

Since not only for tagging but writing synonyms and named entities dictionaries, use cleaner architecture and other function names and move functionality from Django app to this module.

For better performance dedupe labels for each entity, since many multilingual ontologies have multiple same labels (for different languages).

More performant ETL/dataenrichment plugin for normalizing / entity linking of occuring labels.

Disable-Option of enrichment with synonyms since new Solr release support multi word synonyms so we can use Solrs synonym management & synonym search features, instead of embedding/writing them to documents.

It should be read from env variable or config file.

Hi,

I've added synonyms for the name "TREPROSTINIL": "Orenitram", "Remodulin" and "TYVASO" and indexed documents with word "Treprostinil" or "treprostinil".

When I do a search in the web UI for "remodulin", it find the document with "treprostinil" but not the one with "Treprostinil".

Is that expected? How can I make it not case-sensitive? I tried adding lower cased entries to "/solr/opensemanticsearch/schema/analysis/synonyms/skos" but it did not work.

This is what I have on this schema (they are duplicated so that symmetric synonym search can work, otherwise the pref_label finds document with aliases but searching for aliases do not find pref_label nor other aliases):
"Orenitram":["Orenitram",
"Remodulin",
"TREPROSTINIL",
"TYVASO"],
"Remodulin":["Orenitram",
"Remodulin",
"TREPROSTINIL",
"TYVASO"],
"TREPROSTINIL":["TREPROSTINIL",
"orenitram",
"remodulin",
"tyvaso"],
"TYVASO":["Orenitram",
"Remodulin",
"TREPROSTINIL",
"TYVASO"],

Thanks!
Yoann

Hi,

When we upload an RDF ontology (say ontology.xml) to OSS, we have the option of applying it to existing documents. This isn't currently working as expected because in solr_ontology_tagger.py the fields that are updated are only test_xml_ss_preferred_label_ss and test_xml_ss_uri_ss. The field test_xml_ss (which gets turned to a facet) is not updated. To fix this, I've changed

tagdata = add_value_to_facet(facet = target_facet + '_preferred_label_ss', value = preferred_label, data=tagdata)

to

tagdata[target_facet] = preferred_label
tagdata = add_value_to_facet(facet = target_facet + '_preferred_label_ss', value = preferred_label, data=tagdata)

Is it a reasonable fix?

Thanks!
Bassam

PS: As I'm sure you know, there is a difference in how entity_linker tags documents and how solr_ontology_tagger tags documents. (I think) Entity_linker only tags a document if the facet search returns exact matches to a label:

if str(value).lower() == queries[query]['query'].lower():
              match = True

While playing around with OSS, this seems to cause problems when for example a dictionary term is 'hello '. 'hello' will be returned in a facet search but entity_linker won't tag the document because it's not the same as 'hello ' (with the space at the end).

Analyse and export taxonomy structure from external thesauri/ontologies to Open Semantic Named Entities Search API, so can be used by ETL and UI.

Add SKOS:hiddenLabel (for management of misspellings) to query.

Especially if import multi language ontology, there are often same labels for each language, so dedupe them.

Hi,

When I try to apply an ontology (through “Manage structure” -> “Ontologies (Lists of names or concepts, Vocabularies, Dictionaries, Thesauri)” tab -> “Details” of an ontology -> “Apply ontology”), I get the error

No connection adapters were found for 'localhost:8983/solr/…

The error can be traced back to the solr_ontology_tagger.py file. The solr url is set to be

solr = 'localhost:8983/solr/'

instead of

solr = 'http://localhost:8983/solr/'

I no longer get an error. I get the success message "Ontology applied." but none of the indexed documents have tags applied to them.
It seems like the only way to tag documents with entities from an ontology is to re-index them with the enhance_entity_linking plugin.

Best regards,
Bassam

Separate the ontology tagging code for Solr from Django app ontologies manager for more modular architecture and additional general usable command line tools / libraries

Find/highlight not only entity name and aliases but find entity without alias if search for alias, too

For performance issues with bigger thesauri, the synonyms config has to be written for the ontology/skos thesaurus at once, not for each concept.