:smile: Emoji synonyms to build your own emoji-capable search engine (elasticsearch, solr, OpenSearch)
Home Page: https://jolicode.com/blog/elasticsearch-icu-now-understands-emoji
License: MIT License
CLDR is ready with new annotation for the new emoji.
Reading https://www.elastic.co/blog/boosting-the-power-of-elasticsearch-with-synonyms - we quickly see Emoji Search can benefit from the new POST /synonym_test/_reload_search_analyzers API.
Index-time synonyms have several disadvantages:
- The index might get bigger, because all synonyms must be indexed.
- Search scoring, which relies on term statistics, might suffer because synonyms are also counted, and the statistics for less common words become skewed.
- Synonym rules can’t be changed for existing documents without reindexing.
...
Using synonyms in search-time analyzers on the other hand doesn’t have many of the above mentioned problems:
- The index size is unaffected.
- The term statistics in the corpus stay the same.
- Changes in the synonym rules don’t require reindexing of documents.
And:
Starting with Elasticsearch 7.3, this reopening of indices in order to see changes in synonym files is no longer needed.
We must:
I think you should use a package namespace other than elasticsearch since you are creating a community plugin.
Maybe package org.jolicode.[...].
See https://docs.oracle.com/javase/tutorial/java/package/namingpkgs.html
The plugin should provide a new token filter, "emoji_synonym_filter", which would simplify the configuration from this:
PUT /en-emoji
{
"settings": {
"analysis": {
"filter": {
"english_emoji": {
"type": "synonym",
"synonyms_path": "analysis/cldr-emoji-annotation-synonyms-en.txt"
}
},
"analyzer": {
"english_with_emoji": {
"tokenizer": "emoji_tokenizer",
"filter": [
"lowercase",
"english_emoji"
]
}
}
}
}
}To this:
{
"settings": {
"analysis": {
"filter": {
"english_emoji": {
"type": "emoji_synonym_filter",
"language": "en"
}
},
"analyzer": {
"english_with_emoji": {
"tokenizer": "emoji_tokenizer",
"filter": [
"lowercase",
"english_emoji"
]
}
}
}
}
}One of the main feature is that the synonym files could be shipped directly with the bundle!
We should be able to create a file instead of putting them all in the mapping.
Hi @damienalexandre,
We're having the same issue you describe here, when using the dictionary file from this repository.
We've just upgraded from ElasticSearch 5.3 to 7.8 however, so we can't use your plugin to solve this issue (yet).
Is a 7.8 release on your roadmap by any chance?
Thanks in advance!
Hi @damienalexandre ... Thank you for continuing to collate and update the synonyms files ...
Related to #26, I've used your example mappings with one slight change (placing the synonyms file in /etc/elasticsearch/synonyms instead) as follows:
analysis-icu plugin and restarted Elasticsearchsudo cp synonyms/cldr-emoji-annotation-synonyms-en.txt /etc/elasticsearch/synonyms/Then ...
PUT /emoji-capable
{
"settings": {
"analysis": {
"filter": {
"english_emoji": {
"type": "synonym",
"synonyms_path": "synonyms/cldr-emoji-annotation-synonyms-en.txt"
}
},
"analyzer": {
"english_with_emoji": {
"tokenizer": "icu_tokenizer",
"filter": [
"english_emoji"
]
}
}
}
}
}
which gives me ...
{
"error" : {
"root_cause" : [
{
"type" : "illegal_argument_exception",
"reason" : "failed to build synonyms"
}
],
"type" : "illegal_argument_exception",
"reason" : "failed to build synonyms",
"caused_by" : {
"type" : "parse_exception",
"reason" : "Invalid synonym rule at line 859",
"caused_by" : {
"type" : "illegal_argument_exception",
"reason" : "term: * was completely eliminated by analyzer"
}
}
},
"status" : 400
}
Line 859 is the first instance of a synonym which has a non-alpha synonym, in this case:
✨ => ✨, *, sparkle, sparkles, star
Removing the * from the definition works but then the same issue recurs from line 1257 (✅ => ✅, ✓, button, check, mark) onwards.
This is on Elasticsearch 7.8.0 on Ubuntu 20.04.
Is this a problem with the synonyms file or am I missing something very obvious?
The XML is not perfect (see http://unicode.org/cldr/trac/ticket/9315),
color should be changed to semicolon to ensure full compatibility. CLDR 29 is likely to ship with the error...
In the light of what Github Actions can do, we could have it create a new Pull Request everytime there is a new CLDR release.
The process is:
release-tools/build-released.php to update all the files (need a parameter for the $version variable https://github.com/jolicode/emoji-search/blob/master/tools/build-released.php#L9)That would be awesome.
For information we are at version 38.1, CLDR is at 41!
Variation selector break the synonyms.
Tried:
"cleaning_char_filter": {
"type": "mapping",
"mappings": [
"\uFE0E=>",
"\uFE0F=>"
]
}
But it does works.
To test with the modifiers and ZWJ too.
Flag and countries are not part of CLDR annotations anymore.
We must compute them from the countries data.
ES8 should be tested too.
Also, we could leverage the Elastic "official" action like here: https://github.com/jolicode/elastically/blob/5253ed97fa3ebe45becbcf1fb21fcf58af4a0f68/.github/workflows/ci.yml#L94
trying to install with my 5.2.2 version of ES:
Neptune-2:elasticsearch-5.2.2$ bin/elasticsearch-plugin install https://github.com/jolicode/emoji-search/releases/download/5.2.2/analysis-emoji-5.2.2.zip -v
-> Downloading https://github.com/jolicode/emoji-search/releases/download/5.2.2/analysis-emoji-5.2.2.zip
Retrieving zip from https://github.com/jolicode/emoji-search/releases/download/5.2.2/analysis-emoji-5.2.2.zip
[=================================================] 100%
Exception in thread "main" java.lang.IllegalArgumentException: Plugin [analysis-emoji] is incompatible with Elasticsearch [5.2.2]. Was designed for version [5.0.0]
at org.elasticsearch.plugins.PluginInfo.readFromProperties(PluginInfo.java:108)
at org.elasticsearch.plugins.InstallPluginCommand.verify(InstallPluginCommand.java:421)
at org.elasticsearch.plugins.InstallPluginCommand.install(InstallPluginCommand.java:486)
at org.elasticsearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:212)
at org.elasticsearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:195)
at org.elasticsearch.cli.SettingCommand.execute(SettingCommand.java:54)
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122)
at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:69)
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122)
at org.elasticsearch.cli.Command.main(Command.java:88)
at org.elasticsearch.plugins.PluginCli.main(PluginCli.java:47)
is there any work-around or change i can make to fix this?
We need a specific tokenizer to handle emoji properly in Elasticsearch, here is the specification.
Should be based on the standard tokenizer to keep the grammar logic and Unicode Text Segmentation implementation.
But add support for symbols / emoji as specified in tr51.
| Input | Tokens |
|---|---|
| Get some donut's. | Get some donut's |
| Get some 🍩's. | Get some 🍩's |
| "🍩" | 🍩 |
https://issues.apache.org/jira/projects/LUCENE/versions/12340572
We may be able to simplify the deps as https://issues.apache.org/jira/browse/LUCENE-7540 is fixed.
I am trying to build with gradle in my local machine but I got this:
Task :forbiddenApisMain FAILED
FAILURE: Build failed with an exception.
Parsing signatures failed: Method not found while parsing signature: java.lang.Thread#stop(java.lang.Throwable)
Configure project :
=======================================
Elasticsearch Build Hamster says Hello!
=======================================
Gradle Version : 4.10.2
OS Info : Linux 4.15.0-36-generic (amd64)
JDK Version : Oracle Corporation 11.0.1 [Java HotSpot(TM) 64-Bit Server VM 11.0.1+13-LTS]
JAVA_HOME : /usr/lib/jvm/java-11-oracle
Random Testing Seed : BFFF22ECAE0D02CD
The stacktrace prints the following:
Looks like we could just provide a new Rule File instead of tricking the ICU Tokenizer.
As this code show:
The default config is used when there is no file for the current "script" (which was a fear I had about this possibility to change de Rbbi).
What the plugin could do then:
i am sorry for ask this question but my elasticsearch is 6.0.0
Planning to upgrade to 5.3.. Thanks.
New release is out: http://cldr.unicode.org/index/downloads/cldr-31
At the moment each ES release I have to:
gradle clean check assemble)Maybe this can be automatized.
Make sure their is no copyright issue and add a license.
CLDR 30 is almost ready: http://cldr.unicode.org/index/downloads/cldr-30
It seems that ❤ is not working as intended locally.
To investigate.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.