infinilabs / analysis-pinyin Goto Github PK
View Code? Open in Web Editor NEW🛵 This Pinyin Analysis plugin is used to do conversion between Chinese characters and Pinyin.
License: Apache License 2.0
🛵 This Pinyin Analysis plugin is used to do conversion between Chinese characters and Pinyin.
License: Apache License 2.0
RT 谢谢了
处理首字母为何要添加分隔符?
例如 “刘德华” => “liu de hua l d h” 为何不是“liu de hua ldh”
if (this.padding_char.length() > 0) {
if (stringBuilder.length() > 0) stringBuilder.append(this.padding_char);
if (firstLetters.length() > 0) firstLetters.append(this.padding_char);
}
看了代码,firstLetters是否不应该加padding_char? 或者将stringBuilder 和 firstLetters的分隔符分别设置更合理?
另外是否有考虑对多音字的支持?pinyin4j本身支持多音字...
thx...
本来想试试这个pinyin插件,我对应的es是2.3.5, 看说明是需要v1.7.5 release支持。 不过唯独没有这个版本,咋回事???
拼音分词现在不支持多音字啊,多音字都是默认取pinyin4j转化出来的第一个拼音,什么时候支持多音字啊?
try {
String[] strs = PinyinHelper.toHanyuPinyinStringArray(c, format);
if (strs != null) {
//get first result by default
String first_value = strs[0];
//TODO more than one pinyin
比如说我想搜索 重庆酸菜鱼(长城店)
在拼音搜索中,通常是用中文分词器,然后用pinyin过滤器。用ik_smart分完词后的结果是 `重庆酸菜鱼`,`长城店`。如果搜索(chongqingsuancaiyu),(chong qing suan cai yu),(cqscy),(changchengdian),(chang cheng dian),(ccd),(重庆酸菜鱼),(长城店)这个几个关键词应该可以搜索到!
hi,medcl大神,我又来这边看pinyin了,请教您个问题,如果我想要通过pinyin进行正向搜索,应该怎么处理呢?比如,我想搜索 “我们来自**” “**是个大国”,我输入‘zg’ 就会把前面的搜出来,您明白这个意思了吗?就是说我想把正向的内容放在前面,或者是像sql中like zg% 这样的。
另外,我分词设置成如下
index:
analysis:
analyzer:
pinyin_analyzer:
tokenizer: my_pinyin
filter: [standard,nGram]
tokenizer:
my_pinyin:
type: pinyin
first_letter: "prifix"
padding_char: " "
映射设置成如下:
POST /tag/keywords/_mapping
{
"keywords": {
"properties": {
"kwname": {
"type": "multi_field",
"fields": {
"kwname": {
"type": "string",
"store": "no",
"term_vector": "with_positions_offsets",
"analyzer": "pinyin_analyzer",
"boost": 10
},
"primitive": {
"type": "string",
"store": "yes",
"analyzer": "keyword"
}
}
}
}
}
}
http://localhost:9200/tag/_analyze?text=%E5%88%98%E5%BE%B7%E5%8D%8E&analyzer=pinyin_analyzer
结果是
{"tokens":[{"token":"l","start_offset":0,"end_offset":3,"type":"word","position":1},{"token":"ld","start_offset":0,"end_offset":3,"type":"word","position":1},{"token":"d","start_offset":0,"end_offset":3,"type":"word","position":1},{"token":"dh","start_offset":0,"end_offset":3,"type":"word","position":1},{"token":"h","start_offset":0,"end_offset":3,"type":"word","position":1},{"token":"hl","start_offset":0,"end_offset":3,"type":"word","position":1},{"token":"l","start_offset":0,"end_offset":3,"type":"word","position":1},{"token":"li","start_offset":0,"end_offset":3,"type":"word","position":1},{"token":"i","start_offset":0,"end_offset":3,"type":"word","position":1},{"token":"iu","start_offset":0,"end_offset":3,"type":"word","position":1},{"token":"u","start_offset":0,"end_offset":3,"type":"word","position":1},{"token":"ud","start_offset":0,"end_offset":3,"type":"word","position":1},{"token":"d","start_offset":0,"end_offset":3,"type":"word","position":1},{"token":"de","start_offset":0,"end_offset":3,"type":"word","position":1},{"token":"e","start_offset":0,"end_offset":3,"type":"word","position":1},{"token":"eh","start_offset":0,"end_offset":3,"type":"word","position":1},{"token":"h","start_offset":0,"end_offset":3,"type":"word","position":1},{"token":"hu","start_offset":0,"end_offset":3,"type":"word","position":1},{"token":"u","start_offset":0,"end_offset":3,"type":"word","position":1},{"token":"ua","start_offset":0,"end_offset":3,"type":"word","position":1},{"token":"a","start_offset":0,"end_offset":3,"type":"word","position":1}]}
这个有点太细了,怎么让他粗一些
比如就要
{"token":"liu","start_offset":0,"end_offset":3,"type":"word","position":1},{"token":"de","start_offset":0,"end_offset":3,"type":"word","position":1},{"token":"hua","start_offset":0,"end_offset":3,"type":"word","position":1},{"token":"ldh","start_offset":0,"end_offset":3,"type":"word","position":1},
应该怎么设置?
谢谢大神~~~~~
我的配置:
{ "settings":{ "index":{ "number_of_shards": 5, "number_of_replicas": 1, "analysis" : { "analyzer" : { "custom_pinyin_analyzer" : { "tokenizer" : "ik_max_word", "filter" : ["full_pinyin_no_space","full_pinyin_with_space","first_letter_pinyin"] } }, "filter" :{ "full_pinyin_no_space" : { "type" : "pinyin", "first_letter" : "none", "padding_char" : "" }, "full_pinyin_with_space" : { "type" : "pinyin", "first_letter" : "none", "padding_char" : " " }, "first_letter_pinyin" : { "type" : "pinyin", "first_letter" : "only", "padding_char" : "" } } } }, "mappings": { "books": { "properties": { "title": { "type":"string", "index":"analyzed", "analyzer":"custom_pinyin_analyzer", "search_analyzer":"custom_pinyin_analyzer", "boost": 10 } } } } } }
PUT library/books/1
{
"title": "联想召回笔记本电源线"
}
直接搜中文能搜出来,用拼音就不行
GET library/books/_search
{
"size" :10,
"from" : 0,
"query": {
"multi_match": {
"type": "best_fields",
"query": "联想",
"fields": [ "title" ],
"tie_breaker": 0.3,
"minimum_should_match" : "10%"
}
}
}
curl -XPUT http://localhost:9200/medcl/ -d'
{
"index" : {
"analysis" : {
"analyzer" : {
"pinyin_analyzer" : {
"tokenizer" : "my_pinyin",
"filter" : "word_delimiter
] }
},
"tokenizer" : {
"my_pinyin" : {
"type" : "pinyin",
"first_letter" : "none",
"padding_char" : " "
}
}
}
}
}'
这边中扩号其实应该是引号。首页demo
首先,感谢medcl大神~
先描述下需求,我们在商城搜索商品,提供建议时使用该插件。
大体的需求:
1、输入汉字、拼音时,根据商品名称从首字进行匹配,而且是根据声母和全拼都可以。
2、应该过滤特殊字符,比如()*/等
模仿readme中例子,尝试建了index:
{
"index" : {
"analysis" : {
"analyzer" : {
"only_pinyin_analyzer" : {
"tokenizer" : "only_pinyin"
}
,"none_pinyin_analyzer" : {
"tokenizer" : "none_pinyin"
}
}
,"tokenizer" : {
"only_pinyin" : {
"type" : "pinyin",
"first_letter" : "only",
"padding_char" : ""
}
,"none_pinyin" : {
"type" : "pinyin",
"first_letter" : "none",
"padding_char" : ""
}
}
}
}
}
然后进行测试
_analyze?analyzer=only_pinyin_analyzer&pretty&text=(散)奇趣蛋功夫熊猫版20g*3粒/组
结果如下
{
"tokens": [
{
"token": "s",
"start_offset": 1,
"end_offset": 2,
"type": "word",
"position": 0
},
{
"token": "qqdgfxmb20g",
"start_offset": 3,
"end_offset": 14,
"type": "word",
"position": 1
},
{
"token": "3l",
"start_offset": 15,
"end_offset": 17,
"type": "word",
"position": 2
},
{
"token": "z",
"start_offset": 18,
"end_offset": 19,
"type": "word",
"position": 3
}
]
}
文本被断词,而使用none这个tokenizer时
_analyze?analyzer=none_pinyin_analyzer&pretty&text=(散)奇趣蛋功夫熊猫版20g*3粒/组
结果
{
"tokens": [
{
"token": "sanqiqudangongfuxiongmaoban20g 3li zu",
"start_offset": 0,
"end_offset": 19,
"type": "word",
"position": 0
}
]
}
并没有被断词。。
curl -XPUT 'http://localhost:9200/medcl/' -d'
{
"index" : {
"analysis" : {
"analyzer" : {
"pinyin_analyzer" : {
"tokenizer" : "my_pinyin",
"filter" : ["standard","nGram"]
}
},
"tokenizer" : {
"my_pinyin" : {
"type" : "pinyin",
"first_letter" : "prefix",
"padding_char" : ""
}
}
}
}
}'
curl -XPOST 'http://localhost:9200/medcl/folks/_mapping' -d'
{
"folks": {
"properties": {
"name": {
"type": "multi_field",
"fields": {
"name": {
"type": "string",
"store": "no",
"term_vector": "with_positions_offsets",
"analyzer": "pinyin_analyzer",
"boost": 10
},
"primitive": {
"type": "string",
"store": "yes",
"analyzer": "keyword"
}
}
}
}
}
}'
curl -XPOST 'http://localhost:9200/medcl/folks/andy' -d'{"name":"刘德华"}'
curl -XPOST 'http://localhost:9200/medcl/folks/xiaom' -d'{"name":"刘小明"}'
curl 'http://localhost:9200/medcl/folks/_search?pretty' -d '{
"query": {
"query_string":{
"query": "name:dehua",
"default_operator": "AND"
}
}
}'
0.90.7 查询结果小明与德华多匹配成功了!
0.90.2 只会把德华匹配出来
多谢你的辛苦,你的插件很有用.
但是在es 1.2.2下怎么安装都失败,请问可否帮忙看下?
Message:
Error while installing plugin, reason: IllegalArgumentException: Plugin installation assumed to be site plugin, but contains source code, aborting installation.
麻烦你了&多谢
ES 2.0 安装需要 plugin-descriptor.properties 文件 并且需要设置好参数
你的master版本中 plugin-descriptor.properties 放在src里了 而且没有配置好参数
你能说明下如何配置参数嘛
Can you provide details about how to compile and package this to install it for Elastic Search 2.0
I would love to use this!
到第三步时
curl -XPOST http://localhost:9200/medcl/folks/andy -d'{"name":"刘德华"}'
curl -XPOST http://localhost:9200/medcl/folks/andy -d'{"name":"李小明"}'
然后
curl http://localhost:9200/medcl/folks/_search\?q\=name:%e5%88%98\&pretty\=1
{
"took" : 10,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"failed" : 0
},
"hits" : {
"total" : 2,
"max_score" : 2.8713017,
"hits" : [ {
"_index" : "medcl",
"_type" : "folks",
"_id" : "andy",
"_score" : 2.8713017, "_source" : {"name":"刘德华"}
}, {
"_index" : "medcl",
"_type" : "folks",
"_id" : "andy2",
"_score" : 0.89630747, "_source" : {"name":"李小明"}
} ]
}
}%
查找"刘",为什么“李小明”也查的到?
curl http://localhost:9200/medcl/folks/_search?q=name:%e5%88%98%e5%be%b7
curl http://localhost:9200/medcl/folks/_search?q=name:liu
curl http://localhost:9200/medcl/folks/_search?q=name:ldh
curl http://localhost:9200/medcl/folks/_search?q=name:dehua
下面几个也是一样,请问下,这个是什么情况?谢谢
analysis configuration (elasticsearch.yml)
index:
analysis:
analyzer:
ik:
alias: [ik_analyzer]
type: org.elasticsearch.index.analysis.IkAnalyzerProvider
mmseg:
alias: [mmseg_analyzer]
type: org.elasticsearch.index.analysis.MMsegAnalyzerProvider
pinyin:
alias: [pinyin_analyzer]
filter : [standard]
type: org.elasticsearch.index.analysis.PinyinAnalyzerProvider
tokenizer: my_pinyin
tokenizer:
my_pinyin:
type: pinyin
first_letter: [none]
padding_char: [ ]
index.analysis.analyzer.default.type : "mmseg"
not is run
please give me guild
各位大神好
比如: 银行(yin xing) 香桔(xiang jie)
这样输入的是 yinhang 那么查询的结果完全是不正确的,
看了上面的提问, 这个问题忽略
---------------------以下----------
首拼: 香桔市 (xjs) 锦绣世界(jxsj) 这个时候, 锦绣世界就排到了 香桔市的前面去了, 因为词频上多了一个j
怎么样才能用完整的顺序(xjs) 去搜首拼呢? 没有找到
求助
http://localhost:9200/users/_analyze?text=%E8%B2%8C%E7%BE%8E%E5%A6%82%E8%AA%AE&analyzer=pinyin_analyzer
返回
{"error":{"root_cause":[{"type":"remote_transport_exception","reason":"[q_nVrMt][172.17.0.2:9300][indices:admin/analyze[s]]"}],"type":"string_index_out_of_bounds_exception","reason":"String index out of range: 0"},"status":500}
服务器500
版本为最新 elasticsearch 5
比如 3d桌球 first_letter 就是zq 而不是 3dzq
譬如我搜 ceshi(测试等) 他把不属于ceshi的都会搜出来如何改变这个mapping ngram 我用multi_match 添加了operate and这个属性
使用拼音查询时,怎么高亮显示对应的中文啊
我想先使用elasticsearch-analysis-ansj进行分词,然后使用elasticsearch-analysis-pinyin进行拼音处理,处理命令如下:
curl -XGET 'localhost:9200/_analyze?tokenizer=index_ansj&filter=pinyin&pretty' -d '你好,我是小明的同学小强'
但是无法输出分词后的拼音结果,结果如下:
{
"tokens" : [ {
"token" : "你好",
"start_offset" : 0,
"end_offset" : 2,
"type" : "l",
"position" : 0
}, {
"token" : ",",
"start_offset" : 2,
"end_offset" : 3,
"type" : "w",
"position" : 1
}, {
"token" : "我",
"start_offset" : 3,
"end_offset" : 4,
"type" : "r",
"position" : 2
}, {
"token" : "是",
"start_offset" : 4,
"end_offset" : 5,
"type" : "v",
"position" : 3
}, {
"token" : "小明",
"start_offset" : 5,
"end_offset" : 7,
"type" : "nr",
"position" : 4
}, {
"token" : "的",
"start_offset" : 7,
"end_offset" : 8,
"type" : "uj",
"position" : 5
}, {
"token" : "同学",
"start_offset" : 8,
"end_offset" : 10,
"type" : "n",
"position" : 6
}, {
"token" : "小强",
"start_offset" : 10,
"end_offset" : 12,
"type" : "nr",
"position" : 7
} ]
}
拼音和ik_smart如何一起使用?
我现在想用ik_smart 和拼音,不知道mapping如何配置。请高手指点下,谢谢。
{
"py_v2": {
"aliases": {},
"mappings": {},
"settings": {
"index": {
"creation_date": "1466507976098",
"analysis": {
"filter": {
"autocomplete_filter": {
"type": "edge_ngram",
"min_gram": "2",
"max_gram": "20"
}
},
"analyzer": {
"pinyin_analyzer": {
"filter": [
"word_delimiter",
"autocomplete_filter",
"length"
],
"tokenizer": "my_pinyin"
}
},
"tokenizer": {
"my_pinyin": {
"padding_char": " ",
"type": "pinyin",
"first_letter": "none"
}
}
},
"number_of_shards": "5",
"number_of_replicas": "1",
"uuid": "nIa_gLHvRIe2HZtGC4e8Yg",
"version": {
"created": "2020199"
}
}
},
"warmers": {}
}
}
一模一样的,就是查不出来。。。我的天啊。。。。哪位大哥,告诉我一下吧。谢谢谢谢谢谢谢谢。。。
'settings' => [
'goods'=> [
"analysis" => [
"analyzer" => [
"pinyin_analyzer" => [
"tokenizer" => "my_pinyin",
"filter" => ["standard"]
],
],
],
"tokenizer" => [
"my_pinyin" => [
"type" => "pinyin",
"first_letter" => "none",
"padding_char" => " "
],
],
],
],
'mappings' => [
'goods' => [
"properties" => [
"name" => [
"type" => "multi_field",
"fields" => [
'name'=>[
"type" => "string",
"store" => "no",
"term_vector" => "with_positions_offsets",
"analyzer" => "pinyin_analyzer",
"boost" => 10
],
"primitive" => [
"type" => "string",
"store" => "yes",
"analyzer" => "keyword"
],
],
]
]
]
]
今日在测试ES2.1部署新ik
插件和pinyin
插件,同时发现了此问题,就是在plugin-descriptor.properties
文件中,同时用到name=${elasticsearch.plugin.name}
,在启动ES服务的时候会发现:
[node-1] loaded [license, marvel-agent, ${elasticsearch.plugin.name}], sites [kopf, sense, head]
此时在创建索引时用到分词可能会出现索引创建后产生索引异常,分片无法分配的情况;
测试修改为 name=ik
,name=pinyin
后,启动ES服务:
[node-1] loaded [license, marvel-agent,ik,pinyin], sites [kopf, sense, head]
再重新创建索引后,问题消失。
@medcl 大神,你好。我用es2.2.1版本在做关键字搜索建议的时候,想实现suggest的汉字、拼音、简拼搜索,用了各种办法都没法实现。请帮忙看一下,哪些设置错了。另:我看到前面的帖子里有设置属性类型为multi_field。但suggest没办法设置为multi_fileld啊。
1.索引设置如下
curl -X PUT 192.168.1.45:9200/musiczz/ -d '{
"index": {
"analysis": {
"analyzer": {
"pinyin_analyzer": {
"type": "custom",
"tokenizer": "my_pinyin",
"filter": [
"simple_pinyin",
"word_delimiter"
]
}
},
"tokenizer": {
"my_pinyin": {
"type": "pinyin",
"first_letter": "none",
"padding_char": ""
}
},
"filter": {
"simple_pinyin": {
"type": "pinyin",
"first_letter": "only",
"padding_char": ""
}
}
}
}
}'
2、mapping如下:
curl -X PUT 192.168.1.45:9200/musiczz/song/_mapping -d '{
"song" : {
"properties" : {
"name" : {"type" : "string"},
"suggest" : { "type" : "completion",
"analyzer" : "pinyin_analyzer",
"search_analyzer" : "pinyin_analyzer",
"payloads" : true
}
}
}
}'
3、索引数据
curl -X PUT '192.168.1.45:9200/musiczz/song/4?refresh=true' -d '{
"name" : "刘德华",
"suggest" : {
"input": [ "刘德华"],
"output": "刘德华",
"payload" : { "artistId" : 2325 },
"weight" : 34
}
}'
4、查询suggest
1)可以查询到结果
curl -X POST '192.168.1.45:9200/musiczz/_suggest?pretty' -d '{
"song-suggest" : {
"text" : "liudehua",
"completion" : {
"field" : "suggest"
}
}
}'
结果如下:
{
"_shards" : {
"total" : 5,
"successful" : 5,
"failed" : 0
},
"song-suggest" : [ {
"text" : "liudehua",
"offset" : 0,
"length" : 8,
"options" : [ {
"text" : "刘德华",
"score" : 34.0,
"payload" : {
"artistId" : 2325
}
} ]
} ]
}
2)可以查询到结果
curl -X POST '192.168.1.45:9200/musiczz/_suggest?pretty' -d '{
"song-suggest" : {
"text" : "刘德华",
"completion" : {
"field" : "suggest"
}
}
}'
3)简拼不能查询到结果
curl -X POST '192.168.1.45:9200/musiczz/_suggest?pretty' -d '{
"song-suggest" : {
"text" : "ldh",
"completion" : {
"field" : "suggest"
}
}
}'
请问,如果我要实现简拼也可以查到结果,索引应该如何设置???
Caused by: org.elasticsearch.ElasticsearchIllegalArgumentException: failed to find analyzer type [null] or tokenizer for [pinyin_analyzer]
at org.elasticsearch.index.analysis.AnalysisModule.configure(AnalysisModule.java:383)
at org.elasticsearch.common.inject.AbstractModule.configure(AbstractModule.java:60)
at org.elasticsearch.common.inject.spi.Elements$RecordingBinder.install(Elements.java:204)
at org.elasticsearch.common.inject.spi.Elements.getElements(Elements.java:85)
at org.elasticsearch.common.inject.InjectorShell$Builder.build(InjectorShell.java:130)
at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:99)
at org.elasticsearch.common.inject.InjectorImpl.createChildInjector(InjectorImpl.java:131)
at org.elasticsearch.common.inject.ModulesBuilder.createChildInjector(ModulesBuilder.java:69)
at org.elasticsearch.indices.IndicesService.createIndex(IndicesService.java:328)
... 8 more
[2015-05-01 23:39:31,625][WARN ][cluster.action.shard ] [Brynocki] [medcl][1] received shard failed for [medcl][1], node[aUJOyVgfTzmo3P-okd9M3A], [P], s[INITIALIZING], indexUUID [jRV45HekR4Og1tlDYEgs1Q], reason [shard failure [failed to create index][IndexCreationException[[medcl] failed to create index]; nested: ElasticsearchIllegalArgumentException[failed to find analyzer type [null] or tokenizer for [pinyin_analyzer]]; ]]
使用pinyin后集群健康状态就会出问题,请问是什么原因?@medcl
没找到安装的说明,只能去网上扒拉下,参照这个http://my.oschina.net/xiaohui249/blog/214505 编译了pinyin的jar包
生成两个jar包(elasticsearch-analysis-pinyin-1.7.2.jar pinyin4j-2.5.0.jar
),已放在[ES_HOME]/plugins/pinyin
路径下
然后service elasticsearch start
报下边的错误,感觉是版本的问题。
重装了es到2.3.2版本依然不行。。。。只能求教了。。。
正在启动 elasticsearch:Exception in thread "main" java.lang.IllegalStateException: Could not load plugin descriptor for existing plugin [pinyin]. Was the plugin built before 2.0?
Likely root cause: java.nio.file.NoSuchFileException: /usr/share/elasticsearch/plugins/pinyin/plugin-descriptor.properties
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
at java.nio.file.Files.newByteChannel(Files.java:361)
at java.nio.file.Files.newByteChannel(Files.java:407)
at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:384)
at java.nio.file.Files.newInputStream(Files.java:152)
at org.elasticsearch.plugins.PluginInfo.readFromProperties(PluginInfo.java:87)
at org.elasticsearch.plugins.PluginsService.getPluginBundles(PluginsService.java:378)
at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:128)
at org.elasticsearch.node.Node.<init>(Node.java:158)
at org.elasticsearch.node.Node.<init>(Node.java:140)
at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:143)
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:178)
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:270)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)
Refer to the log for complete error details.
[失败]
我的例子如下:
curl -XPOST http://localhost:9200/medcl/_close
curl -XPUT http://localhost:9200/medcl/_settings -d'
{
"index" : {
"analysis" : {
"analyzer" : {
"pinyin_analyzer" : {
"tokenizer" : "my_pinyin",
"filter" : ["standard","nGram"]
}
},
"tokenizer" : {
"my_pinyin" : {
"type" : "pinyin",
"first_letter" : "prefix",
"padding_char" : ""
}
}
}
}
}'
curl -XPOST http://localhost:9200/medcl/_open
curl -XPOST http://localhost:9200/medcl/folks/_mapping -d'
{
"folks": {
"properties": {
"name": {
"type": "string",
"term_vector": "with_positions_offsets",
"analyzer": "pinyin_analyzer"
}
}
}
}'
curl -XPOST http://localhost:9200/medcl/folks/1 -d'{"name":"你好,我是abc公司"}'
curl -XPOST http://localhost:9200/medcl/folks/2 -d'{"name":"你好,我是abc"}'
curl http://localhost:9200/medcl/folks/_search?q=name:gongsi
但是返回结果把1和2都返回回来了
elasticsearch版本是1.6.0
git clone 工程 后, 执行mvn package,然后通过如下方式安装插件
plugin --install analysis-pinyin --url file:///#{path}/elasticsearch-analysis-pinyin/target/releases/elasticsearch-analysis-pinyin-1.3.0.zip
之后照着README上的步骤操作时, 在Creat Mapping时老是会报错:
{"error":"IndexCreationException[[medcl] failed to create index]; nested: ElasticsearchIllegalArgumentException[failed to find analyzer type [null] or tokenizer for [pinyin_analyzer]]; ","status":400}
服务器端的错误日志如下:
Caused by: org.elasticsearch.ElasticsearchIllegalArgumentException: failed to find analyzer type [null] or tokenizer for [pinyin_analyzer]
at org.elasticsearch.index.analysis.AnalysisModule.configure(AnalysisModule.java:383)
at org.elasticsearch.common.inject.AbstractModule.configure(AbstractModule.java:60)
at org.elasticsearch.common.inject.spi.Elements$RecordingBinder.install(Elements.java:204)
at org.elasticsearch.common.inject.spi.Elements.getElements(Elements.java:85)
at org.elasticsearch.common.inject.InjectorShell$Builder.build(InjectorShell.java:130)
at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:99)
at org.elasticsearch.common.inject.InjectorImpl.createChildInjector(InjectorImpl.java:131)
at org.elasticsearch.common.inject.ModulesBuilder.createChildInjector(ModulesBuilder.java:69)
at org.elasticsearch.indices.IndicesService.createIndex(IndicesService.java:336)
请问我是遗漏了哪一个关键步骤吗?
您好,在拜读代码。
我有一个疑问,在PinyinTokenFilter中如果first_letter设为prefix,那么每个字的拼音前会前缀其拼音的声母。例如“刘”会被拼音转为liu 然后最终产生的termAttribute是“lliu”。这个操作的意义是神马?
初次接触soundex search,请多指教。
配置:
pinyin_ik_first_letter:
type: custom
tokenizer: ik
filter:
- pinyin_first_letter
- lowercase
- unique
测试:
GET /test/_analyze?text=abc北京***&analyzer=pinyin_ik_first_letter&pretty
{
"tokens" : [ {
"token" : "",
"start_offset" : 0,
"end_offset" : 3,
"type" : "ENGLISH",
"position" : 1
}, {
"token" : "bj",
"start_offset" : 3,
"end_offset" : 5,
"type" : "CN_WORD",
"position" : 2
}, {
"token" : "j",
"start_offset" : 4,
"end_offset" : 5,
"type" : "CN_WORD",
"position" : 3
}, {
"token" : "tam",
"start_offset" : 5,
"end_offset" : 8,
"type" : "CN_WORD",
"position" : 4
}, {
"token" : "ta",
"start_offset" : 5,
"end_offset" : 7,
"type" : "CN_WORD",
"position" : 5
}, {
"token" : "m",
"start_offset" : 7,
"end_offset" : 8,
"type" : "CN_CHAR",
"position" : 6
} ]
}
abc 不见了,是什么原因呀?
大神,我又来了
刚才正向确实可以,但有个比较实际的问题
http://localhost:9200/tag/_analyze?text=%E5%88%98%E5%BE%B7%E5%8D%8E&analyzer=pinyin_analyzer
后结果是
{"tokens":[{"token":"ldh liu de hua ","start_offset":0,"end_offset":3,"type":"word","position":1}]}
用prefix搜索 ldh 确实从正向搜出结果的,但是如果搜 liu 这样就没有结果了,这个该怎么办呢?
现在就是想在正向搜索的基础上可以实现 开头字母和pinyin都可以搜出结果。
我碰到的问题就是如标题所述。我的想法是只按照完整的拼音来建索引,所以按照文档里的第一种方式来配置pinyin analyzer:
curl -XPUT http://localhost:9200/medcl/ -d'
{
"index" : {
"analysis" : {
"analyzer" : {
"pinyin_analyzer" : {
"tokenizer" : "my_pinyin",
"filter" : ["standard"]
}
},
"tokenizer" : {
"my_pinyin" : {
"type" : "pinyin",
"first_letter" : "none",
"padding_char" : " "
}
}
}
}
}'
mapping的配置如下:
curl -XPOST localhost:9200/searchshowindex_v3/searchshowtype/_mapping -d'
{
"searchshowtype":{
"_all":{"analyzer":"pinyin_analyzer","term_vector":"no","store":false}
}}'
但是对数据索引之后,在地址栏中随便输入无论什么样的拼音,即便是一个字母的也得不到任何结果,必须在filter里面配置nGram才有结果,但这又不是我要的,配了nGram无论输入什么拼音基本上所有的数据都能被搜索到,这个体验不好。
所以想问一下medcl为什么会发生这种问题,是我哪里理解错了,没有配好?如果方便的话麻烦能帮我解决这一疑惑。
目前是221
实验证明输入任何的内容都会显示所有的的记录,跟关键字无关。请教这是怎么回事?
大神,pinyin里的"nGram"分词太细了,基本上所有内容都会出来,去掉后只能输入完整的拼音或字符才能出来,请问有没有什么好的解决办法。
这是我按教程的配置:
{
"index" : {
"analysis" : {
"analyzer" : {
"pinyin_analyzer" : {
"tokenizer" : "my_pinyin",
"filter" : ["word_delimiter","nGram"]
}
},
"tokenizer" : {
"my_pinyin" : {
"type" : "pinyin",
"first_letter" : "prefix",
"padding_char" : ""
}
}
}
}
}
如题?
{
"es-goods": {
"properties": {
"name": {
"type": "multi_field",
"fields": {
"name": {
"type": "string",
"store": "no",
"term_vector": "with_positions_offsets",
"analyzer": "pinyin_analyzer",
"boost": 10
},
"primitive": {
"type": "string",
"store": "yes",
"analyzer": "keyword"
}
}
}
}
}
}
最新版本支持 2.2 么? 有没有更详细的操作文档?
I'm using elasticsearch v2.1.1. ./plugin install medcl/elasticsearch-analysis-pinyin
would fail.
hi,medcl,
pinyin analyzer can not meet the requirement for production use as we first need to tokenize our input sentence and then convert each term to pinyin for query. I can help to add a pinyin filter for this module but I do not know how to contribute.
[WARNING]
[WARNING] Some problems were encountered while building the effective model for org.elasticsearch:elasticsearch-analysis-pinyin:jar:1.7.4
[WARNING] 'dependencies.dependency.systemPath' for net.sourceforge.pinyin4j:pinyin4j:jar should not point at files within the project directory, ${basedir}/lib/pinyin4j-2.5.0.jar will be unresolvable by dependent projects @ line 64, column 25
[WARNING]
[WARNING] It is highly recommended to fix these problems because they threaten the stability of your build.
[WARNING]
[WARNING] For this reason, future Maven versions might no longer support building such malformed projects.
[WARNING]
hi, @medcl大神,我下载了最新的elasticsearch-rtf,按照pinyin插件一步步执行,最后查询结果不准确,总是无法匹配查询结果,反复重复了几遍,不知问题出现在哪里,希望有时间指教一下,先谢过 :)
Caused by: java.lang.IllegalArgumentException: Analyzer [pinyin_analyzer] must have a type associated with it
at org.elasticsearch.index.analysis.AnalysisModule.configure(AnalysisModule.java:313)
at org.elasticsearch.common.inject.AbstractModule.configure(AbstractModule.java:61)
at org.elasticsearch.common.inject.spi.Elements$RecordingBinder.install(Elements.java:233)
at org.elasticsearch.common.inject.spi.Elements.getElements(Elements.java:105)
at org.elasticsearch.common.inject.InjectorShell$Builder.build(InjectorShell.java:143)
at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:99)
at org.elasticsearch.common.inject.InjectorImpl.createChildInjector(InjectorImpl.java:159)
at org.elasticsearch.common.inject.ModulesBuilder.createChildInjector(ModulesBuilder.java:55)
at org.elasticsearch.indices.IndicesService.createIndex(IndicesService.java:358)
我先作用的是es2.2.1的版本,ik,pinyin插件的版本都与 这个版本匹配,想用combo但是好像版本对不上,不知道怎么办了
只有 ik 或者只有 pinyin,es2.1.1 都能正常启动,但是一旦两个都安装,es 就会启动失败
[2015-12-30 03:07:04,397][INFO ][node ] [Eleggua] version[2.1.1], pid[1], build[40e2c53/2015-12-15T13:05:55Z]
[2015-12-30 03:07:04,399][INFO ][node ] [Eleggua] initializing ...
Exception in thread "main" java.lang.IllegalStateException: failed to load bundle [file:/usr/share/elasticsearch/plugins/pinyin/elasticsearch-analysis-pinyin-1.5.2.jar, file:/usr/share/elasticsearch/plugins/pinyin/pinyin4j-2.5.0.jar, file:/usr/share/elasticsearch/plugins/ik/commons-logging-1.2.jar, file:/usr/share/elasticsearch/plugins/ik/httpclient-4.4.1.jar, file:/usr/share/elasticsearch/plugins/ik/httpcore-4.4.1.jar, file:/usr/share/elasticsearch/plugins/ik/elasticsearch-analysis-ik-1.6.2.jar, file:/usr/share/elasticsearch/plugins/ik/commons-codec-1.9.jar] due to jar hell
Likely root cause: java.lang.IllegalStateException: jar hell!
class: org.elasticsearch.analysis.PinyinIndicesAnalysis$3
jar1: /usr/share/elasticsearch/plugins/pinyin/elasticsearch-analysis-pinyin-1.5.2.jar
jar2: /usr/share/elasticsearch/plugins/ik/elasticsearch-analysis-ik-1.6.2.jar
at org.elasticsearch.bootstrap.JarHell.checkClass(JarHell.java:280)
at org.elasticsearch.bootstrap.JarHell.checkJarHell(JarHell.java:186)
at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:336)
at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:109)
at org.elasticsearch.node.Node.<init>(Node.java:146)
at org.elasticsearch.node.Node.<init>(Node.java:128)
at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:145)
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:178))
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:285)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)
Refer to the log for complete error details.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.