apache / rocketmq-client-go

Apache RocketMQ go client

RocketMQ Client Go

A product ready RocketMQ Client in pure go, which supports almost the full features of Apache RocketMQ, such as pub and sub messages, ACL, tracing and so on.

---

Due Diligence

Here, we sincerely invite you to take a minute to feedback on your usage scenario.

---

Features

For 2.X version, it supports:

• sending message in synchronous mode
• sending message in asynchronous mode
• sending message in oneway mode
• sending message in batch mode
• sending orderly messages
• sending delay messages
• sending transaction messages
• consuming message using push model
• consuming message using pull model
• consuming message using broadcast model
• message tracing for pub and sub messages
• ACL for producers and consumers
• request-reply model

---

How to use

• Step-by-step instruction are provided in RocketMQ Go Client Introduction
• Consult RocketMQ Quick Start to setup rocketmq broker and nameserver.

---

Apache RocketMQ Community

• RocketMQ Community Projects

---

Contact us

• Mailing Lists: https://rocketmq.apache.org/about/contact/
• Home: https://rocketmq.apache.org
• Docs: https://rocketmq.apache.org/docs/quick-start/
• Issues: https://github.com/apache/rocketmq-client-go/issues
• Ask: https://stackoverflow.com/questions/tagged/rocketmq
• Slack: https://rocketmq-community.slack.com/

---

How to Contribute

Contributions are warmly welcome! Be it trivial cleanup, major new feature or other suggestion. Read this how to contribute guide for more details.

---

License

Apache License, Version 2.0 Copyright (C) Apache Software Foundation

rocketmq-client-go's Issues

Add sendBack for PushConsumer

FEATURE REQUEST

1. Please describe the feature you are requesting.
   Add sendBack for PushConsumer

2. Provide any additional detail on your proposed use case for this feature.

3. concurrently consumer support

4. orderly consumer support

5. Indicate the importance of this issue to you (blocker, must-have, should-have, nice-to-have). Are you currently using any workarounds to address this issue?
   must-have

6. If there are some sub-tasks using -[] for each subtask and create a corresponding issue to map to the sub task:

   • sub-task1-issue-number: sub-task1 description here,
   • sub-task2-issue-number: sub-task2 description here,
   • ...

add delay for message send

FEATURE REQUEST

1. Please describe the feature you are requesting.
   add message delay time to send

2. Provide any additional detail on your proposed use case for this feature.

3. Indicate the importance of this issue to you (blocker, must-have, should-have, nice-to-have). Are you currently using any workarounds to address this issue?
   must-have

4. If there are some sub-tasks using -[] for each subtask and create a corresponding issue to map to the sub task:

centos上编译遇到“未定义的引用”问题

go build .

configcenter/vendor/github.com/apache/rocketmq-client-go/core

/tmp/go-build465039094/b200/_x006.o：在函数‘_cgo_8957cb566e0b_Cfunc_SetProducerNameServerDomain’中：
/tmp/go-build/cgo-gcc-prolog:320：对‘SetProducerNameServerDomain’未定义的引用
/tmp/go-build465039094/b200/_x007.o：在函数‘_cgo_8957cb566e0b_Cfunc_SetPullConsumerNameServerDomain’中：
/tmp/go-build/cgo-gcc-prolog:256：对‘SetPullConsumerNameServerDomain’未定义的引用
/tmp/go-build465039094/b200/_x008.o：在函数‘_cgo_8957cb566e0b_Cfunc_SetPushConsumerNameServerDomain’中：
/tmp/go-build/cgo-gcc-prolog:255：对‘SetPushConsumerNameServerDomain’未定义的引用
collect2: 错误：ld 返回 1

How Does "rocketmq-client-go" Support Transaction

How Does This SDK Support The Methods Of 'Producer And Consumer' Such As : 'BeginTransaction' 'CommitTransaction' 'Rollback'

NewPushConsumer/NewPullConsumer may be move to root dir

Now NewPushConsumer/NewPullConsumer/NewProducer in internal, but other golang projects cannot call it. please move it!

发一收三

用producer发了一条消息。
push_consumer收到三个消息？
打印日志时，一个消息显示三次。
是多发了，还是多打印了？

Separate logic and data-struct

FEATURE REQUEST

1. Please describe the feature you are requesting.
   move all public data-struct to an independent package

2. Provide any additional detail on your proposed use case for this feature.
   none

3. Indicate the importance of this issue to you (blocker, must-have, should-have, nice-to-have). Are you currently using any workarounds to address this issue?
   must-have

[Feature][Alpha2] DelayMessage

Goal

Support delay message in alpha2

Design

TODO

go get rocketmq-client-go error

go get github.com/apache/rocketmq-client-go/core

github.com/apache/rocketmq-client-go/core

../../../Documents/git/go/src/github.com/apache/rocketmq-client-go/core/cfuns.go:21:10: fatal error: 'rocketmq/CMessageExt.h' file not found
#include "rocketmq/CMessageExt.h"
^~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.

##################
How to install?

[Feature][Alpha2] Consuming Message in order

Goal

support consume message orderly

Design

TODO

add trace feature

FEATURE REQUEST

1. Please describe the feature you are requesting.
   add trace feature

2. Provide any additional detail on your proposed use case for this feature.
   add trace feature

3. Indicate the importance of this issue to you (blocker, must-have, should-have, nice-to-have). Are you currently using any workarounds to address this issue?
   must-have

4. If there are some sub-tasks using -[] for each subtask and create a corresponding issue to map to the sub task:

use golangci to improve code quality

FEATURE REQUEST

1. Please describe the feature you are requesting.
   I found that the code in native is so terrible to maintain, maybe author of client is not deeply familar with golang, so in order to improve code quality and pr quality, I think it is time to introduce golangci tool. more info about golangci: https://github.com/golangci/golangci-lint

2. Provide any additional detail on your proposed use case for this feature.

• avoid race case
• useless code
• more concise impl
• field arrangement in struct for less byte

2. Indicate the importance of this issue to you (blocker, must-have, should-have, nice-to-have).
   nice-to-have

3. If there are some sub-tasks using -[] for each subtask and create a corresponding issue to map to the sub task:

Features were postponed to alpha2 from alpha1

docs

• Config

producer

• Load balance
• MessageId Generate

common

• PollNameServer // Move To alpha2

RemoteClient

• Timeout // Move To alpha2
• InvokeAsync

fix ut

fix ut and run ut in CI

windows环境下一直报mingw32/bin/ld.exe: cannot find -lrocketmq

windows环境下一直报mingw32/bin/ld.exe: cannot find -lrocketmq

Add producer async method

FEATURE REQUEST

1. Please describe the feature you are requesting.
   impl producer async method.

2. Provide any additional detail on your proposed use case for this feature.
   fix remote client async timeout bug
   add async method on client and producer

3. Indicate the importance of this issue to you (blocker, must-have, should-have, nice-to-have). Are you currently using any workarounds to address this issue?
   must have

4. If there are some sub-tasks using -[] for each subtask and create a corresponding issue to map to the sub task:

[Optimization][Alpha2] Shutdown with gracefully

TODO

Does it support delay message?

Functional Options

FEATURE REQUEST

1. Please describe the feature you are requesting.
   Using private config members to protect them with functional options

2. Provide any additional detail on your proposed use case for this feature.

• Functional Options Pattern in Go
• gRPC examples

3. Indicate the importance of this issue to you (blocker, must-have, should-have, nice-to-have). Are you currently using any workarounds to address this issue?
   must-have

Stress Testing

FEATURE REQUEST

1. Please describe the feature you are requesting.
   Add Stress Testing for RocketMQ Client Go
2. Provide any additional detail on your proposed use case for this feature.
   TODO
3. Indicate the importance of this issue to you (blocker, must-have, should-have, nice-to-have). Are you currently using any workarounds to address this issue?
   must-have

Add Interceptor for producer and consumer.

FEATURE REQUEST

1. Add Interceptor for producer and consumer.

2. Provide any additional detail on your proposed use case for this feature.
   Sometimes, we want print log、add metrics or timeout check when we send message or consumer messages. But it is ugly to add function around the inner method, and it is not
   necessary demand, so it is a good to add interceptor to decoupling the user custom logic from the inner impl.

3. Indicate the importance of this issue to you (blocker, must-have, should-have, nice-to-have).
   should-have

4. If there are some sub-tasks using -[] for each subtask and create a corresponding issue to map to the sub task:

   • sub-task1-issue-number: add producer and consumer interceptor.

请问后续会提供 fetchOffset，rebalance 之类的功能吗

Support golang client of rocketMQ consume message orderly?

The issue tracker is ONLY used for the python client (feature request of RocketMQ need to follow RIP process). Keep in mind, please check whether there is an existing same report before your raise a new one.

Alternately (especially if your communication is not a bug report), you can send mail to our mailing lists. We welcome any friendly suggestions, bug fixes, collaboration, and other improvements.

Please ensure that your bug report is clear and that it is complete. Otherwise, we may be unable to understand it or to reproduce it, either of which would prevent us from fixing the bug. We strongly recommend the report(bug report or feature request) could include some hints as to the following:

BUG REPORT

1. Please describe the issue you observed:

   • What did you do (The steps to reproduce)?

   • What did you expect to see?

   • What did you see instead?

2. Please tell us about your environment:

   • What is your OS?

   • What is your client version?

   • What is your RocketMQ version?

3. Other information (e.g. detailed explanation, logs, related issues, suggestions on how to fix, etc):

FEATURE REQUEST

1. Please describe the feature you are requesting.

2. Provide any additional detail on your proposed use case for this feature.

3. Indicate the importance of this issue to you (blocker, must-have, should-have, nice-to-have). Are you currently using any workarounds to address this issue?

4. If there are some sub-tasks using -[] for each subtask and create a corresponding issue to map to the sub task:

   • sub-task1-issue-number: sub-task1 description here,
   • sub-task2-issue-number: sub-task2 description here,
   • ...

Add orderly consumer

FEATURE REQUEST

1. Please describe the feature you are requesting.
   Add orderly consumer

2. Provide any additional detail on your proposed use case for this feature.
   Add orderly consumer

3. Indicate the importance of this issue to you (blocker, must-have, should-have, nice-to-have). Are you currently using any workarounds to address this issue?
   must-have

4. If there are some sub-tasks using -[] for each subtask and create a corresponding issue to map to the sub task:

2.0.0-alpha2 Roadmap

The alpha2 has been started, welcome anyone request for an unassigned item or have any contributions for us!

Duration

6.27-7.14

Target

1. refactor API, make implementation, basic data-struct and API have a separation
2. improve stability of features are released in alpha1
3. add new features
4. enrich unit tests

Bugfix

• Missing Tag and Properties in Message #65, PR: #94 @xujianhai666
• The timeout of RPC doesn't work @gaufung PR: #97
• The InvokeAsync doesn't work @gaufung #97

Optimization

• refactor API @wenfengwang #93, PR1: #82, PR2: #103
• implement shutdown @gaufung #97
• support context unassigned postponed to alpha3
• change fatal to error in start() of producer and consumer @gaufung #107

Feature

• Support multiple NameServer @xujianhai666 #96
• Support PullConsumer @xujianhai666 #106
• Support SendAsync @xujianhai666 #100
• Support QueueSelector @xujianhai666 #102
• Support order message in PushConsumer @xujianhai666 #115
• Support Statistics of client @gaufung #113
• Support retry when consume failed in PushConsumer @xujianhai666 #119
• Support Interceptor @xujianhai666 #86
• Support ACL @wenfengwang #117

Unit Test

• package of remote @gaofeng
• producer @xujianhai666 #109
• internal/route.go @luther-siyuan #116
• internal/client.go unassigned postponed to alpha3

use goimport to fmt import order

FEATURE REQUEST

1. Please describe the feature you are requesting.
   In my golang experience, we use goimort to fmt import order, reference by https://godoc.org/golang.org/x/tools/cmd/goimports. Besides, there are lots of pugins for various editor.

2. Provide any additional detail on your proposed use case for this feature.
   consistent view of import order.

3. Indicate the importance of this issue to you (blocker, must-have, should-have, nice-to-have). Are you currently using any workarounds to address this issue?
   nice-to-have

4. If there are some sub-tasks using -[] for each subtask and create a corresponding issue to map to the sub task:

2.0.0-alpha1: Produce message success but missing tag and properties.

BUG REPORT

1. Please describe the issue you observed:

• What did you do (The steps to reproduce)?

Here is my code, it's a producer:

	_msg := kernel.NewMessage(cli.opts.pTopic, body)
	_msg.PutProperty(kernel.PropertyTags, "test_tag")
	_msg.PutProperty("user_defined_key", "hi")
	result, err := cli.p.SendSync(context.Background(), _msg)

	zap.S().Infow("RMQ message", "msg", _msg.String())
	zap.S().Infow("Send Result", "result", result)

• What did you expect to see?

Tag and Property should be shown in RocketMQ Dashboard.

• What did you see instead?

There is not.

Here is my log:

Here is RocketMQ Dashboard:

2. Please tell us about your environment:

   • What is your OS?
     OSX 10.14.4

   • What is your client version?
     2.0.0-alpha1

   • What is your RocketMQ version?
     Sorry, I dont know

3. Other information (e.g. detailed explanation, logs, related issues, suggestions on how to fix, etc):

I notice that ./kernel/request.go, there's no properties in SendRequest.Encode(), So I fixed it as follow:

But It still did not work.

So Maybe the whole rmq message header encode has problem.

Can provide a C++ independent version?

如果没有C++经验，感觉第一步就跪在了编译上

More usability API

FEATURE REQUEST

1. Please describe the feature you are requesting.
   make API more usability and follow the standard of Go

2. Provide any additional detail on your proposed use case for this feature.

• Functional Options
• Separate logic and data-struct
• Minimize functions in each API

3. Indicate the importance of this issue to you (blocker, must-have, should-have, nice-to-have). Are you currently using any workarounds to address this issue?
   must-have

4. If there are some sub-tasks using -[] for each subtask and create a corresponding issue to map to the sub task:

   • Functional Options: Using private config members to protect them
   • Separate logic and data-struct: move all public data-struct to an independent package
   • Refactor APIs: refactor APIs of Producer/PushConsumer/PullConsumer

Can not compile from the scratch

vendor/github.com/apache/rocketmq-client-go/core/push_consumer.go:147:14: could not determine kind of name for C.SetProducerLogFileNumAndSize

vendor/github.com/apache/rocketmq-client-go/core/push_consumer.go:152:14: could not determine kind of name for C.SetProducerLogLevel

vendor/github.com/apache/rocketmq-client-go/core/push_consumer.go:141:14: could not determine kind of name for C.SetProducerLogPath
编译时产生的错误。

Add QueueSelector in producer

FEATURE REQUEST

1. Please describe the feature you are requesting.
   add MesssageSelector in producer

2. Provide any additional detail on your proposed use case for this feature.
   add MessageSelector in ProducerOptions

3. Indicate the importance of this issue to you (blocker, must-have, should-have, nice-to-have). Are you currently using any workarounds to address this issue?
   must-have

4. If there are some sub-tasks using -[] for each subtask and create a corresponding issue to map to the sub task:

example can not work

[root@localhost examples]# go test main.go
./main.go:55:4: undefined: sendMessageOrderly
./main.go:57:4: undefined: sendMessage
./main.go:71:3: undefined: ConsumeWithPush
[root@localhost examples]#

[Feature][Alpha2] Statistic

Goal

statistic metrics

Design

TODO

go-process abort when no topic in MQ-Cluster

When the MQ-cluster does not have any topics, call the SendMessageSync method will cause the GO-program to directly interrupt the exit.
stderr like this:
terminate called after throwing an instance of 'rocketmq::MQException'
what():msg:No route info of this topic,,error:in file </home/jonnxu/git/rocketmq-client-cpp/src/producer/DefaultMQProducer.cpp> line:319
terminate called recursively
SIGABRT:abort
PC=xxxxx

Can you catch that Exception to error, don't let the go-program exit directly?

when application run for some hours，some of all queue consumerOffset don't update，and diffTotal Increasing .

os : mac os、linux
cpp: rocketmq-client-cpp-1.2.2
golang:rocketmq-client-go-1.2.0

当应用启动之后，消费者开始消费，应用运用了几个小时之后，某些broker中的queue 消费者位点不在更新，消费时间也不在更新，diffTotal一直在增长。有没有人遇到过这种问题？

2.0.0-alpha2 PushConsumer Shutdown

2.0.0-alpha2 PushConsumer Shutdown function is just return nil，when can this approach be implemented?

func (pc *pushConsumer) Shutdown() error {
	return nil
}

[native]2.0.0-alpha3 Roadmap

The 2.0.0-rc1 has been started, welcome anyone request for an unassigned item or have any contributions for us!

Duration
7.19-8.7（delayed）

Target

1. Improve stability and reliability
2. More Unit Test: 50% coverage
3. Remain features
4. Optimize API & bugfix

Bugfix

• #135 bufio.Scanner: token too long
• #139 fatal error: concurrent map writes fixed at #146
• #130 missing shutdown in PushConsumer #172
• #144 consumer examples run error fixed at #145
• #166 concurrent map read and map write on message.Properties #197

Optimization

• unified implementation of interceptor @gaufung #132
• Optimize Pull API @wenfengwang #151
• Go Context @xujianhai666 #177
• implement unpress func #164

Features

• Support full AllocateStrategy algorithm @luther-siyuan #141
• Support delay message @xujianhai666 #122
• Support message tracing @xujianhai666 #124
• Support Transaction message @xujianhai666 #136
• Support Namersrv domain @wenfengwang #147
• Support multiple namespace @wenfengwang #147
• Support multiple namesrv cluster @xujianhai666 #162

Test

Unit Test

• consumer/consumer.go @xujianhai666 #179
• consumer/offset_store.go @luther-siyuan #141
• consumer/process_queue.go @gaufung

consumer/pull_consumer.go @wenfengwang(move to v2.1.0)

• consumer/push_consumer.go @cloes #156

• consumer/statistic.go @gaufung #148

• consumer/strategy.go @luther-siyuan #141

• internal/client.go @luther-siyuan

• internal/route.go, internal/namesrv.go, @cloes

• internal/request.go internal/request.go @cloes

• internal/utils @wenfengwang

core/api.go:32:20: undefined: LogConfig

on mac, already installed cpp client

go build

can work.
But, i want to build a ubuntu version on my mac:

env CGO_ENABLED=1 GOOS=linux GOARCH=amd64 go build

throws a error: core/api.go:32:20: undefined: LogConfig

How to fix this?
Thanks

Add PullConsumer

FEATURE REQUEST

1. Please describe the feature you are requesting.
   Add PullConsumer

2. Provide any additional detail on your proposed use case for this feature.
   Add PullConsumer

3. Indicate the importance of this issue to you (blocker, must-have, should-have, nice-to-have). Are you currently using any workarounds to address this issue?
   must-have

4. If there are some sub-tasks using -[] for each subtask and create a corresponding issue to map to the sub task:

Support ACL

FEATURE REQUEST

1. Please describe the feature you are requesting.
   Add ACL Feature

2. Provide any additional detail on your proposed use case for this feature.
   Add credential for ACL

3. Indicate the importance of this issue to you (blocker, must-have, should-have, nice-to-have). Are you currently using any workarounds to address this issue?
   must have

[Feature][Alpha2] MessageRetry

Goal

send message back to broker for retry

Design

TODO

When is it supported native golang?

The current integration into the project is too costly, build hard, deploy hard.

Is it a plan to support native golang?

Support multiple NameServer

FEATURE REQUEST

1. Please describe the feature you are requesting.
   Support multiple NameServer

2. Provide any additional detail on your proposed use case for this feature.
   In production, we actually use a list of NameServer addr, so it is necessary to add addr list to client configuration.

3. Indicate the importance of this issue to you (blocker, must-have, should-have, nice-to-have). Are you currently using any workarounds to address this issue?
   must have

RocketMQ Go Client 2.0.0-alpha1 release Checklist

Description

RocketMQ Go Client in pure go is coming soon, here are some checkpoint before release

Docs

• How to use
• Examples

Producer

MessageType

• NormalMessage

SendWith

• Sync
• OneWay

Consumer

• Consuming With Push
• Consuming With Concurrently
• AllocateMessageQueueAveragely

MessageModel

• CLUSTERING
• BROADCASTING

Other

• Offset Sotre
• Flow Control
• Rebalance
• ConsumeFromWhere
• PersistAllConsumerOffset

Common

• Route Manage
• CleanOfflineBroker
• HeartBeat

Remote

• API
  • InvokeSync
  • InvokeOneWay
• Serialize
  • JSON
  • ROCKETMQ

update mockgen command line

1. Please describe the issue you observed:

when using mockgen to generate mock file ,the generated mock file could cause "import cycle not allowed". I update the mockgen command line to fix this problem. Adding "self_package"in mockgen command line could solve this problem.

2. Please tell us about your environment:

3. Other information (e.g. detailed explanation, logs, related issues, suggestions on how to fix, etc):

4. If there are some sub-tasks using -[] for each subtask and create a corresponding issue to map to the sub task:

Refactor APIs

FEATURE REQUEST

1. Please describe the feature you are requesting.
   refactor APIs of Producer/PushConsumer/PullConsumer

2. Provide any additional detail on your proposed use case for this feature.

• reduce export funcs
• standard function signature

3. Indicate the importance of this issue to you (blocker, must-have, should-have, nice-to-have). Are you currently using any workarounds to address this issue?
   must have

Introduction url is error in readme

maybe need some wiki

core\api.go:32:20: undefined: LogConfig

rocketmq-client-cpp已经安装，
在windows上测试时报：core\api.go:32:20: undefined: LogConfig

[Native]2.0.0 Production Ready RoadMap

Current Status: preparing to releasing 2.0.0-alpha2

Link: RocketMQ Go Client 2.0.0-alpha2 Roadmap

2.0.0-alpha1

• release date: 2019.5.12
• goal:
  • native implementation of RocketMQ Go Client
  • gather feedback
  • attract more contributors
• include:
  • producer with sync/async
  • consumer with push
  • consuming with concurrently
  • clustering & broadcast
  • load balance
  • offset store
  • topic route
  • RocketMQ Protocol & RPC

2.0.0-alpha2

• goal:
  • bugfix
  • more features
  • more tests
• include:
  • delay message
  • Hook
  • consuming with orderly
  • statistics
  • message retry when consume failed(sendBack)

2.0.0-rc1

• goal:
  • bugfix
  • test

2.0.0

• goal: production ready & full feature supported

Add producer unit test

FEATURE REQUEST

1. Please describe the feature you are requesting.
   add producer unit test

2. Provide any additional detail on your proposed use case for this feature.

3. Indicate the importance of this issue to you (blocker, must-have, should-have, nice-to-have). Are you currently using any workarounds to address this issue?
   must-have

4. If there are some sub-tasks using -[] for each subtask and create a corresponding issue to map to the sub task: