streamnative / pulsarctl

a CLI for Apache Pulsar written in Go

Pulsarctl

A CLI tool for the Apache Pulsar project.

Install pulsarctl

Mac operation system

Use homebrew to install pulsarctl on the Mac operation system.

brew tap streamnative/streamnative
brew install pulsarctl

Linux operation system

Use this command to install pulsarctl on the Linux operation system.

sh -c "$(curl -fsSL https://raw.githubusercontent.com/streamnative/pulsarctl/master/install.sh)"

Windows operation system

To install pulsarctl on the Windows operation system, follow these steps:

1. Download the package from here.
2. Add the pulsarctl directory to your system PATH.
3. Type pulsarctl -h at CMD to verify that pulsarctl is work.

As project dependency

Use go get to install the pulsarctl as dependency on your project:

# Using master branch
go get github.com/streamnative/pulsarctl@master

# Or using v2.10.1.3 tag
go get github.com/streamnative/[email protected]

# Or using v2.9.3.3 tag
go get github.com/streamnative/[email protected]

Note: We will not release v0.y.z tags, so you need to specify the branch or tag.

Build pulsarctl from codes

Prerequisite

• Go 1.11 +

  If you have not installed Go, install it according to the installation instruction.

  Since the go mod package management tool is used in this project, Go 1.11 or higher version is required.

Steps

1. Download codes.

   1. Clone the project from GitHub to your local.

      git clone https://github.com/streamnative/pulsarctl.git

   2. Use go mod to get the dependencies needed for the project.

      go mod download

      After entering the go mod download command, if some libs can not be downloaded, then you can download them by referring to the proxy provided by GOPROXY.io.

2. Build pulsarctl

   make pulsarctl

   If you want to print version of pulsarctl, you can use follow commands:

   bin/pulsarctl --version
   

   Output:

   Release Version: v2.8.0.3
   Git Commit Hash: add1e586e8d56fcc1cbd2fe94ed75b10d6c54672
   Git Branch: HEAD
   UTC Build Time: 2021-06-26 06:39:16
   Go Version: go version go1.16.3 darwin/amd64
   

Enable Auto-Completion

If you want to enable autocompletion in shell, see enable_completion.

Use contexts

If you want to cache information of multiple clusters, and can switch between multiple clusters, see How to use pulsarctl context.

Project Status

The following is an incomplete list of features that are not yet implemented:

Functions

• localrun

Sources

• localrun
• available-sources
• reload

Sinks

• localrun
• available-sources
• reload

Schemas

• extract

Namespaces

• delete-bookie-affinity-group
• get-bookie-affinity-group
• set-bookie-affinity-group

Bookies

• racks-placement
• get-bookie-rack
• delete-bookie-rack
• set-bookie-rack

Different With Java Pulsar Admin

We move the subscription commands from the topics to the subscriptions in pulsarctl.

pulsar-admin	pulsarctl
bin/pulsar-admin topics create-subscription	pulsarctl subscription create
bin/pulsar-admin topics unsubscribe	pulsarctl subscription delete
bin/pulsar-admin topics skip	pulsarctl subscription skip
bin/pulsar-admin topics expire-messages	pulsarctl subscription expire
bin/pulsar-admin topics peek-messages	pulsarctl subscription peek
bin/pulsar-admin topics reset-cursor	pulsarctl subscription seek
bin/pulsar-admin topics subscriptions	pulsarctl subscription list

Contribute

Contributions are welcomed and greatly appreciated. For more information about how to submit a patch and the contribution workflow, see CONTRIBUTING.md.

License

Licensed under the Apache License Version 2.0: http://www.apache.org/licenses/LICENSE-2.0

pulsarctl's Issues

Provide GPG signature and SHA3-512 hashsum for binaries

Thank you for the project!

Since pulsar is a mission critical software for us, it would be nice to add some security for the installation process from the binaries. It would be great to have a way of validating the correct hashsum and signature of the binary.

can't get full properties of message

When i use pulsarctl peek message, I can not see full properties. When use pulsar-admin it work fine.

Every message was set Content-Type inside properties. Here is my log:

./bin/pulsarctl subscriptions peek default/public/changed LogJobConsumer

Message ID : 17334:60:-1:-1
Properties :
{
    "publish-time": "2020-01-03T15:41:40.12+08:00"
}
Message :
00000000  7b 22 70 61 79 6c 6f 61  64 22 3a 7b 22 62 65 66  |{"payload":{"bef|
00000010  6f 72 65 22 3a 7b 7d 2c  22 61 66 74 65 72 22 3a  |ore":{},"after":|
00000020  7b 7d 2c 22 73 6f 75 72  63 65 22 3a 7b 22 6d 6f  |{},"source":{"mo|
00000030  64 65 6c 22 3a 22 4f 72  64 65 72 52 65 6c 65 74  |del":"OrderRelet|
00000040  4c 6f 67 22 2c 22 74 61  62 6c 65 22 3a 22 6f 72  |Log","table":"or|
00000050  64 65 72 5f 72 65 6c 65  74 5f 6c 6f 67 73 22 2c  |der_relet_logs",|
00000060  22 69 64 22 3a 22 35 37  66 63 61 36 34 35 2d 38  |"id":"57fca645-8|
00000070  32 66 34 2d 34 39 61 39  2d 61 37 30 38 2d 61 37  |2f4-49a9-a708-a7|
00000080  33 65 33 61 64 66 35 31  63 61 22 7d 7d 2c 22 6f  |3e3adf51ca"}},"o|
00000090  70 22 3a 22 75 22 2c 22  74 73 5f 6d 73 22 3a 31  |p":"u","ts_ms":1|
000000a0  35 37 38 30 33 37 33 30  30 31 32 30 7d           |578037300120}|

./pulsar-admin persistent peek-messages --subscription ReletLogJobConsumer default/public/changed 

Message ID: 17334:44
Tenants:
{
  "Content-Type" : "application/json; charset=utf-8",
  "publish-time" : "2020-01-03T14:59:32.532+08:00"
}
         +-------------------------------------------------+
         |  0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f |
+--------+-------------------------------------------------+----------------+
|00000000| 7b 22 70 61 79 6c 6f 61 64 22 3a 7b 22 62 65 66 |{"payload":{"bef|
|00000010| 6f 72 65 22 3a 7b 7d 2c 22 61 66 74 65 72 22 3a |ore":{},"after":|
|00000020| 7b 7d 2c 22 73 6f 75 72 63 65 22 3a 7b 22 6d 6f |{},"source":{"mo|
|00000030| 64 65 6c 22 3a 22 4f 72 64 65 72 52 65 6c 65 74 |del":"OrderRelet|
|00000040| 4c 6f 67 22 2c 22 74 61 62 6c 65 22 3a 22 6f 72 |Log","table":"or|
|00000050| 64 65 72 5f 72 65 6c 65 74 5f 6c 6f 67 73 22 2c |der_relet_logs",|
|00000060| 22 69 64 22 3a 22 31 66 37 66 66 38 63 38 2d 33 |"id":"1f7ff8c8-3|
|00000070| 36 39 37 2d 34 33 37 31 2d 39 63 62 38 2d 39 39 |697-4371-9cb8-99|
|00000080| 31 33 34 30 31 36 63 31 39 65 22 7d 7d 2c 22 6f |134016c19e"}},"o|
|00000090| 70 22 3a 22 75 22 2c 22 74 73 5f 6d 73 22 3a 31 |p":"u","ts_ms":1|
|000000a0| 35 37 38 30 33 34 37 37 32 35 33 32 7d          |578034772532}   |
+--------+-------------------------------------------------+----------------+

Correctly use stdout vs stderr for piping purposes

Problem
It is desirable to be able to pipe output from pulsarctl to other commands, for example to jq (assuming #126):

$ pulsarctl topics list public/default -o json | jq

This doesn't work when an error occurs, because the error text is written to stdout rather than to stderr.

Suggested Improvement
Basically, error output should be directed to stderr. There appears to be two styles in the code.

Style 1:

if err != nil {
	cmdutils.PrintError(vc.Command.OutOrStderr(), err)
	return err
}

I believe there's two issues here, one is that vc.Command.OutOrStderr() should be vc.Command.ErrOrStderr() (note that some tests call rootCmd.SetOut and may require adjustment). The other issue is that it returns the err, causing the global handler to be invoked, but should simply return nil since the error was handled (though the exit code should still be 1).

Style 2: return err from the command to the global error handler (defined in verb.go):

var defaultExecErrorHandler = func(err error) {
	logger.Critical("%s\n", err.Error())
	os.Exit(1)
}

The issue here is that the kris-nova/logger package writes only to stdout. A different package may be needed. Note that logger has a colorizer and a timestamp feature to be taken into account.

Topics operates should redirect to the owner broker

The following operations should redirect to the owner broker, otherwise the pulsarctl will block and can't return any information.

1. compact-status
2. internal-info
3. internal-stats
4. last-message-id
5. offload
6. offload-status
7. stats
8. unload

Add a plugin framework for building plugins/extensions to pulsarctl

Output format

Maybe I missing something, but is that possible to omit table view and get values directly?
Like instead of

+------------------------------+
|        SUBSCRIPTIONS         |
+------------------------------+
| signup                       |

just output directly:

signup

?

As for now its hard to create any chain of calls or use the output in scripts

[pulsarctl] [schemas] `Extract ` command is not supported

In CmdSchemas.java, code logic as follows:

File file  = new File(jarFilePath);
            ClassLoader cl = new URLClassLoader(new URL[]{ file.toURI().toURL() });
            Class cls = cl.loadClass(className);

            PostSchemaPayload input = new PostSchemaPayload();
            SchemaDefinition<Object> schemaDefinition =
                    SchemaDefinition.builder()
                                    .withPojo(cls)
                                    .withAlwaysAllowNull(alwaysAllowNull)
                                    .build();

but in go language, no ClassLoader and pojo, so we can't use go to parse the jar generated in java.

Add docs of enable pulsarctl autocompletion

Motivation

Currently, the docs of enable pulsarctl autocompletion missing.

[context] Add default context behavior

Motivation

• Automatically use a context when it is added.
• Set a tenant and/or namespace which would be used by default in various commands, refer to: #119 (comment)

Support plugin list for listing all available plugins

Motivation

After we support the plugin handler, we need a command to list all available plugins.

Pulsarctl context can not switch to the another cluster

Reproduce:

1. Start two Pulsar standalone.
2. Create two contexts for the two Pulsar standalone.
3. Use the first context to connect the pulsar and list the topics under the default namespace, it lists the topics successfully.
4. Use the second context to connect the pulsar and list the topics on another pulsar standalone. It will lists the first cluster topics.

Release process should use tag name to checkout and release

We should support the release process to use the tag name to release a specified version.

Add Apache License Header and enforce license header check in CI

[Discuss] pulsarctl should use "--non-partitioned" instead of "partitioned" for topic as default type

pulsarctl won't delete a normal topic unless given --non-partioned flag:

$ ./pulsarctl topic delete persistent://public/default/temp
[✖]  code: 404 reason: Partitioned topic does not exist
$ ./pulsarctl topic delete --non-partitioned persistent://public/default/temp
Delete topic persistent://public/default/temp successfully

However, pulsar-admin does not need to be told this.

$ apache-pulsar-2.4.1/bin/pulsar-admin topics delete persistent://public/default/temp
$ 

Add executables for other platforms

Go Support cross-compilation, we can use this feature to provide a richer executable file for different platforms.

For examples:

CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build main.go
CGO_ENABLED=0 GOOS=windows GOARCH=amd64 go build main.go
CGO_ENABLED=0 GOOS=darwin GOARCH=amd64 go build main.go

investigate how pulsarctl can integrate with gcloud

Motivation

investigate how pulsarctl can integrate with different cloud providers and come up with a design.

There are some tests failed by unknown reason

Motivation

There are some tests failed by unknown reasons in PR #229 #231.

Support different ways to dump the result

improve the CLI by adding an option -o or --output to dump the results, either PLAIN, TABLE, or JSON.

Enable RBAC for Pulsar Helm Chart

Motivation

enable k8s RBAC for Pulsar Helm Chart.

[Question]Use the env for token

I have to add the --token paramter when i use pulsarctl in the auth request, Can i use the env for token?

Please remind me if I missed something!

Thank!

functions querystate generates invalid API request when `--key` not provided

If you run the pulsarctl functions querystate command without the --key option, you get:

$ ./pulsarctl functions querystate --name womble
error: Unknown pulsar error

Digging with tcpdump on port 3080, the HTTP exchange can be seen as:

GET /admin/v3/functions/public/default/womble/state HTTP/1.1
Host: localhost:8080
User-Agent: Pulsarctl-Go-v0.1.0
Accept: application/json
Content-Type: application/json
Accept-Encoding: gzip

HTTP/1.1 404 Not Found
Date: Sat, 02 Nov 2019 08:27:18 GMT
Cache-Control: must-revalidate,no-cache,no-store
Content-Length: 0
Server: Jetty(9.4.12.v20180830)

That is: pulsarctl is making a HTTP request to a URL which is not defined as part of the REST API.

ISTM that the --key argument should be made mandatory.

To be fair: the --key argument is optional in pulsar-admin too; but in that case pulsar-admin crashes with a null pointer exception before it sends any request to the REST API.

investigate how pulsarctl can install a helm chart

Motivation

investigate how pulsarctl can install a helm chart.

adding a new "scope" section in the command description to indicate which scope that this command applies.

#44

for topic related commands, I would suggest adding a new "scope" section in the command description to indicate which scope that this command applies.

SCOPE:
     non-partitioned topic, a partition of a partitioned topic 

Unable to manage namespaces with slashes

On a default standalone installation, I find there is a "sample" tenant with a namespace "sample/standalone/ns1" in it, but pulsarctl rejects this as invalid when trying to list topics.

ubuntu@ldex-pulsar:~$ ./pulsarctl tenants list
+-------------+
| TENANT NAME |
+-------------+
| public      |
| sample      |
+-------------+
ubuntu@ldex-pulsar:~$ ./pulsarctl namespaces list public
+------------------+
|  NAMESPACE NAME  |
+------------------+
| public/default   |
| public/functions |
+------------------+
ubuntu@ldex-pulsar:~$ ./pulsarctl namespaces list sample
+-----------------------+
|    NAMESPACE NAME     |
+-----------------------+
| sample/standalone/ns1 |
+-----------------------+
ubuntu@ldex-pulsar:~$ ./pulsarctl topics list sample/standalone/ns1
[✖]  The complete name of namespace is invalid. complete name : [sample/standalone/ns1]
ubuntu@ldex-pulsar:~$ ./pulsarctl topics list persistent://sample/standalone/ns1
[✖]  The complete name of namespace is invalid. complete name : [persistent://sample/standalone/ns1]
ubuntu@ldex-pulsar:~$ ./pulsarctl topics list persistent://sample%2Fstandalone%2Fns1
[✖]  The complete name of namespace is invalid. complete name : [persistent://sample%!F(MISSING)standalone%!F(MISSING)ns1]

But pulsar-admin gives no error:

ubuntu@ldex-pulsar:~$ apache-pulsar-2.4.1/bin/pulsar-admin topics list sample/standalone/ns1
ubuntu@ldex-pulsar:~$

Note: this "sample/standalone/ns1" namespace is mentioned in old pulsar docs but not in current ones, except in sample output of "show schemas in pulsar" in the SQL getting started doc.

HTTP info

tcpdump for pulsarctl (any of the above variants) shows no activity on loopback port 8080 - it seems the request was rejected lcoally.

tcpdump for "pulsar-admin topics list sample/standalone/ns1" shows:

>>>[new conn: port 54116]
GET /admin/persistent/sample/standalone/ns1 HTTP/1.1
Accept: application/json
host: localhost:8080
user-agent: Pulsar-Java-v2.4.1

<<<[54116]
...
Content-Type: application/json
Content-Length: 2
Server: Jetty(9.4.12.v20180830)

[]

>>[54116]
GET /admin/non-persistent/sample/standalone/ns1 HTTP/1.1
Accept: application/json
host: localhost:8080
user-agent: Pulsar-Java-v2.4.1

>>[new conn: port 60654]
GET /admin/non-persistent/sample/standalone/ns1/0x40000000_0x80000000 HTTP/1.1
Accept: application/json
host: ldex-pulsar.int.soundmouse.net:8080
user-agent: Pulsar-Java-v2.4.1

>>[new conn: port 60652]
GET /admin/non-persistent/sample/standalone/ns1/0xc0000000_0xffffffff HTTP/1.1
Accept: application/json
host: ldex-pulsar.int.soundmouse.net:8080
user-agent: Pulsar-Java-v2.4.1

>>[new conn: port 60656]
GET /admin/non-persistent/sample/standalone/ns1/0x80000000_0xc0000000 HTTP/1.1
Accept: application/json
host: ldex-pulsar.int.soundmouse.net:8080
user-agent: Pulsar-Java-v2.4.1

>>[new conn: port 60658]
GET /admin/non-persistent/sample/standalone/ns1/0x00000000_0x40000000 HTTP/1.1
Accept: application/json
host: ldex-pulsar.int.soundmouse.net:8080
user-agent: Pulsar-Java-v2.4.1

<<[60658]
HTTP/1.1 204 No Content
Date: Mon, 18 Nov 2019 11:58:54 GMT
broker-address: ldex-pulsar.int.soundmouse.net
Server: Jetty(9.4.12.v20180830)

<<[60654]
HTTP/1.1 204 No Content
Date: Mon, 18 Nov 2019 11:58:54 GMT
broker-address: ldex-pulsar.int.soundmouse.net
Server: Jetty(9.4.12.v20180830)

<<[60656]
HTTP/1.1 204 No Content
Date: Mon, 18 Nov 2019 11:58:54 GMT
broker-address: ldex-pulsar.int.soundmouse.net
Server: Jetty(9.4.12.v20180830)

<<[60652]
HTTP/1.1 204 No Content
Date: Mon, 18 Nov 2019 11:58:54 GMT
broker-address: ldex-pulsar.int.soundmouse.net
Server: Jetty(9.4.12.v20180830)

<<[55116]
HTTP/1.1 200 OK
Date: Mon, 18 Nov 2019 11:58:54 GMT
Content-Type: application/json
Content-Length: 2
Server: Jetty(9.4.12.v20180830)

[]

tcpdump for "pulsarctl namespaces list sample" (which works) shows:

>>>
GET /admin/v2/namespaces/sample HTTP/1.1
Host: localhost:8080
User-Agent: None
Accept: application/json
Content-Type: application/json
Accept-Encoding: gzip

<<<
HTTP/1.1 200 OK
Date: Mon, 18 Nov 2019 12:06:51 GMT
Content-Type: application/json
Content-Length: 25
Server: Jetty(9.4.12.v20180830)

["sample/standalone/ns1"]

Add pulsarctl to homebrew

Motivation

Currently users have to download pulsarctl from github releases. It would be great if MacOS users can just install pulsarctl via brew install pulsarctl

[Bug] try to check if an interface is nil

The issue was introduced by #196

authProvider, err := auth.GetAuthProvider(config)
if authProvider != nil {
	fmt.Printf("Found Auth provider %T", authProvider)
	c.Client.HTTPClient.Transport = *authProvider
} else {
	fmt.Printf("No Auth Provider found")
}

The authProvider is an interface, but interfaces in Go contain both type and value. So you can have cases when —

• Type is null-able(like map, pointer etc) and value is nil
• Type itself is nil (of course value will be nil)

A nil will only work with option 2 above, as for option 1 a variable still have some type.

`pulsarctl functions-worker get-function-assignments` has incorrect output

Problem
I observe that the output from functions-worker get-function-assignments is incorrect; expected is the assignments, actual is the same output from functions-worker function-stats.

$ pulsarctl functions-worker get-function-assignments
[
  {
    "name": "public/default/test-functions-update-failure:0",
    "metrics": {
      "oneMin": {
        "receivedTotal": 0,
        "processedSuccessfullyTotal": 0,
        "systemExceptionsTotal": 0,
        "userExceptionsTotal": 0,
        "avgProcessLatency": 0
      },
      "lastInvocation": 0,
      "userMetrics": {},
      "receivedTotal": 0,
      "processedSuccessfullyTotal": 0,
      "systemExceptionsTotal": 0,
      "userExceptionsTotal": 0,
      "avgProcessLatency": 0
    }
  },
...

Suggested Improvement
It appears to be a copy-paste error. Simply update get_function_assignments.go to obtain the assignments.

Broken bash completion

• Type "./pulsarctl topics de"
• Hit Tab

Result:

$ ./pulsarctl topics de-su: aliashash[""]: bad array subscript

Add offload-policies commands to pulsarctl

• pulsarctl namespaces set-offload-policies [options]
• pulsarctl namespaces get-offload-policies [options]

Add bookkeeper client support

Motivation

I have submitted the pr about supporting bookie commands. Because the project structure makes a big change and there are too many conflicts, I need to refactor base on the master. I will close #83 #86 #89 and reopen a new pr.

Issue

Currently, Pulsarctl only support sends a request to Pulsar service. So the HTTP client is bind with Pulsar client. We need to separate the HTTP client and the Pulsar Admin client. Then the HTTP client can be used in both Pulsar and BookKeeper clients.

I will send several pull requests to complete this work.

• Separate HTTP client and Pulsar Admin client. #132
• Add bookie commands
  • ledger #134
  • bookies #135
  • autorecovery #136

`pulsarctl namespaces policies` fails for certain auth policies

Problem
Pulsar server version: 2.4.1

The client is unable to display certain namespace policies. Given this policy:

  "auth_policies": {
    "namespace_auth": {},
    "destination_auth": {
      "persistent://public/default/test-grant-permission-non-partitioned-topic": {
        "grant-non-partitioned-role": [
          "produce"
        ]
      },
      "persistent://public/default/test-grant-permission-partitioned-topic": {
        "grant-partitioned-role": [
          "consume"
        ]
      },
      "persistent://public/default/test-revoke-partitioned-topic": {},
      "persistent://public/default/test-revoke-non-partitioned-topic": {}
    },
    "subscription_auth_roles": {}
  },

Results in the following error:

$ pulsarctl namespaces policies public/default
[✖]  json: cannot unmarshal array into Go struct field AuthPolicies.auth_policies.destination_auth of type common.AuthAction
exit status 1

Suggested Improvement
The common.AuthPolicies struct seems not to expect an array of AuthAction strings (i.e. produce, consume, functions).

[BUG] [pulasr] [Topic] non-persistent topic list get persistent topics result

./bin/pulsarctl topics list rental_car/promotions

+--------------------------------------------+---------------+
|                 TOPIC NAME                 | PARTITIONED ? |
+--------------------------------------------+---------------+
| persistent://rental_car/promotions/changed | N             |
| persistent://rental_car/promotions/created | N             |
| persistent://rental_car/promotions/changed | N             |
| persistent://rental_car/promotions/created | N             |
+--------------------------------------------+---------------+

after try these rest apis

/admin/v2/persistent/:tenant/:namespace
/admin/v2/non-persistent/:tenant/:namespace

get same response cause the problem.

Add test for the pull request #165

[pulsarctl] [get] checkSuccessful response status code error

pulsar admin ./bin/pulsar-admin schemas get aaa as follows:

12:56:27.658 [ForkJoinPool.commonPool-worker-10] INFO  org.eclipse.jetty.server.RequestLog - 127.0.0.1 - - [11/九月/2019:12:56:27 +0800] "GET /admin/v2/schemas/public/default/aaa/schema HTTP/1.1" 404 0 "-" "Pulsar-Java-v2.4.0" 15

client output as follows:

HTTP 404 Not Found

Reason: HTTP 404 Not Found

pulsarctl ./pulsarctl schemas get aaa as follows:

12:57:17.367 [ForkJoinPool.commonPool-worker-3] INFO  org.eclipse.jetty.server.RequestLog - 127.0.0.1 - - [11/九月/2019:12:57:17 +0800] "GET /admin/v2/schemas/public/default/aaa/schema HTTP/1.1" 404 0 "-" "pulsarctl (go)" 4

client output as follows:

[✖]  code: 206 reason:

We can see that when requesting a non-existent topic name, the broker's response status code is 404. But pulsarctl received the status code is 206.

Reason as follows:

        var e Error
	body, err := ioutil.ReadAll(resp.Body)
	if err != nil {
		e.Reason = err.Error()
		e.Code = resp.StatusCode
		return e
	}

	jsonErr := json.Unmarshal(body, &e)

	if jsonErr != nil {
		e.Code = http.StatusPartialContent
	} 

Outdated docs for namespaces set-retention command

Hello. 👋

I'm currently trying to use pulsarctl (0.4.1) to do some operations on a Pulsar 2.6.0 cluster.

The documentation for the namespaces set-retention command seems wrong or at least is missing some details.

The two examples on the documentation are the following:

pulsarctl namespaces set-retention tenant/namespace --time 100m

pulsarctl namespaces set-retention tenant/namespace --size 1G

They are wrong because both --time and --size options are required. So you get the following result in both cases:

Error: required flag(s) "time" not set
Usage: pulsarctl namespaces set-retention [flags]

Examples:
    #Set the retention policy for a namespace
    pulsarctl namespaces set-retention tenant/namespace --time 100m

    #Set the retention policy for a namespace
    pulsarctl namespaces set-retention tenant/namespace --size 1G

[...]

Also, it says to use -1 to infinite time or size retention.

However, when using -1 for the --time it fails with the following error:

$ pulsarctl --admin-service-url=http://localhost:8080 namespaces set-retention public/default --size 1G --time -1                                          
[✖]  invalid time '-'

Same result when adding quotes to "-1".

What's the proper way to specify infinite time retention?

pulsarctl functions update behaves differently to pulsar-admin

With pulsarctl:

$ ./pulsarctl functions update --name womble --window-length-count 5 --sliding-interval-count 3
error: Window length is not specified
[✖]  code: 400 reason: Window length is not specified

With pulsar-admin:

$ apache-pulsar-2.4.1/bin/pulsar-admin functions update --name womble --window-length-count 5 --sliding-interval-count 3
There is currently no support windowing in python

Reason: There is currently no support windowing in python

Now, these commands must be sending different requests.

Using tcpdump, I can see that pulsarctl is sending:

PUT /admin/v3/functions/public/default/womble HTTP/1.1
Host: localhost:8080
User-Agent: Pulsarctl-Go-v0.1.0
Content-Length: 1195
Content-Type: multipart/form-data; boundary=9a93b1cbcc8d9ebded4b798bfc634b747b683591ccc23bc34fd513fb95b0
Accept-Encoding: gzip

--9a93b1cbcc8d9ebded4b798bfc634b747b683591ccc23bc34fd513fb95b0
Content-Disposition: form-data; name="functionConfig"
Content-Type: application/json

{"timeoutMs":null,"topicsPattern":null,"cleanupSubscription":false,"retainOrdering":false,"autoAck":true,"parallelism":1,"maxMessageRetries":0,"output":"","outputSerdeClassName":"","logTopic":"","processingGuarantees":"","outputSchemaType":"","runtime":"","deadLetterTopic":"","subName":"","fqfn":"","jar":"","py":"","go":"","runtimeFlags":"","tenant":"public","namespace":"default","name":"womble","className":"","resources":null,"windowConfig":{"WindowLengthCount":5,"WindowLengthDurationMs":0,"SlidingIntervalCount":3,"SlidingIntervalDurationMs":0,"LateDataTopic":"","MaxLagMs":0,"WatermarkEmitIntervalMs":0,"TimestampExtractorClassName":"","ActualWindowFunctionClassName":""},"inputs":null,"userConfig":{},"customSerdeInputs":null,"customSchemaInputs":null,"inputSpecs":null,"secrets":null}
--9a93b1cbcc8d9ebded4b798bfc634b747b683591ccc23bc34fd513fb95b0
Content-Disposition: form-data; name="updateOptions"
Content-Type: application/json

{"UpdateAuthData":false}
--9a93b1cbcc8d9ebded4b798bfc634b747b683591ccc23bc34fd513fb95b0--

Response:

HTTP/1.1 400 Bad Request
Date: Thu, 31 Oct 2019 13:55:04 GMT
Content-Type: application/json
Content-Length: 43
Server: Jetty(9.4.12.v20180830)

{"reason":"Window length is not specified"}

pulsar-admin is sending:

PUT /admin/v3/functions/public/default/womble HTTP/1.1
content-length: 507
content-type: multipart/form-data; boundary=7BWo8szw5xOPw3kNV7b4zwG1y7eXVudA-
host: localhost:8080
accept: */*
user-agent: Pulsar-Java-v2.4.1

--7BWo8szw5xOPw3kNV7b4zwG1y7eXVudA-
Content-Disposition: form-data; name="functionConfig"
Content-Type: application/json; charset=UTF-8

{"tenant":"public","namespace":"default","name":"womble","outputSchemaType":"","userConfig":{},"windowConfig":{"windowLengthCount":5,"slidingIntervalCount":3}}
--7BWo8szw5xOPw3kNV7b4zwG1y7eXVudA-
Content-Disposition: form-data; name="updateOptions"
Content-Type: application/json; charset=UTF-8

{"updateAuthData":false}

Response:

HTTP/1.1 400 Bad Request
Date: Thu, 31 Oct 2019 13:55:59 GMT
Content-Type: application/json
Content-Length: 62
Server: Jetty(9.4.12.v20180830)

{"reason":"There is currently no support windowing in python"}

I think there are two problems.

1. pulsar-admin is doing the right thing: only sending those things which I have asked to update. And pulsarctl is doing the wrong thing: overwriting all settings with defaults, even the ones I didn't ask to change (and is therefore very dangerous)

2. pulsarctl is sending attributes with wrong initial capitalisation, e.g. "WindowLengthCount" whereas pulsaradmin is sending "windowLengthCount".

[pulsarctl] [Schemas] KeyValue schema is not supported

Now KeyValue schema is not supported in pulsarctl

Update the commands scope after the https://github.com/apache/pulsar/issues/5412 resolved

As the title, we need to update the command scope after the issue resolved.

`pulsarctl schemas get --version 0` produces incorrect results

Problem
The schemas get command has two variants: one without a version specified, and one with a version specified. The former variant obtains the latest version, and produces output that includes the version number. The latter variant obtains the requested version, and produces output that doesn't include the version number.

The problem is that the specific version number of 0 is treated as a sentinel value and hence behaves as the first variant.

For example, the following two commands should produce similar output (but don't):

$ pulsarctl schemas get public/default/topic-27 --version 0 -o json
{
  "version": 0,
  "schemaInfo": {
    "name": "topic-27",
    "schema": "",
    "type": "INT32",
    "properties": {}
  }
}

$ pulsarctl schemas get public/default/topic-27 --version 1 -o json
{
  "name": "topic-27",
  "schema": "",
  "type": "INT32",
  "properties": {}
}

Re-enable the tests

#59

Crash when using client certifiactes

Using the May 17th release. Tried with key in pk8 and regular pkcs12

./pulsarctl -s https://pulsar-admin-proxy:8443/ --auth-params "tlsCertFile:/pulsar/pulsar-admin.pem,tlsKeyFile:/pulsar/pulsar-admin.key.pem" --auth-plugin "org.apache.pulsar.client.impl.auth.AuthenticationTls" topics list tenant/ns

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x6c2e66]

goroutine 1 [running]:
github.com/streamnative/pulsarctl/pkg/auth.(*TLSAuthProvider).configTLS(0xc00032de00, 0xc00032de00, 0x2)
/home/runner/work/streamnative-ci/streamnative-ci/pulsarctl/pkg/auth/tls.go:86 +0x126
github.com/streamnative/pulsarctl/pkg/auth.NewAuthenticationTLS(0x7ffeea46cd36, 0x18, 0x7ffeea46cd5a, 0x1c, 0xa92ec0, 0xe08c00, 0xc0003431a0, 0x2, 0x2)
/home/runner/work/streamnative-ci/streamnative-ci/pulsarctl/pkg/auth/tls.go:44 +0x9c
github.com/streamnative/pulsarctl/pkg/auth.NewAuthenticationTLSFromAuthParams(0x7ffeea46cd2a, 0x4c, 0xa92ec0, 0xe08c00, 0x2, 0xc0003599f0, 0x81b06a)
/home/runner/work/streamnative-ci/streamnative-ci/pulsarctl/pkg/auth/tls.go:64 +0x1b0
github.com/streamnative/pulsarctl/pkg/auth.GetAuthProvider(0xc000359ae0, 0xc00032ddd0, 0xc000359a00, 0x1)
/home/runner/work/streamnative-ci/streamnative-ci/pulsarctl/pkg/auth/auth_provider.go:47 +0x348
github.com/streamnative/pulsarctl/pkg/pulsar.New(0xc000359ae0, 0x0, 0x7ffeea46cdcf, 0x4, 0xc000343001)
/home/runner/work/streamnative-ci/streamnative-ci/pulsarctl/pkg/pulsar/admin.go:77 +0x11e
github.com/streamnative/pulsarctl/pkg/cmdutils.(*ClusterConfig).Client(0xc00001ee60, 0x1, 0xc000343080, 0x0)
/home/runner/work/streamnative-ci/streamnative-ci/pulsarctl/pkg/cmdutils/config.go:194 +0xfa
github.com/streamnative/pulsarctl/pkg/cmdutils.NewPulsarClient(...)
/home/runner/work/streamnative-ci/streamnative-ci/pulsarctl/pkg/cmdutils/cmdutils.go:87
github.com/streamnative/pulsarctl/pkg/ctl/topic.doListTopics(0xc00008d720, 0x7ffeea46cdc6, 0xd)
/home/runner/work/streamnative-ci/streamnative-ci/pulsarctl/pkg/ctl/topic/list.go:82 +0x98
github.com/streamnative/pulsarctl/pkg/ctl/topic.ListTopicsCmd.func1(0xc000210f00, 0xc000359cf8)
/home/runner/work/streamnative-ci/streamnative-ci/pulsarctl/pkg/ctl/topic/list.go:65 +0x2a
github.com/streamnative/pulsarctl/pkg/cmdutils.run(0xc0001ecd00)
/home/runner/work/streamnative-ci/streamnative-ci/pulsarctl/pkg/cmdutils/verb.go:100 +0x27
github.com/streamnative/pulsarctl/pkg/cmdutils.(*VerbCmd).SetRunFuncWithNameArg.func1(0xc00021aa00, 0xc00031a310, 0x1, 0x7)
/home/runner/work/streamnative-ci/streamnative-ci/pulsarctl/pkg/cmdutils/verb.go:75 +0xb3
github.com/spf13/cobra.(*Command).execute(0xc00021aa00, 0xc0001a0c00, 0x7, 0xc, 0xc00021aa00, 0xc0001a0c00)
/home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:830 +0x29d
github.com/spf13/cobra.(*Command).ExecuteC(0xc0000ad680, 0xc000342e80, 0xc00001e0b0, 0x9)
/home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:914 +0x2fb
github.com/spf13/cobra.(*Command).Execute(...)
/home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:864
main.main()
/home/runner/work/streamnative-ci/streamnative-ci/pulsarctl/main.go:41 +0x1b8

Add `context` resource to pulsarctl

Problem

Currently users have to specify the cluster url when running commands. It will be great if pulsarctl provides a context resource, so pulsarctl can switch the default clusters between contexts.

pulsarctl MVP

• General
  • Support resource-verb style commands
    （- [ ] Support verb-resource style commands）
  • Security #9 #20 #91
  • Auto-Completion #3
  • Enable CI #5 #6 #112
  • Add and check License Header #69
  • Go lint #78
• Pulsar
  • Clusters #4 #8 #11 #13 #14 #17 #18 #19 #21 #22 #23 #24 #36
  • Tenants #28
  • Namespaces #57 #61 #66 #64 #67 #68 #71
  • Topics . #35 #48 #51 #56 #58 #76 #75 #74 #73
  • Schemas #52
  • Functions #29 #30 #31 #32 #26 #25 #15
  • Sinks #47
  • Sources #45
  • broker #87
  • broker-stats #84
  • resource-quotas #90
  • ns-isolation-policy #88
  • functions-worker #92
• BookKeeper
  • bookies #86
  • ledger #83
  • auto-recovery #89
• Documentation
  • How to add a new command #79
  • Command description convention #77
  • Overview of pulsarctl #108

Improve tab-alignment of help text

Problem
The help text includes a list of sub-commands for a given command, and attempts to align the short descriptions. The output is imperfect for certain lists containing sub-commands longer than 10 characters, e.g. namespaces:

Usage: pulsarctl namespaces [flags]

Commands:
  clear-backlog  Clear backlog for all topics of a namespace
  clear-offload-deletion-lag  Clear offload deletion lag of a namespace
  create      Create a new namespace
  delete      Delete a namespace. The namespace needs to be empty
  delete-anti-affinity-group  Delete an anti-affinity group of a namespace
  get-anti-affinity-group  Get the anti-affinity group of a namespace
  get-anti-affinity-namespaces  Get the list of namespaces in the same anti-affinity group.

Suggested Improvement
Use text/tabwriter package (ref) which implements the elastic tabstops technique.

Show help text when positional arguments aren't supplied

Problem
When a command expects positional arguments but the user didn't supply any, the following error message is shown:

$ go run ./main.go context set
[✖]  the context name is not specified or the context name is specified more than one

In this situation, it would be useful to show the help text, as would occur if a required flag was't supplied or wasn't parseable.

Suggestion
Separate the validation from the execution, to be able to distinguish whether the help text should be shown.

Clarify args error output for all commands

Creating testcontainer framework to test pulsarctl

Problem deploying python function

Attempting to use pulsarctl to deploy a python function fails:

$ cat func1.py
from pulsar import Function

class FirstFunction(Function):
    def process(self, item, context):
        log = context.get_logger()
        log.info("Got %r with properties %r" % (item, context.get_message_properties()))

$ ./pulsarctl functions create --name womble --inputs my-topic --py func1.py
exists func1.py 222 2019-10-31 10:31:42.007953886 +0000 UTC
USED FOR:
    This command is used for creating a new Pulsar Function in cluster mode.
<<< snip pages of usage information >>>
Use 'pulsarctl functions create [command] --help' for more information about a command.

[✖]  no Function Classname specified

$ ./pulsarctl functions create --name womble --inputs my-topic --py func1.py --classname FirstFunction
exists func1.py 222 2019-10-31 10:31:42.007953886 +0000 UTC
error: Function class FirstFunction must be in class path
[✖]  code: 400 reason: Function class FirstFunction must be in class path

But I can successfully deploy it using pulsar-admin:

$ apache-pulsar-2.4.1/bin/pulsar-admin functions create --name womble --inputs my-topic --py /home/ubuntu/func1.py --classname FirstFunction
"Created successfully"

(I need to give an absolute path to the python source file; it looks like pulsar-admin changes the working directory when it runs)

Then pulsarctl can see it:

$ ./pulsarctl functions list
+----------------------+
| PULSAR FUNCTION NAME |
+----------------------+
| womble               |
+----------------------+

Flaky tests list

The issue will track all flaky tests of pulsarctl

• TestStatusSink

Add `policy` resources

Motivation

Currently we have some many commands under namespace resource. It introduces a few inconveniences.

• We have too many getters and setters. It is really hard to people to find the right command, especially these policy operations are coupled with CRUD operations on namespaces.
• Each time we introduce a name field in the policy. We have to add new getter and setter in the command.
• We are introduce a topic-level policy in Pulsar. It means that we might need to add the operations under topic resource as well. It results in duplicated code.

Proposal

I am proposing introducing a policy resource for handling operations to policies at both namespace or topic level.

The proposed command syntax is described as followings:

pulsarctl policies [policy-property] [action] [options] [resource-entity]

• policy-property is the property within the policy, such as backlog-quotas, subscribe-rate.
• action is the operation can be applied on the policy-property, e.g. get and set.
• options are the options for the action.
• resource-entity can be a namespace or a topic (if we support topic-level policy)

The examples of the commands using this syntax are listed as follows:

• get backlog quotas from a namespace

pulsarctl policies backlog-quota get tenant/ns

• get backlog quotas from a topic

pulsarctl policies backlog-quota get tenant/ns/topic