emqx / emqx-management

EMQX Management API

emqx-management

EMQX Management API

How to Design RESTful API?

http://restful-api-design.readthedocs.io/en/latest/scope.html

emqx-management's Issues

HTTPs Settings are not working

OS: Ubuntu 18
EMQX:4.2

So I set the https settings in the emqx_management.conf file like this:

management.listener.https = 8081
management.listener.https.acceptors = 2
management.listener.https.max_clients = 512
management.listener.https.backlog = 512
management.listener.https.send_timeout = 15s
management.listener.https.send_timeout_close = on
management.listener.https.certfile = etc/certs/cert.pem
management.listener.https.keyfile = etc/certs/privkey.pem
management.listener.https.cacertfile = etc/certs/chain.pem

This is the same setting I am using for the dashboard config. It works fine in the dashboard but does not work in the the management.

When I send a GET request to https://url:8081/api/v4/, the server does not give any response. In the dashboard it shows the listener is active.

Should add base64 support to mqtt_publish/mqtt_publish_batch api?

The mqtt_publish/mqtt_publish_batch api can only handle string data currently, but sometime it is necessary to publish payload such as protobuf.
Is it possible to add a 'payload_encoding' field to the pub api‘s params like Rabbitmq?

Error kicking a connection using REST api

I'm having problems kicking an active connection using the REST api. The API seems to work fine for querying data such as active connections and the like, but fails when using the DELETE api/v3/connections/${clientid} endpoint. I am running v. 3.0 on a stock ubuntu 18.04.1 LTS image.

I get the following error on the console:

curl -v -X DELETE --basic -u <appid>:<secret> -k http://myserver:8080/api/v3/connections/[email protected]%7C12345

2019-01-30 16:10:44.619 [error] DELETE /api/v3/connections/[email protected]%7C12345 error: function_clause, stacktrace:
[{emqx_mgmt,return,
            [{error,not_found}],
            [{file,"src/emqx_mgmt.erl"},{line,454}]},
 {minirest_handler,dispatch,3,[{file,"src/minirest_handler.erl"},{line,55}]},
 {minirest,handle_request,2,[{file,"src/minirest.erl"},{line,66}]},
 {minirest,init,2,[{file,"src/minirest.erl"},{line,51}]},
 {cowboy_handler,execute,2,[{file,"src/cowboy_handler.erl"},{line,37}]},
 {cowboy_stream_h,execute,3,[{file,"src/cowboy_stream_h.erl"},{line,274}]},
 {cowboy_stream_h,request_process,3,
                  [{file,"src/cowboy_stream_h.erl"},{line,252}]},
 {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]

A 500 error is returned:

*   Trying <myip>...
* TCP_NODELAY set
* Connected to <myserver> (<serverip>) port 8080 (#0)
* Server auth using Basic with user '<appid>'
> DELETE /api/v3/connections/[email protected]%7C12345 HTTP/1.1
> Host: <myserver>:8080
> Authorization: Basic <secret>
> User-Agent: curl/7.54.0
> Accept: */*
> 
< HTTP/1.1 500 Internal Server Error
< content-length: 14
< content-type: text/plain
< date: Wed, 30 Jan 2019 16:23:41 GMT
< server: Cowboy
< 
* Connection #0 to host <myserver> left intact

The connection for [email protected]|12345 exists and can be queried by:

curl -v -X GET --basic -u <appid>:<secret> -k http://myserver:8080/api/v3/connections/[email protected]%7C12345

{  
   "code":0,
   "data":[  
      {  
         "clean_start":true,
         "client_id":"[email protected]|12345",
         "connected_at":"2019-01-30 16:07:54",
         "heap_size":610,
         "ipaddress":"<ipaddy>",
         "is_bridge":false,
         "is_super":false,
         "keepalive":60,
         "mailbox_len":0,
         "mountpoint":"undefined",
         "node":"[email protected]",
         "peercert":"undefined",
         "port":18579,
         "proto_name":"MQTT",
         "proto_ver":4,
         "recv_cnt":17,
         "recv_msg":0,
         "recv_oct":3120,
         "recv_pkt":17,
         "reductions":51396,
         "send_cnt":17,
         "send_msg":0,
         "send_oct":39,
         "send_pkt":17,
         "username":"<username>",
         "zone":"external"
      }
   ]
}

Default `http.acceptors` is too low

The default value of management.listener.http.acceptors = 2(defined in emqx_management.conf file) is way too low according to the Ranch doc : https://ninenines.eu/docs/en/ranch/2.0/guide/internals/#_number_of_acceptors

This default value can affects the concurrency of EMQx management API

can't create app in dashboard

version: 4.0.2
platform: linux & macos

after i clicked submit, display create success. However, nothing will be list in the page.

I checked result with emqx_ctl mgmt list. It's created.

So, the real problem is that dashboard can't display app list.

How to use management.application.default_secrect?

I'm using the docker image of emqx to setup my broker and I wanted to use default_secret feature.

How I do see if the default secret is added? And what's the application id for the default secret?

Maybe this is a bug?

Today I ready to add a UDP listener in emqx,When I add it to esockd , it work normal but detected an error in "managerment listener". My code in here:
This code used to support raw udp:

start_urap_listener()->
    Opts = [{udp_options, [binary, {reuseaddr, true}]}],
    MFA = {emqx_urap_connection, start_link, []},
    esockd:open_udp('urap:udp', 1885, Opts, MFA).

This callback:

%%%-------------------------------------------------------------------
%%% @author admin
%%% @copyright (C) 2020, <COMPANY>
%%% @doc
%%%
%%% @end
%%% Created : 8. 4月 2020 11:20
%%%-------------------------------------------------------------------
-module(emqx_urap_connection).
-author("wwhai").

-behaviour(gen_server).

%% API
-export([start_link/2, init/2]).

%% gen_server callbacks
-export([init/1, handle_call/3, handle_cast/2, handle_info/2, terminate/2,
  code_change/3]).

-define(SERVER, ?MODULE).
-define(UDP_OPTS, [binary, {reuseaddr, true}]).

-record(emqx_urap_state, {}).

%%%===================================================================
%%% API
% %%%===================================================================
% start(Port) ->
%   Opts = [{udp_options, [binary, {reuseaddr, true}]}],
%   MFA = {?MODULE, start_link, []},
%   esockd:open_udp('urap:udp', Port, Opts, MFA).


start_link(Transport, Peer) ->
  {ok, proc_lib:spawn_link(?MODULE, init, [Transport, Peer])}.

init(_T, _P) ->
  gen_server:enter_loop(?MODULE, [], #emqx_urap_state{}).

init(_) -> ok.

%%
%% 消息接收处理
%%

handle_call(_Request, _From, State) ->
  {reply, ok, State}.

handle_info(Msg, State) ->
  io:format("MSG:~p~n", [Msg]),
  {noreply, State}.

terminate(_Reason, _State) ->
  ok.

code_change(_OldVsn, State, _Extra) ->
  {ok, State}.


handle_cast(_Msg, State) ->
  {noreply, State}.

It can work:

• Client send:
  
• Server received:
  

But front appear this:

And error log:

2020-04-08 21:14:46.394 [error] [esockd_udp]: Unexpected call: which_children
2020-04-08 21:14:46.394 [error] GET /api/v4/nodes/[email protected]/listeners error: function_clause, stacktrace:
[{esockd_listener_sup,'-child_pid/2-lc$^0/1-0-',
                      [ignore],
                      [{file,"/mnt/d/github/emqx-rel/_checkouts/esockd/src/esockd_listener_sup.erl"},
                       {line,87}]},
 {esockd_listener_sup,child_pid,2,
                      [{file,"/mnt/d/github/emqx-rel/_checkouts/esockd/src/esockd_listener_sup.erl"},
                       {line,87}]},
 {esockd,get_acceptors,1,
         [{file,"/mnt/d/github/emqx-rel/_checkouts/esockd/src/esockd.erl"},
          {line,194}]},
 {emqx_mgmt,'-list_listeners/1-fun-0-',1,
            [{file,"/mnt/d/github/emqx-rel/_checkouts/emqx_management/src/emqx_mgmt.erl"},
             {line,391}]},
 {lists,map,2,[{file,"lists.erl"},{line,1239}]},
 {lists,map,2,[{file,"lists.erl"},{line,1239}]},
 {emqx_mgmt,list_listeners,1,
            [{file,"/mnt/d/github/emqx-rel/_checkouts/emqx_management/src/emqx_mgmt.erl"},
             {line,388}]},
 {emqx_mgmt_api_listeners,list,2,
                          [{file,"/mnt/d/github/emqx-rel/_checkouts/emqx_management/src/emqx_mgmt_api_listeners.erl"},
                           {line,37}]}]

I checked code fond in here:

It looks like current version only filter tcpkeywords but no udp?

API issues in the cluster

When I cluster, and I use the API to access, I will get an error. The error is as follows：
2020-04-24 22:29:40.714 [error] GET /api/v4/clients error: {badmatch,
{error,
{badrpc,
{'EXIT',
{undef,
[{emqx_mgmt_api,do_query,
['[email protected]',
{emqx_channel_info,
[{{'$1',#{},'_'},[],['$1']}],
[]},
0,10001],
[]},
{rpc,'-handle_call_call/6-fun-0-',5,
[{file,"rpc.erl"},{line,197}]}]}}}}}, stacktrace:
[{emqx_mgmt_api,do_cluster_query,5,
[{file,"/home/emqx-rel/_checkouts/emqx_management/src/emqx_mgmt_api.erl"},
{line,129}]},
{emqx_mgmt_api,cluster_query,3,
[{file,"/home/emqx-rel/_checkouts/emqx_management/src/emqx_mgmt_api.erl"},
{line,118}]},
{emqx_mgmt_api_clients,list,2,
[{file,"/home/emqx-rel/_checkouts/emqx_management/src/emqx_mgmt_api_clients.erl"},
{line,112}]},
{minirest_handler,dispatch,2,
[{file,"/home/emqx-rel/_checkouts/minirest/src/minirest_handler.erl"},
{line,84}]},
{minirest,handle_request,2,
[{file,"/home/emqx-rel/_checkouts/minirest/src/minirest.erl"},
{line,94}]},
{minirest,init,2,
[{file,"/home/emqx-rel/_checkouts/minirest/src/minirest.erl"},
{line,86}]},
{cowboy_handler,execute,2,
[{file,"/home/emqx-rel/_checkouts/cowboy/src/cowboy_handler.erl"},
{line,41}]},
{cowboy_stream_h,execute,3,
[{file,"/home/emqx-rel/_checkouts/cowboy/src/cowboy_stream_h.erl"},
{line,320}]}]

When I access the API on the web page, an internal error is returned.
How can i solve this problem？？？

While compiling emqx-management receiving emqx_mgmt_api_alarms.erl failed

Container: docker
Erlang/OTP: 25.3
EMQX: 4.2.13

While compiling emqx-management receiving the following error. Could you please help resolve this issue.

#12 121.5 ===> Compiling emqx_rule_engine
#12 121.5 ===> compile options: {erl_opts, [compressed,warn_unused_vars,
#12 121.5 warn_shadow_vars,warn_unused_import,
#12 121.5 warn_obsolete_guard,compressed,
#12 121.5 {parse_transform}]}.
#12 121.5 ===> files to analyze ["/codes/_build/emqx/lib/emqx_rule_engine/src/emqx_rule_funcs.erl",
#12 121.5 "/codes/_build/emqx/lib/emqx_rule_engine/src/emqx_rule_id.erl",
#12 121.5 "/codes/_build/emqx/lib/emqx_rule_engine/src/emqx_rule_events.erl",
#12 121.5 "/codes/_build/emqx/lib/emqx_rule_engine/src/emqx_rule_sqlparser.erl",
#12 121.5 "/codes/_build/emqx/lib/emqx_rule_engine/src/emqx_rule_runtime.erl",
#12 121.5 "/codes/_build/emqx/lib/emqx_rule_engine/src/emqx_rule_sqltester.erl",
#12 121.5 "/codes/_build/emqx/lib/emqx_rule_engine/src/emqx_rule_registry.erl",
#12 121.5 "/codes/_build/emqx/lib/emqx_rule_engine/src/emqx_rule_engine_app.erl",
#12 121.5 "/codes/_build/emqx/lib/emqx_rule_engine/src/emqx_rule_engine_sup.erl",
#12 121.5 "/codes/_build/emqx/lib/emqx_rule_engine/src/emqx_rule_utils.erl",
#12 121.5 "/codes/_build/emqx/lib/emqx_rule_engine/src/emqx_rule_engine_cli.erl",
#12 121.5 "/codes/_build/emqx/lib/emqx_rule_engine/src/emqx_rule_validator.erl",
#12 121.5 "/codes/_build/emqx/lib/emqx_rule_engine/src/emqx_rule_maps.erl",
#12 121.5 "/codes/_build/emqx/lib/emqx_rule_engine/src/emqx_rule_engine_api.erl",
#12 121.5 "/codes/_build/emqx/lib/emqx_rule_engine/src/emqx_rule_actions.erl",
#12 121.5 "/codes/_build/emqx/lib/emqx_rule_engine/src/emqx_rule_metrics.erl",
#12 121.5 "/codes/_build/emqx/lib/emqx_rule_engine/src/emqx_rule_engine.erl"]
#12 121.6 ===> Compiled emqx_rule_actions.erl
#12 121.6 ===> Compiled emqx_rule_utils.erl
#12 121.6 ===> Compiled emqx_rule_events.erl
#12 121.7 ===> Compiled emqx_rule_funcs.erl
#12 121.7 ===> Compiled emqx_rule_engine_sup.erl
#12 121.7 ===> Compiled emqx_rule_engine_app.erl
#12 121.7 ===> Compiled emqx_rule_metrics.erl
#12 121.7 ===> Compiled emqx_rule_id.erl
#12 121.8 _build/emqx/lib/emqx_rule_engine/src/emqx_rule_runtime.erl:270:10: Warning: ambiguous call of overridden auto-imported BIF alias/1
#12 121.8 - use erlang:alias/1 or "-compile({no_auto_import,[alias/1]})." to resolve name clash
#12 121.8
#12 121.8 ===> Compiled emqx_rule_runtime.erl
#12 121.8 ===> Compiled emqx_rule_sqltester.erl
#12 121.8 ===> Compiled emqx_rule_registry.erl
#12 121.9 ===> Compiled emqx_rule_engine_cli.erl
#12 121.9 ===> Compiled emqx_rule_maps.erl
#12 121.9 ===> Compiled emqx_rule_validator.erl
#12 122.0 ===> Compiled emqx_rule_sqlparser.erl
#12 122.0 ===> Compiled emqx_rule_engine_api.erl
#12 122.1 ===> Compiled emqx_rule_engine.erl
#12 122.1 ===> Compiling emqx_reloader
#12 122.1 ===> compile options: {erl_opts, [compressed,warn_unused_vars,
#12 122.1 warn_shadow_vars,warn_unused_import,
#12 122.1 warn_obsolete_guard,debug_info,
#12 122.1 {parse_transform}]}.
#12 122.1 ===> files to analyze ["/codes/_build/emqx/lib/emqx_reloader/src/emqx_reloader_app.erl",
#12 122.1 "/codes/_build/emqx/lib/emqx_reloader/src/emqx_reloader_cli.erl",
#12 122.1 "/codes/_build/emqx/lib/emqx_reloader/src/emqx_reloader.erl",
#12 122.1 "/codes/_build/emqx/lib/emqx_reloader/src/emqx_reloader_sup.erl"]
#12 122.1 ===> Compiled emqx_reloader_sup.erl
#12 122.1 ===> Compiled emqx_reloader_cli.erl
#12 122.1 ===> Compiled emqx_reloader_app.erl
#12 122.1 ===> Compiled emqx_reloader.erl
#12 122.1 ===> Compiling emqx_management
#12 122.1 ===> compile options: {erl_opts, [compressed,warn_unused_vars,
#12 122.1 warn_shadow_vars,warn_unused_import,
#12 122.1 warn_obsolete_guard,debug_info,
#12 122.1 {parse_transform}]}.
#12 122.1 ===> files to analyze ["/codes/_build/emqx/lib/emqx_management/src/emqx_mgmt_util.erl",
#12 122.1 "/codes/_build/emqx/lib/emqx_management/src/emqx_mgmt_api_subscriptions.erl",
#12 122.1 "/codes/_build/emqx/lib/emqx_management/src/emqx_mgmt_api_listeners.erl",
#12 122.1 "/codes/_build/emqx/lib/emqx_management/src/emqx_mgmt_api.erl",
#12 122.1 "/codes/_build/emqx/lib/emqx_management/src/emqx_mgmt.erl",
#12 122.1 "/codes/_build/emqx/lib/emqx_management/src/emqx_mgmt_api_alarms.erl",
#12 122.1 "/codes/_build/emqx/lib/emqx_management/src/emqx_mgmt_api_metrics.erl",
#12 122.1 "/codes/_build/emqx/lib/emqx_management/src/emqx_mgmt_auth.erl",
#12 122.1 "/codes/_build/emqx/lib/emqx_management/src/emqx_mgmt_sup.erl",
#12 122.1 "/codes/_build/emqx/lib/emqx_management/src/emqx_mgmt_api_data.erl",
#12 122.1 "/codes/_build/emqx/lib/emqx_management/src/emqx_mgmt_api_nodes.erl",
#12 122.1 "/codes/_build/emqx/lib/emqx_management/src/emqx_mgmt_http.erl",
#12 122.1 "/codes/_build/emqx/lib/emqx_management/src/emqx_mgmt_api_banned.erl",
#12 122.1 "/codes/_build/emqx/lib/emqx_management/src/emqx_mgmt_app.erl",
#12 122.1 "/codes/_build/emqx/lib/emqx_management/src/emqx_mgmt_api_clients.erl",
#12 122.1 "/codes/_build/emqx/lib/emqx_management/src/emqx_mgmt_api_pubsub.erl",
#12 122.1 "/codes/_build/emqx/lib/emqx_management/src/emqx_mgmt_api_plugins.erl",
#12 122.1 "/codes/_build/emqx/lib/emqx_management/src/emqx_mgmt_api_stats.erl",
#12 122.1 "/codes/_build/emqx/lib/emqx_management/src/emqx_mgmt_api_apps.erl",
#12 122.1 "/codes/_build/emqx/lib/emqx_management/src/emqx_mgmt_api_brokers.erl",
#12 122.1 "/codes/_build/emqx/lib/emqx_management/src/emqx_mgmt_cli.erl",
#12 122.1 "/codes/_build/emqx/lib/emqx_management/src/emqx_mgmt_api_routes.erl"]
#12 122.1 ===> Compiled emqx_mgmt_api_metrics.erl
#12 122.1 ===> Compiled emqx_mgmt_auth.erl
#12 122.2 ===> Compiling _build/emqx/lib/emqx_management/src/emqx_mgmt_api_alarms.erl failed
#12 122.2 _build/emqx/lib/emqx_management/src/emqx_mgmt_api_alarms.erl:70:18: record alarm undefined
#12 122.2 _build/emqx/lib/emqx_management/src/emqx_mgmt_api_alarms.erl:75:30: variable 'Severity' is unbound
#12 122.2 _build/emqx/lib/emqx_management/src/emqx_mgmt_api_alarms.erl:76:47: variable 'Title' is unbound
#12 122.2 _build/emqx/lib/emqx_management/src/emqx_mgmt_api_alarms.erl:77:47: variable 'Summary' is unbound
#12 122.2 _build/emqx/lib/emqx_management/src/emqx_mgmt_api_alarms.erl:78:71: variable 'Ts' is unbound
#12 122.2 _build/emqx/lib/emqx_management/src/emqx_mgmt_api_alarms.erl:82:18: record alarm undefined
#12 122.2 _build/emqx/lib/emqx_management/src/emqx_mgmt_api_alarms.erl:87:34: variable 'Severity' is unbound
#12 122.2 _build/emqx/lib/emqx_management/src/emqx_mgmt_api_alarms.erl:88:51: variable 'Title' is unbound
#12 122.2 _build/emqx/lib/emqx_management/src/emqx_mgmt_api_alarms.erl:89:51: variable 'Summary' is unbound
#12 122.2 _build/emqx/lib/emqx_management/src/emqx_mgmt_api_alarms.erl:90:75: variable 'Ts' is unbound
#12 122.2
#12 122.2 make: *** [Makefile:51: emqx] Error 1

executor failed running [/bin/sh -c make ${EMQX_NAME} DEBUG=1]: exit code: 2

allow to listen management plugin only internally

The management plugin currently listens on port 8081 on all IPs. The only setting to control it is:

management.listener.http = 8081

This includes all the public IPs as well:

$ netstat -tunl
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp        0      0 0.0.0.0:8081            0.0.0.0:*               LISTEN     

While authentication is required to access the API I still think that the setting should allow to set the IP address to listen on as well, e.g.:

management.listener.http = 127.0.0.1:8081

This way the API could only listen on specific IP addresses, which would improve security.

Does the mgmt plugins support https?

emqx version: emqxv3.0rc3
I found https options in the plugins mgmt config file, but couldn't find the related docs in http://emqtt.com/docs/v3/rest.html. It seems emqx only support basic Authentication(with app id and secret) for REST API . So does the emqx-managerment plugins support the https access with certificates?

Wrong datatype for management.listener.https.verify?

When I uncomment the https listener and start emqx, the mgmt plugins doesn't work. The error msg is about the verify option.
2019-01-21 14:53:28.781 [error] Failed to start Ranch listener 'https:management' in ranch_ssl:listen([{next_protocols_advertised,[<<"h2">>,<<"http/1.1">>]},{alpn_preferred_protocols,[<<"h2">>,<<"http/1.1">>]},{port,8081},{backlog,512},{send_timeout,15000},{send_timeout_close,true},{nodelay,true},{keyfile,"etc/certs/client.key"},{certfile,"etc/certs/client.crt"},{cacertfile,"etc/certs/ca.crt"},{verify,"verify_peer"},{fail_if_no_peer_cert,true}]) for reason {options,^M {verify,^M "verify_peer"}} (unknown POSIX error)^M

After checked out the verify options in emqx.conf, I find that the datatype of verify option is atom in emqx.conf while it is string in mgmt. Does it cause the problem emqx failed to start mgmt plugin with https listener?

{mapping, "management.listener.https.verify", "emqx_management.listeners", [
  {datatype, string}
]}.

{mapping, "listener.ssl.$name.verify", "emqx.listeners", [
 {datatype, atom}
]}.

allow change of management.default_application.secret

the value of management.default_application.secret can not be changed currently. If changing it in the configuration file it does not get applied. The only working password remains "public".