Running the example(s) on microk8s about diy-kubernetes-examples

[michaelklishin]

Thank you for your time.

Team RabbitMQ uses GitHub issues for specific actionable items engineers can work on. This assumes that we have a certain amount of information to work with.

Getting all the details necessary to reproduce an issue, make a conclusion or even form a hypothesis about what's happening can take a fair amount of time. Our team is multiple orders of magnitude smaller than the RabbitMQ community. Please help others help you by providing a way to reproduce the behavior you're
observing and sharing as much relevant information as possible on the list:

• Server, client library and plugin (if applicable) versions used
• Server logs
• A code example or terminal transcript that can be used to reproduce
• Full exception stack traces (not a single line message)
• rabbitmqctl status (and, if possible, rabbitmqctl environment output)
• Other relevant things about the environment and workload, e.g. OS/systemd logs, a traffic capture, deployment tool logs, etc

Feel free to edit out hostnames and other potentially sensitive information.

When/if we have a complete enough understanding of what's going on, a recommendation will be provided or a new issues with more context will be filed.

Thank you.

[michaelklishin]

We cannot suggest anything without a reasonably detailed amount of information about your environment. See logs of all nodes. Enabling debug logging will significantly simplify troubleshooting of peer discovery as every K8S API request/response and decision made by peer discovery will be logged.

We do not guess in this community. If you are looking for help, be mindful of others' time and help them help you.

[michaelklishin]

Also, two node clusters are explicitly recommended against.

[disssid]

Well being a developer myself wouldn't want to waste other developers/community's time, if only the issue template was there before I created the issue I would have used that. That being said apologies for the lack of information.

Also yes I read that we should choose odd number of nodes. But as I was just trying out the setup I went with 2, but now changed it to 3

[disssid]

Server, client library and plugin (if applicable) versions used

RabbitMQ    - 3.8.9
Erlang      - 23.1.5
rabbitmqctl - 3.8.9

Plugins enabled

- rabbitmq_peer_discovery_k8s
- rabbitmq_management
- rabbitmq_prometheus
(couldn't find the versions)

Server logs

rabbitmq-0.log
rabbitmq-1.log
rabbitmq-2.log

A code example or terminal transcript that can be used to reproduce

rabbitmq@rabbitmq-1:/$ rabbitmqctl join_cluster [e-mail]

RABBITMQ_ERLANG_COOKIE env variable support is deprecated and will be REMOVED in a future version. Use the $HOME/.erlang.cookie file or the --erlang-cookie switch instead.
Clustering node rabbit@rabbitmq-1 with [e-mail]
Error:
{:badarg, [{:rpc, :rpcify_exception, 2, [file: 'rpc.erl', line: 467]}, {:rpc, :call, 5, [file: 'rpc.erl', line: 410]}, {:lists, :foldl, 3, [file: 'lists.erl', line: 1267]}, {:rabbit_mnesia, :discover_cluster, 1, [file: 'src/rabbit_mnesia.erl', line: 795]}, {:rabbit_mnesia, :join_cluster, 2, [file: 'src/rabbit_mnesia.erl', line: 227]}]}

Full exception stack traces (not a single line message)

rabbitmq@rabbitmq-1:/$ rabbitmqctl stop_app

RABBITMQ_ERLANG_COOKIE env variable support is deprecated and will be REMOVED in a future version. Use the $HOME/.erlang.cookie file or the --erlang-cookie switch instead.
Stopping rabbit application on node rabbit@rabbitmq-1 ...

rabbitmq@rabbitmq-1:/$ rabbitmqctl join_cluster [e-mail]

RABBITMQ_ERLANG_COOKIE env variable support is deprecated and will be REMOVED in a future version. Use the $HOME/.erlang.cookie file or the --erlang-cookie switch instead.
Clustering node rabbit@rabbitmq-1 with [e-mail]
Error:
{:badarg, [{:rpc, :rpcify_exception, 2, [file: 'rpc.erl', line: 467]}, {:rpc, :call, 5, [file: 'rpc.erl', line: 410]}, {:lists, :foldl, 3, [file: 'lists.erl', line: 1267]}, {:rabbit_mnesia, :discover_cluster, 1, [file: 'src/rabbit_mnesia.erl', line: 795]}, {:rabbit_mnesia, :join_cluster, 2, [file: 'src/rabbit_mnesia.erl', line: 227]}]}

rabbitmq@rabbitmq-1:/$ rabbitmqctl start_app

RABBITMQ_ERLANG_COOKIE env variable support is deprecated and will be REMOVED in a future version. Use the $HOME/.erlang.cookie file or the --erlang-cookie switch instead.
Starting node rabbit@rabbitmq-1 ...

rabbitmqctl status (and, if possible, rabbitmqctl environment output)

rabbitmq@rabbitmq-0:/etc/rabbitmq$ rabbitmqctl status
rabbitmq-status.log

### The status is same for the other two nodes as well other than the node name. If you want to those as well I can attach them too

Cluster status

rabbitmq@rabbitmq-0:/etc/rabbitmq$ rabbitmqctl cluster_status

rabbitmqctl-cluster_status.log

environment-details

rabbitmq@rabbitmq-0:/etc/rabbitmq$ rabbitmqctl environment
rabbitmqctl-environment.log

Other relevant things about the environment and workload, e.g. OS/systemd logs, a traffic capture, deployment tool logs, etc

• OS - Ubuntu 20.04.1 LTS
• Deployment - microk8s

qaadmin@qa-ubuntu:~$ microk8s kubectl version

Client Version: version.Info{Major:"1", Minor:"19+", GitVersion:"v1.19.3-34+a56971609ff35a", GitCommit:"a56971609ff35ac8cc90b2aef89165208bff3fe1", GitTreeState:"clean", BuildDate:"2020-11-06T11:56:24Z", GoVersion:"go1.15.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19+", GitVersion:"v1.19.3-34+a56971609ff35a", GitCommit:"a56971609ff35ac8cc90b2aef89165208bff3fe1", GitTreeState:"clean", BuildDate:"2020-11-06T11:57:19Z", GoVersion:"go1.15.3", Compiler:"gc", Platform:"linux/amd64"}

[disssid]

Here is the microk8s status

qaadmin@qa-ubuntu:~$ microk8s kubectl get all
NAME             READY   STATUS    RESTARTS   AGE
pod/rabbitmq-0   1/1     Running   0          28m
pod/rabbitmq-1   1/1     Running   0          27m
pod/rabbitmq-2   1/1     Running   0          26m

NAME                        TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                                          AGE
service/rabbitmq-headless   ClusterIP   None             <none>        4369/TCP,25672/TCP                               2d4h
service/rabbitmq-client-1   NodePort    10.152.183.191   <none>        15672:30200/TCP,15692:31360/TCP,5672:32472/TCP   28h

NAME                        READY   AGE
statefulset.apps/rabbitmq   3/3     28m

[lukebakken]

If you have the time to edit your comment and attach files rather than pasting huge amounts of text it would be very helpful.

On the rabbitmq-1 node, please run the following command. Capture the command itself and the output to a file, and attach the file in your comment. Thanks.

rabbitmqctl -n [e-mail] status

[disssid]

@lukebakken Sorry about that, its just been a long day trying to figure this out.

Yes I did edit my comment with links to the logs

Here is the file attached with the command you asked to run,
rabbitmqctl-node_status.log

From the output I'm guessing my configuration is wrong, here is the configmap.yaml for you to quickly take a look

apiVersion: v1
kind: ConfigMap
metadata:
  name: rabbitmq-config
  namespace: test-rabbitmq
data:
  enabled_plugins: |
    [rabbitmq_peer_discovery_k8s, rabbitmq_management, rabbitmq_prometheus].
  rabbitmq.conf: |
    cluster_formation.peer_discovery_backend = k8s
    cluster_formation.k8s.host = kubernetes.default.svc.cluster.local
    cluster_formation.k8s.address_type = hostname
    cluster_formation.k8s.service_name = rabbitmq-headless

    queue_master_locator=min-masters
    log.file.level = debug
    log.file = /var/lib/rabbitmq/rabbit.log

Thank you so very much for the quick response.

[disssid]

@lukebakken Did you get a chance to look at the log and the config file?

Thanks

[lukebakken]

No, I had and have higher priority items to work on, some of which are for paying customers. Please be patient.

This is the relevant error message from the rabbitmqctl ... status command:

[e-mail]
If using FQDN node names, use the -l / --longnames argument

Your environment is using long node names. That means that the RABBITMQ_USE_LONGNAME variable must be set when starting RabbitMQ and when running commands like rabbitmqctl.

The usual way of doing this is to create the /etc/rabbitmq/rabbitmq-env.conf file with the following contents:

USE_LONGNAME=true

(Note that the RABBITMQ_ prefix is unnecessary in this file).

Your other option is to ensure that RABBITMQ_USE_LONGNAME=true is set in the environment prior to RabbitMQ starting and when running rabbitmqctl commands. Please run this command and provide the output as you have done before:

RABBITMQ_USE_LONGNAME=true rabbitmqctl -n [e-mail] status

[disssid]

@lukebakken Thank you for all the help

@michaelklishin Thank you too