dangled meta node about influxdb-cluster

[chengshiwen]

If you have multiple meta nodes, DO NOT USE localhost:8091, which will cause network communication problem.

[andyli029]

@chengshiwen
DO NOT USE localhost:8091, how to make it healthy again?
Multiple meta nodes exist different nodes.

# influxd-ctl show

Sometimes it works, sometimes it doesn't

[chengshiwen]

@andyli029 What is the result of influxd-ctl show?

[andyli029]

@chengshiwen

# influxd-ctl show
Data Nodes
==========
ID	TCP Address		Version
4	10.80.51.86:8088	1.8.10-c1.1.2
5	10.80.51.87:8088	1.8.10-c1.1.2

Meta Nodes
==========
ID	TCP Address		Version
2	10.80.51.86:8091	1.8.10-c1.1.2
3	10.80.51.87:8091	1.8.10-c1.1.2

Execute after a few seconds

# influxd-ctl show
Data Nodes
==========
ID	TCP Address	Version

Meta Nodes
==========
ID	TCP Address	Version

[chengshiwen]

The result sometimes disappears, which node are you executing on?

[andyli029]

@chengshiwen

The result sometimes disappears, which node are you executing on?

Meta Node

[chengshiwen]

Is it on node 10.80.51.86:8091 or 10.80.51.87:8091?

[andyli029]

Is it on node 10.80.51.86:8091 or 10.80.51.87:8091?

@chengshiwen yes

[chengshiwen]

Are the influxd-ctl show commands executed twice on the same node?

[chengshiwen]

If you want to remove a meta node, you should specify the -bind option, like

influxd-ctl -bind meta-node-02:8091 remove-meta meta-node-01:8091

[chengshiwen]

Also, could you provide steps to reproduce this issue?

[andyli029]

Are the influxd-ctl show commands executed twice on the same node?

@chengshiwen Yes

[andyli029]

Also, could you provide steps to reproduce this issue?

@chengshiwen
remove-meta $node
add-meta $node
.....

[chengshiwen]

This step description is too simple, while detailed and stable reproduction steps include: operating system and version, configuration file information, operation steps, etc., so it can help me reproduce the same issue in my environment.

[Ansible-NW]

Hi,
First of all, I wann thank you for this project.
I have the same issues. I do not manage to add all my 3 nodes to my influxdb cluster because of that error:

add-meta: operation exited with error: dangled meta node at "localhost:8091" already has state present, cannot add another meta node

When I run influxd-ctl show it shows me nothing:

Data Nodes
==========
ID      TCP Address     Version

Meta Nodes
==========
ID      TCP Address     Version

I made a service to launch my node by creating this file:

[Unit]
Description=InflusDB Meta Node
After=syslog.target network.target

[Service]
Type=simple
KillMode=process

# The healthchecker potentially needs to open a lot of network connections.
LimitNOFILE=8192
LimitNPROC=8192

ExecStart=/usr/bin/influxd-meta -config /etc/influxdb/influxdb-meta.conf
Restart=always
RestartSec=30
TimeoutSec=30

PrivateTmp=true
User=root
Group=root

[Install]
WantedBy=multi-user.target

An it seems to work perfectly.
Here is the result of my systemctl status :

influxdb-meta.service - InflusDB Meta Node
     Loaded: loaded (/etc/systemd/system/influxdb-meta.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2023-10-13 15:01:53 UTC; 7min ago
   Main PID: 800 (influxd-meta)
      Tasks: 5 (limit: 2220)
     Memory: 22.4M
        CPU: 548ms
     CGroup: /system.slice/influxdb-meta.service
             └─800 /usr/bin/influxd-meta -config /etc/influxdb/influxdb-meta.conf

oct. 13 15:01:53 influxdb-meta02 systemd[1]: Started InflusDB Meta Node.
oct. 13 15:01:53 influxdb-meta02 influxd-meta[800]: ts=2023-10-13T15:01:53.612661Z lvl=info msg="InfluxDB Meta starting" log_id=0krOmDJ0000 version=1.8.10-c1.1.2 b>
oct. 13 15:01:53 influxdb-meta02 influxd-meta[800]: ts=2023-10-13T15:01:53.612686Z lvl=info msg="Go runtime" log_id=0krOmDJ0000 version=go1.16.15 maxprocs=2
oct. 13 15:01:53 influxdb-meta02 influxd-meta[800]: ts=2023-10-13T15:01:53.620732Z lvl=info msg="Listening on TCP: [::]:8089" log_id=0krOmDJ0000
oct. 13 15:01:53 influxdb-meta02 influxd-meta[800]: ts=2023-10-13T15:01:53.620776Z lvl=info msg="Starting meta service" log_id=0krOmDJ0000
oct. 13 15:01:53 influxdb-meta02 influxd-meta[800]: ts=2023-10-13T15:01:53.620802Z lvl=info msg="Listening on HTTP" log_id=0krOmDJ0000 addr=[::]:8091 https=false
oct. 13 15:01:53 influxdb-meta02 influxd-meta[800]: ts=2023-10-13T15:01:53.631909Z lvl=info msg="Using data dir" log_id=0krOmDJ0000 service=metastore path=/var/lib>
oct. 13 15:01:53 influxdb-meta02 influxd-meta[800]: ts=2023-10-13T15:01:53.699793Z lvl=info msg="Raft state" log_id=0krOmDJ0000 service=metastore state="Node at in>

I don't know if I should create an influxdb cluster as it seems to be the case when running the Influxdb Enterprise installer.
I don't know if I'm doing the installation correctly.

My system info:
Linux 5.15.0-86-generic Ubuntu x86_64 x86_64 x86_64 GNU/Linux

Can you please help me with this problem?

Best regards

[chengshiwen]

@Ansible-NW Could you provide steps to reproduce this issue?

[Ansible-NW]

@chengshiwen On one of my meta node, each time I want to add my third node running this command:
influxd-ctl add-meta influxdb-meta03:8091

It show me this error:
add-meta: operation exited with error: dangled meta node at "localhost:8091" already has state present, cannot add another meta node

I never added localhost as a nod in my cluster though

[chengshiwen]

@Ansible-NW Reset the cluster and all nodes. Then execute the command like

influxd-ctl -bind influxdb-meta01:8091 add-meta influxdb-meta01:8091
influxd-ctl -bind influxdb-meta01:8091 add-meta influxdb-meta02:8091
influxd-ctl -bind influxdb-meta01:8091 add-meta influxdb-meta03:8091

Please refer to Manage Clusters to learn about the option -bind

[Ansible-NW]

@chengshiwen thank you. I resetted the cluster and manage to add all the nodes (meta and data) to the cluster.
I think the problem came from my first integration. I mad sure the node were available on the FQDN but not just from the hostname.

Everything felt to place after I setted up my conf to allow hostname to be pinged.

Just on more question, to install chronograf should I take the deb package from the influxdb official page or do you provide one?

[chengshiwen]

@Ansible-NW Just to download he official installation package of chronograf

[Ansible-NW]

Thank you for your time @chengshiwen