linbit / drbd-reactor Goto Github PK

I'm installing Linstor on CentOS7 and making HA Controller.
I have 3 disk nodes that the linstor database resource is placed as written in the guide/documentation.
If I stop one of the machines the drbd storage is ok but in the logs of the machine it tries to mount the database resource on both of the machines.
If I stop parallel two of the machines - the resource is never recovering and the drbd status says all the nodes are diskless.

The one that isn't rebooted:
linstor_db role:Secondary
disk:Diskless quorum:no
node-2 connection:Connecting
node-3 connection:Connecting

The other two:
linstor_db role:Secondary
disk:Diskless quorum:no
node-2 connection:StandAlone
node-3 connection:StandAlone

linstor_db role:Secondary
disk:Diskless quorum:no
node-1 connection:StandAlone
node-2 connection:StandAlone

What should be the problem? :/

Thanks in advance.

Dockerfile

FROM debian:buster as reactor-builder

ARG REACTOR_GITREPO=https://github.com/LINBIT/drbd-reactor
ARG REACTOR_VERSION=0.6.1

ENV DEBIAN_FRONTEND noninteractive
RUN apt-get update \
 && apt-get -y install build-essential debhelper git cargo rustc dh-python bash-completion \
 && apt-get clean \
 && rm -rf /var/lib/apt/lists/*

RUN git clone --recurse-submodules ${REACTOR_GITREPO} /drbd-reactor
WORKDIR /drbd-reactor
RUN git reset --hard v${REACTOR_VERSION} \
 && install /dev/null /usr/local/bin/lbvers.py \
 && make debrelease VERSION=${REACTOR_VERSION} \
 && mv drbd-reactor-${REACTOR_VERSION}.tar.gz ../drbd-reactor_${REACTOR_VERSION}.orig.tar.gz \
 && tar -C / -xvf ../drbd-reactor_${REACTOR_VERSION}.orig.tar.gz
WORKDIR /drbd-reactor-${REACTOR_VERSION}
RUN dpkg-buildpackage -us -uc

Output:

cargo build --offline --release
   Compiling proc-macro2 v1.0.37
   Compiling unicode-xid v0.2.2
   Compiling syn v1.0.91
   Compiling serde_derive v1.0.136
   Compiling version_check v0.9.4
   Compiling serde v1.0.136
   Compiling libc v0.2.124
   Compiling memchr v2.4.1
   Compiling unicode-width v0.1.9
   Compiling log v0.4.16
   Compiling serde_json v1.0.79
   Compiling unicode-segmentation v1.9.0
   Compiling ryu v1.0.9
   Compiling bitflags v1.3.2
   Compiling itoa v1.0.1
   Compiling anyhow v1.0.57
   Compiling cfg-if v1.0.0
   Compiling signal-hook v0.3.13
   Compiling regex-syntax v0.6.25
   Compiling lazy_static v1.4.0
   Compiling shell-words v1.1.0
   Compiling glob v0.3.0
   Compiling textwrap v0.11.0
   Compiling heck v0.3.3
   Compiling clap v2.34.0
error[E0658]: use of unstable library feature 'matches_macro'
   --> /drbd-reactor-0.6.1/vendor/clap/src/errors.rs:392:10
    |
392 |         !matches!(
    |          ^^^^^^^
    |
    = note: for more information, see https://github.com/rust-lang/rust/issues/65721

error[E0658]: `if` is not allowed in a `const fn`
  --> /drbd-reactor-0.6.1/vendor/clap/src/app/settings.rs:7:1
   |
7  | / bitflags! {
8  | |     struct Flags: u64 {
9  | |         const SC_NEGATE_REQS       = 1;
10 | |         const SC_REQUIRED          = 1 << 1;
...  |
51 | |     }
52 | | }
   | |_^
   |
   = note: for more information, see https://github.com/rust-lang/rust/issues/49146
   = note: this error originates in a macro outside of the current crate (in Nightlybuilds, run with -Z external-macro-backtrace for more info)

error[E0658]: `if` is not allowed in a `const fn`
  --> /drbd-reactor-0.6.1/vendor/clap/src/args/settings.rs:6:1
   |
6  | / bitflags! {
7  | |     struct Flags: u32 {
8  | |         const REQUIRED         = 1;
9  | |         const MULTIPLE         = 1 << 1;
...  |
28 | |     }
29 | | }
   | |_^
   |
   = note: for more information, see https://github.com/rust-lang/rust/issues/49146
   = note: this error originates in a macro outside of the current crate (in Nightlybuilds, run with -Z external-macro-backtrace for more info)

   Compiling proc-macro-error-attr v1.0.4
   Compiling proc-macro-error v1.0.4
   Compiling aho-corasick v0.7.18
error: aborting due to 3 previous errors

For more information about this error, try `rustc --explain E0658`.
error: could not compile `clap`.
warning: build failed, waiting for other jobs to finish...
error: build failed
make[1]: Leaving directory '/drbd-reactor-0.6.1'
make[1]: *** [Makefile:35: build] Error 101
dh_auto_build: make -j1 returned exit code 2
make: *** [debian/rules:4: build] Error 2
dpkg-buildpackage: error: debian/rules build subprocess returned exit status 2

I don't know if this is a linstor limitation or misunderstanding by myself ( LINBIT/linstor-server#261), but I can't find a procedure to smoothly take a node off my cluster and do some maintenance.

I use drbd-reactor with linstor NFS gateway and actually if I do:

drbd-reactorctl disable

the DRBD resources on that node become Secondary (which is what I want) but NFS clients actually connected get stucked and NFS shares are not available for them. Whereas the failover seems to have worked fine, only new NFS mount succeed (or remount / reboot previously connected NFS clients)

I'm testing the latest version of drbd-reactor. I've created 2 plugins (promoter and prometheus). For some reason their IDs are showing as "none". Is this a syntax error on my side or a bug ?

root@pve1:~# drbd-reactorctl
Prometheus (ID: '') listening on 10.1.2.12:9942
Promoter (ID: '')
Most likely active on node "pve3"
drbd-services@linstor_db.target
● ├─drbd-promote@linstor_db.service
● ├─linstor-controller.service
● └─var-lib-linstor.mount

root@pve1:~# drbd-reactorctl ls
/etc/drbd-reactor.d/prometheus.toml
Prometheus (ID: '')
/etc/drbd-reactor.d/linstor-ha.toml
Promoter (ID: '')

root@pve1:~# drbd-reactorctl cat
/etc/drbd-reactor.d/prometheus.toml:
id = "prometheus"
[[prometheus]]
enums = true
address = "10.1.2.12:9942"
/etc/drbd-reactor.d/linstor-ha.toml:
id = "linstorha"
[[promoter]]
[promoter.resources.linstor_db]
start = ["var-lib-linstor.mount", "linstor-controller.service"]

config:

[[prometheus]]
address = "127.0.0.1:9942"
enums = true

logs:

INFO [drbd_reactor::plugin::prometheus] tcp_handler: error in stream, most likely harmless shutdown

In kubernetes environment prometheus metric drbdreactor_up does not contain node name (hostname) or ip address. Just instance name (pod name):

drbdreactor_up{app_kubernetes_io_instance="linstor-piraeus-ns", app_kubernetes_io_managed_by="piraeus-operator", app_kubernetes_io_name="piraeus-node", controller_revision_hash="74d8dc587c", env="preprod", instance="linstor-piraeus-ns-node-5g66p", job="pod", kubernetes_namespace="linstor", location="SITE1", name="linstor-piraeus-ns", pod_template_generation="6"}

After upgrading to the latest version of drbd-reactor and linstor-controller, reactor fails to bring up linstor-controller. I'm able to bring up the controller (on all 3 nodes) manually with drbd-reactor disabled.

Could you please check if this is reproducible on your side as well ?

drbd-dkms 9.2.1-1ppa1focal1
drbd-reactor 0.10.2-1ppa1focal1
idrbd-utils 9.22.0-1ppa1~focal1

linstor-client 1.16.0-1ppa1focal1
linstor-common 1.20.1-1ppa1focal1
linstor-controller 1.20.1-1ppa1focal1
python-linstor 1.16.0-1ppa1focal1

Dec 15 11:25:11 linctrl03 systemd[1]: Reloading DRBD-Reactor Service.
Dec 15 11:25:11 linctrl03 systemd[1]: Reloaded DRBD-Reactor Service.
Dec 15 11:25:11 linctrl03 sshd[7396]: Received disconnect from 10.10.12.8 port 56708:11: disconnected by user
Dec 15 11:25:11 linctrl03 sshd[7396]: Disconnected from user root 10.10.12.8 port 56708
Dec 15 11:25:11 linctrl03 systemd[1]: session-33.scope: Succeeded.
Dec 15 11:25:11 linctrl03 sshd[7396]: pam_unix(sshd:session): session closed for user root
Dec 15 11:25:11 linctrl03 systemd-logind[692]: Session 33 logged out. Waiting for processes to exit.
Dec 15 11:25:11 linctrl03 drbd-reactor[733]: INFO [drbd_reactor::plugin::promoter] adjust_resources: waiting for backing device '/dev/ubuntu-vg/linstor_db' to become ready
Dec 15 11:25:11 linctrl03 systemd-logind[692]: Removed session 33.
Dec 15 11:25:11 linctrl03 drbd-reactor[733]: INFO [drbd_reactor::plugin::promoter] adjust_resources: backing device '/dev/ubuntu-vg/linstor_db' now ready
Dec 15 11:25:11 linctrl03 drbd-reactor[733]: INFO [drbd_reactor::plugin::promoter] Checking DRBD options for resource 'linstor_db'
Dec 15 11:25:11 linctrl03 drbd-reactor[733]: WARN [drbd_reactor::plugin::promoter] resource 'linstor_db': DRBD option 'on-suspended-primary-outdated' should be 'force-secondary', but is 'disconnect'
Dec 15 11:25:11 linctrl03 drbd-reactor[733]: INFO [drbd_reactor::plugin::promoter] systemd_write_unit: creating "/run/systemd/system/drbd-promote@linstor_db.service.d/reactor.conf"
Dec 15 11:25:11 linctrl03 drbd-reactor[733]: INFO [drbd_reactor::plugin::promoter] systemd_write_unit: creating "/run/systemd/system/drbd-demote-or-escalate@linstor_db.service.d/reactor.conf"
Dec 15 11:25:11 linctrl03 drbd-reactor[733]: INFO [drbd_reactor::plugin::promoter] systemd_write_unit: creating "/run/systemd/system/var-lib-linstor.mount.d/reactor-50-mount.conf"
Dec 15 11:25:11 linctrl03 drbd-reactor[733]: INFO [drbd_reactor::plugin::promoter] systemd_write_unit: creating "/run/systemd/system/var-lib-linstor.mount.d/reactor.conf"
Dec 15 11:25:11 linctrl03 drbd-reactor[733]: INFO [drbd_reactor::plugin::promoter] systemd_write_unit: creating "/run/systemd/system/linstor-controller.service.d/reactor.conf"
Dec 15 11:25:11 linctrl03 drbd-reactor[733]: INFO [drbd_reactor::plugin::promoter] systemd_write_unit: creating "/run/systemd/system/drbd-services@linstor_db.target.d/reactor.conf"
Dec 15 11:25:11 linctrl03 drbd-reactor[733]: INFO [drbd_reactor::plugin::promoter] systemd_write_unit: creating "/run/systemd/system/drbd-services@linstor_db.target.d/reactor-50-before.conf"
Dec 15 11:25:11 linctrl03 drbd-reactor[733]: INFO [drbd_reactor::plugin::promoter] systemd_daemon_reload: reloading daemon
Dec 15 11:25:11 linctrl03 drbd-reactor[733]: INFO [drbd_reactor::plugin] system: sh -c systemctl daemon-reload
Dec 15 11:25:11 linctrl03 systemd[1]: Reloading.
Dec 15 11:25:11 linctrl03 kernel: drbd linstor_db linctrl02: Preparing remote state change 2101683609
Dec 15 11:25:11 linctrl03 kernel: drbd linstor_db linctrl02: Committing remote state change 2101683609 (primary_nodes=2)
Dec 15 11:25:11 linctrl03 kernel: drbd linstor_db linctrl02: peer( Secondary -> Primary )
Dec 15 11:25:11 linctrl03 drbd-reactor[733]: INFO [drbd_reactor::plugin::promoter] run: resource 'linstor_db' may promote after 0ms
Dec 15 11:25:11 linctrl03 drbd-reactor[733]: INFO [drbd_reactor::plugin::promoter] systemd_start: systemctl start drbd-services@linstor_db.target
Dec 15 11:25:11 linctrl03 systemd[1]: Mounting drbd-reactor controlled var-lib-linstor...
Dec 15 11:25:11 linctrl03 mount[7533]: mount: /var/lib/linstor: mount(2) system call failed: Wrong medium type.
Dec 15 11:25:11 linctrl03 systemd[1]: var-lib-linstor.mount: Mount process exited, code=exited, status=32/n/a
Dec 15 11:25:11 linctrl03 systemd[1]: var-lib-linstor.mount: Failed with result 'exit-code'.
Dec 15 11:25:11 linctrl03 drbd-reactor[7532]: A dependency job for drbd-services@linstor_db.target failed. See 'journalctl -xe' for details.
Dec 15 11:25:11 linctrl03 systemd[1]: Failed to mount drbd-reactor controlled var-lib-linstor.
Dec 15 11:25:11 linctrl03 drbd-reactor[733]: INFO [drbd_reactor::plugin::promoter] stop_actions (could trigger failure actions (e.g., reboot)): linstor_db
Dec 15 11:25:11 linctrl03 drbd-reactor[733]: INFO [drbd_reactor::plugin::promoter] stop_actions: stopping 'drbd-services@linstor_db.target'
Dec 15 11:25:11 linctrl03 systemd[1]: Dependency failed for Services for DRBD resource linstor_db.
Dec 15 11:25:11 linctrl03 systemd[1]: drbd-services@linstor_db.target: Job drbd-services@linstor_db.target/start failed with result 'dependency'.
Dec 15 11:25:11 linctrl03 systemd[1]: Dependency failed for drbd-reactor controlled linstor-controller.
Dec 15 11:25:11 linctrl03 systemd[1]: linstor-controller.service: Job linstor-controller.service/start failed with result 'dependency'.
Dec 15 11:25:11 linctrl03 drbd-reactor[733]: INFO [drbd_reactor::plugin::promoter] systemd_stop: systemctl stop drbd-services@linstor_db.target
Dec 15 11:25:11 linctrl03 systemd[1]: Stopping Promotion of DRBD resource linstor_db...
Dec 15 11:25:11 linctrl03 systemd[1]: drbd-promote@linstor_db.service: Succeeded.
Dec 15 11:25:11 linctrl03 systemd[1]: Stopped Promotion of DRBD resource linstor_db.
Dec 15 11:25:21 linctrl03 kernel: drbd linstor_db linctrl02: Preparing remote state change 103241576
Dec 15 11:25:21 linctrl03 kernel: drbd linstor_db linctrl02: Committing remote state change 103241576 (primary_nodes=0)
Dec 15 11:25:21 linctrl03 kernel: drbd linstor_db linctrl02: peer( Primary -> Secondary )
Dec 15 11:25:21 linctrl03 drbd-reactor[733]: INFO [drbd_reactor::plugin::promoter] run: resource 'linstor_db' may promote after 0ms
Dec 15 11:25:21 linctrl03 drbd-reactor[733]: INFO [drbd_reactor::plugin::promoter] systemd_start: systemctl start drbd-services@linstor_db.target
Dec 15 11:25:21 linctrl03 systemd[1]: Starting Promotion of DRBD resource linstor_db...
Dec 15 11:25:21 linctrl03 kernel: drbd linstor_db linctrl02: Preparing remote state change 2755994662
Dec 15 11:25:21 linctrl03 kernel: drbd linstor_db linctrl02: Committing remote state change 2755994662 (primary_nodes=2)
Dec 15 11:25:21 linctrl03 kernel: drbd linstor_db linctrl02: peer( Secondary -> Primary )
Dec 15 11:25:21 linctrl03 kernel: drbd linstor_db linctrl01: Preparing remote state change 3882556691
Dec 15 11:25:21 linctrl03 kernel: drbd linstor_db linctrl01: Aborting remote state change 3882556691
Dec 15 11:25:22 linctrl03 drbd-linstor_db[7554]: linstor_db: State change failed: (-1) Multiple primaries not allowed by config
Dec 15 11:25:22 linctrl03 systemd[1]: drbd-promote@linstor_db.service: Main process exited, code=exited, status=11/n/a
Dec 15 11:25:22 linctrl03 systemd[1]: drbd-promote@linstor_db.service: Failed with result 'exit-code'.
Dec 15 11:25:22 linctrl03 systemd[1]: Failed to start Promotion of DRBD resource linstor_db.
Dec 15 11:25:22 linctrl03 systemd[1]: Dependency failed for drbd-reactor controlled var-lib-linstor.
Dec 15 11:25:22 linctrl03 drbd-reactor[7553]: A dependency job for drbd-services@linstor_db.target failed. See 'journalctl -xe' for details.
Dec 15 11:25:22 linctrl03 systemd[1]: Dependency failed for Services for DRBD resource linstor_db.
Dec 15 11:25:22 linctrl03 systemd[1]: drbd-services@linstor_db.target: Job drbd-services@linstor_db.target/start failed with result 'dependency'.
Dec 15 11:25:22 linctrl03 systemd[1]: Dependency failed for drbd-reactor controlled linstor-controller.
Dec 15 11:25:22 linctrl03 systemd[1]: linstor-controller.service: Job linstor-controller.service/start failed with result 'dependency'.
Dec 15 11:25:22 linctrl03 drbd-reactor[733]: INFO [drbd_reactor::plugin::promoter] stop_actions (could trigger failure actions (e.g., reboot)): linstor_db
Dec 15 11:25:22 linctrl03 drbd-reactor[733]: INFO [drbd_reactor::plugin::promoter] stop_actions: stopping 'drbd-services@linstor_db.target'
Dec 15 11:25:22 linctrl03 systemd[1]: var-lib-linstor.mount: Job var-lib-linstor.mount/start failed with result 'dependency'.
Dec 15 11:25:22 linctrl03 systemd[1]: drbd-promote@linstor_db.service: Triggering OnFailure= dependencies.
Dec 15 11:25:22 linctrl03 systemd[1]: Starting Demotion of DRBD resource linstor_db...
Dec 15 11:25:22 linctrl03 drbd-linstor_db[7565]: about to demote (or escalate to the FailureAction)
Dec 15 11:25:22 linctrl03 drbd-reactor[733]: INFO [drbd_reactor::plugin::promoter] systemd_stop: systemctl stop drbd-services@linstor_db.target
Dec 15 11:25:22 linctrl03 systemd[1]: drbd-demote-or-escalate@linstor_db.service: Succeeded.
Dec 15 11:25:22 linctrl03 systemd[1]: Finished Demotion of DRBD resource linstor_db.
Dec 15 11:25:36 linctrl03 kernel: drbd linstor_db linctrl02: Preparing remote state change 3160842494
Dec 15 11:25:36 linctrl03 kernel: drbd linstor_db linctrl02: Committing remote state change 3160842494 (primary_nodes=0)
Dec 15 11:25:36 linctrl03 kernel: drbd linstor_db linctrl02: peer( Primary -> Secondary )
Dec 15 11:25:36 linctrl03 drbd-reactor[733]: INFO [drbd_reactor::plugin::promoter] run: resource 'linstor_db' may promote after 0ms
Dec 15 11:25:36 linctrl03 drbd-reactor[733]: INFO [drbd_reactor::plugin::promoter] systemd_start: systemctl start drbd-services@linstor_db.target
Dec 15 11:25:36 linctrl03 kernel: drbd linstor_db linctrl02: Preparing remote state change 822542183
Dec 15 11:25:36 linctrl03 kernel: drbd linstor_db linctrl02: Committing remote state change 822542183 (primary_nodes=2)
Dec 15 11:25:36 linctrl03 kernel: drbd linstor_db linctrl02: peer( Secondary -> Primary )
Dec 15 11:25:36 linctrl03 systemd[1]: Starting Promotion of DRBD resource linstor_db...
Dec 15 11:25:36 linctrl03 drbd-linstor_db[7575]: linstor_db: State change failed: (-1) Multiple primaries not allowed by config
Dec 15 11:25:36 linctrl03 systemd[1]: drbd-promote@linstor_db.service: Main process exited, code=exited, status=11/n/a
Dec 15 11:25:36 linctrl03 drbd-reactor[7574]: A dependency job for drbd-services@linstor_db.target failed. See 'journalctl -xe' for details.
Dec 15 11:25:36 linctrl03 systemd[1]: drbd-promote@linstor_db.service: Failed with result 'exit-code'.
Dec 15 11:25:36 linctrl03 drbd-reactor[733]: INFO [drbd_reactor::plugin::promoter] stop_actions (could trigger failure actions (e.g., reboot)): linstor_db
Dec 15 11:25:36 linctrl03 drbd-reactor[733]: INFO [drbd_reactor::plugin::promoter] stop_actions: stopping 'drbd-services@linstor_db.target'
Dec 15 11:25:36 linctrl03 systemd[1]: Failed to start Promotion of DRBD resource linstor_db.
Dec 15 11:25:36 linctrl03 systemd[1]: Dependency failed for drbd-reactor controlled var-lib-linstor.
Dec 15 11:25:36 linctrl03 systemd[1]: Dependency failed for Services for DRBD resource linstor_db.
Dec 15 11:25:36 linctrl03 systemd[1]: drbd-services@linstor_db.target: Job drbd-services@linstor_db.target/start failed with result 'dependency'.
Dec 15 11:25:36 linctrl03 systemd[1]: Dependency failed for drbd-reactor controlled linstor-controller.
Dec 15 11:25:36 linctrl03 systemd[1]: linstor-controller.service: Job linstor-controller.service/start failed with result 'dependency'.
Dec 15 11:25:36 linctrl03 systemd[1]: var-lib-linstor.mount: Job var-lib-linstor.mount/start failed with result 'dependency'.
Dec 15 11:25:36 linctrl03 systemd[1]: drbd-promote@linstor_db.service: Triggering OnFailure= dependencies.
Dec 15 11:25:36 linctrl03 systemd[1]: Starting Demotion of DRBD resource linstor_db...
Dec 15 11:25:36 linctrl03 drbd-linstor_db[7586]: about to demote (or escalate to the FailureAction)
Dec 15 11:25:36 linctrl03 drbd-reactor[733]: INFO [drbd_reactor::plugin::promoter] systemd_stop: systemctl stop drbd-services@linstor_db.target
Dec 15 11:25:36 linctrl03 systemd[1]: drbd-demote-or-escalate@linstor_db.service: Succeeded.
Dec 15 11:25:36 linctrl03 systemd[1]: Finished Demotion of DRBD resource linstor_db.
Dec 15 11:25:51 linctrl03 kernel: drbd linstor_db linctrl02: Preparing remote state change 3838794161
Dec 15 11:25:51 linctrl03 kernel: drbd linstor_db linctrl02: Committing remote state change 3838794161 (primary_nodes=0)
Dec 15 11:25:51 linctrl03 kernel: drbd linstor_db linctrl02: peer( Primary -> Secondary )
Dec 15 11:25:51 linctrl03 drbd-reactor[733]: INFO [drbd_reactor::plugin::promoter] run: resource 'linstor_db' may promote after 0ms
Dec 15 11:25:51 linctrl03 drbd-reactor[733]: INFO [drbd_reactor::plugin::promoter] systemd_start: systemctl start drbd-services@linstor_db.target
Dec 15 11:25:51 linctrl03 systemd[1]: Starting Promotion of DRBD resource linstor_db...
Dec 15 11:25:51 linctrl03 kernel: drbd linstor_db linctrl02: Preparing remote state change 617163246
Dec 15 11:25:51 linctrl03 kernel: drbd linstor_db linctrl02: Committing remote state change 617163246 (primary_nodes=2)
Dec 15 11:25:51 linctrl03 kernel: drbd linstor_db linctrl02: peer( Secondary -> Primary )
Dec 15 11:25:51 linctrl03 drbd-linstor_db[7598]: linstor_db: State change failed: (-1) Multiple primaries not allowed by config
Dec 15 11:25:51 linctrl03 systemd[1]: drbd-promote@linstor_db.service: Main process exited, code=exited, status=11/n/a
Dec 15 11:25:51 linctrl03 systemd[1]: drbd-promote@linstor_db.service: Failed with result 'exit-code'.
Dec 15 11:25:51 linctrl03 systemd[1]: Failed to start Promotion of DRBD resource linstor_db.
Dec 15 11:25:51 linctrl03 systemd[1]: Dependency failed for drbd-reactor controlled var-lib-linstor.
Dec 15 11:25:51 linctrl03 systemd[1]: Dependency failed for Services for DRBD resource linstor_db.
Dec 15 11:25:51 linctrl03 systemd[1]: drbd-services@linstor_db.target: Job drbd-services@linstor_db.target/start failed with result 'dependency'.
Dec 15 11:25:51 linctrl03 systemd[1]: Dependency failed for drbd-reactor controlled linstor-controller.
Dec 15 11:25:51 linctrl03 systemd[1]: linstor-controller.service: Job linstor-controller.service/start failed with result 'dependency'.
Dec 15 11:25:51 linctrl03 systemd[1]: var-lib-linstor.mount: Job var-lib-linstor.mount/start failed with result 'dependency'.
Dec 15 11:25:51 linctrl03 systemd[1]: drbd-promote@linstor_db.service: Triggering OnFailure= dependencies.
Dec 15 11:25:51 linctrl03 drbd-reactor[7597]: A dependency job for drbd-services@linstor_db.target failed. See 'journalctl -xe' for details.
Dec 15 11:25:51 linctrl03 drbd-reactor[733]: INFO [drbd_reactor::plugin::promoter] stop_actions (could trigger failure actions (e.g., reboot)): linstor_db
Dec 15 11:25:51 linctrl03 drbd-reactor[733]: INFO [drbd_reactor::plugin::promoter] stop_actions: stopping 'drbd-services@linstor_db.target'
Dec 15 11:25:51 linctrl03 systemd[1]: Starting Demotion of DRBD resource linstor_db...
Dec 15 11:25:51 linctrl03 drbd-linstor_db[7610]: about to demote (or escalate to the FailureAction)
Dec 15 11:25:51 linctrl03 drbd-reactor[733]: INFO [drbd_reactor::plugin::promoter] systemd_stop: systemctl stop drbd-services@linstor_db.target
Dec 15 11:25:51 linctrl03 systemd[1]: drbd-demote-or-escalate@linstor_db.service: Succeeded.
Dec 15 11:25:51 linctrl03 systemd[1]: Finished Demotion of DRBD resource linstor_db.
Dec 15 11:25:59 linctrl03 kernel: drbd linstor_db linctrl02: Preparing remote state change 2634231081
Dec 15 11:25:59 linctrl03 kernel: drbd linstor_db linctrl02: Committing remote state change 2634231081 (primary_nodes=0)
Dec 15 11:25:59 linctrl03 kernel: drbd linstor_db linctrl02: peer( Primary -> Secondary )
Dec 15 11:25:59 linctrl03 drbd-reactor[733]: INFO [drbd_reactor::plugin::promoter] run: resource 'linstor_db' may promote after 0ms
Dec 15 11:25:59 linctrl03 drbd-reactor[733]: INFO [drbd_reactor::plugin::promoter] systemd_start: systemctl start drbd-services@linstor_db.target
Dec 15 11:25:59 linctrl03 systemd[1]: Starting Promotion of DRBD resource linstor_db...
Dec 15 11:25:59 linctrl03 kernel: drbd linstor_db linctrl02: Preparing remote state change 100716824
Dec 15 11:25:59 linctrl03 kernel: drbd linstor_db linctrl02: Committing remote state change 100716824 (primary_nodes=2)
Dec 15 11:25:59 linctrl03 kernel: drbd linstor_db linctrl02: peer( Secondary -> Primary )
Dec 15 11:26:00 linctrl03 drbd-linstor_db[7620]: linstor_db: State change failed: (-1) Multiple primaries not allowed by config
Dec 15 11:26:00 linctrl03 systemd[1]: drbd-promote@linstor_db.service: Main process exited, code=exited, status=11/n/a
Dec 15 11:26:00 linctrl03 systemd[1]: drbd-promote@linstor_db.service: Failed with result 'exit-code'.
Dec 15 11:26:00 linctrl03 systemd[1]: Failed to start Promotion of DRBD resource linstor_db.
Dec 15 11:26:00 linctrl03 systemd[1]: Dependency failed for drbd-reactor controlled var-lib-linstor.
Dec 15 11:26:00 linctrl03 drbd-reactor[7619]: A dependency job for drbd-services@linstor_db.target failed. See 'journalctl -xe' for details.
Dec 15 11:26:00 linctrl03 systemd[1]: Dependency failed for Services for DRBD resource linstor_db.
Dec 15 11:26:00 linctrl03 systemd[1]: drbd-services@linstor_db.target: Job drbd-services@linstor_db.target/start failed with result 'dependency'.
Dec 15 11:26:00 linctrl03 systemd[1]: Dependency failed for drbd-reactor controlled linstor-controller.
Dec 15 11:26:00 linctrl03 systemd[1]: linstor-controller.service: Job linstor-controller.service/start failed with result 'dependency'.
Dec 15 11:26:00 linctrl03 systemd[1]: var-lib-linstor.mount: Job var-lib-linstor.mount/start failed with result 'dependency'.
Dec 15 11:26:00 linctrl03 systemd[1]: drbd-promote@linstor_db.service: Triggering OnFailure= dependencies.
Dec 15 11:26:00 linctrl03 drbd-reactor[733]: INFO [drbd_reactor::plugin::promoter] stop_actions (could trigger failure actions (e.g., reboot)): linstor_db
Dec 15 11:26:00 linctrl03 drbd-reactor[733]: INFO [drbd_reactor::plugin::promoter] stop_actions: stopping 'drbd-services@linstor_db.target'
Dec 15 11:26:00 linctrl03 systemd[1]: Starting Demotion of DRBD resource linstor_db...
Dec 15 11:26:00 linctrl03 drbd-linstor_db[7631]: about to demote (or escalate to the FailureAction)
Dec 15 11:26:00 linctrl03 drbd-reactor[733]: INFO [drbd_reactor::plugin::promoter] systemd_stop: systemctl stop drbd-services@linstor_db.target
Dec 15 11:26:00 linctrl03 systemd[1]: drbd-demote-or-escalate@linstor_db.service: Succeeded.
Dec 15 11:26:00 linctrl03 systemd[1]: Finished Demotion of DRBD resource linstor_db.
Dec 15 11:26:00 linctrl03 sshd[7642]: Accepted publickey for root from 10.10.12.8 port 57524 ssh2: RSA SHA256:I2+1ZwpwbgQP/izgNkR6QcvBkUVvBa+YxKjYokgeG9I
Dec 15 11:26:00 linctrl03 sshd[7642]: pam_unix(sshd:session): session opened for user root by (uid=0)
Dec 15 11:26:00 linctrl03 systemd-logind[692]: New session 34 of user root.
Dec 15 11:26:00 linctrl03 systemd[1]: Started Session 34 of user root.
Dec 15 11:26:01 linctrl03 systemd[1]: Reloading DRBD-Reactor Service.
Dec 15 11:26:01 linctrl03 systemd[1]: Reloaded DRBD-Reactor Service.
Dec 15 11:26:01 linctrl03 sshd[7642]: Received disconnect from 10.10.12.8 port 57524:11: disconnected by user
Dec 15 11:26:01 linctrl03 sshd[7642]: Disconnected from user root 10.10.12.8 port 57524
Dec 15 11:26:01 linctrl03 sshd[7642]: pam_unix(sshd:session): session closed for user root
Dec 15 11:26:01 linctrl03 systemd[1]: session-34.scope: Succeeded.
Dec 15 11:26:01 linctrl03 systemd-logind[692]: Session 34 logged out. Waiting for processes to exit.
Dec 15 11:26:01 linctrl03 systemd-logind[692]: Removed session 34.
Dec 15 11:26:07 linctrl03 kernel: drbd linstor_db linctrl02: Preparing remote state change 2041306110
Dec 15 11:26:07 linctrl03 kernel: drbd linstor_db linctrl02: Committing remote state change 2041306110 (primary_nodes=0)
Dec 15 11:26:07 linctrl03 kernel: drbd linstor_db linctrl02: peer( Primary -> Secondary )

I want to use the telegraf prometheus input plugin to scrape the reactor prometheus metrics. This fails because your prometheus is missing the metric name for every HELP line:

# TYPE drbd_device_lowerpending gauge
# HELP Number of open requests to the local I/O sub-system issued by DRBD
[...]
# TYPE drbd_peerdevice_outofsync_bytes gauge 
# HELP Number of bytes currently out of sync with this peer, according to the bitmap that DRBD has for it 

Since both HELP lines have as second word 'Number' the import fails with

2024-02-27T10:36:45Z E! [inputs.prometheus] Error in plugin: error reading metrics for "http://127.0.0.1:9942/metrics": reading text format failed: text format parsing error in line 15: second HELP line for metric name "Number"

Please add the name of the metric like for the TYPE line to solve this issue.

Running Version: drbd-reactor 1.4.0

Hello! How can i transfer linstor-controller role thats currently running on node, to another node in cluster? I have 3 node cluster setup with proxmox.
For example - i want make sure, that drbd-reactor change linstor-controller node correctly.