Forwarder restarts with FATAL error in case of registry and nsmgr reboot about cmd-forwarder-vpp

[richardstone]

@edwarnicke Sure. Meet you there :)

[edwarnicke]

@richardstone Thanks for finding this.

@denis-tingaikin @glazychev-art Any ideas?

[denis-tingaikin]

This is a good catch. We'll consider the FATAL problem within networkservicemesh/sdk#1170

Moreover, we're planning to add advanced healing when two components die. @glazychev-art Could you remind issue number?

[edwarnicke]

@richardstone Looking at that logs and the code, this looks like the Forwarder is in the process of starting up and hits:

cmd-forwarder-vpp/main.go

Lines 252 to 254 in 6898aff

	if err != nil {
	log.FromContext(ctx).Fatalf("failed to connect to registry: %+v", err)
	}

Due to not being able to register with the Registry on startup (probably because the NSMgr isn't running yet).

Normally, the Forwarder would restart until the NSMgr is available to be registered with and then everything would converge... did that happen for you?

While we could consider other, potentially more graceful ways of handling this situation, it is the case that things should converge no matter what order the Forwarder, NSMgr, etc start in. If that's not happening, I'm very concerning bug. If convergence isn't happening fast enough, I'm quite open to talking about that as well, but would be curious if you have a notion of how fast you need it to be :)

[richardstone]

@edwarnicke In my case everything started to work just fine, shortly after the unexpected restart of the forwarder.

My concern is not about the time until we reach convergence in these cases, but the unexpected restart of the forwarder when it's unable to reach certain NSM components at the right time.
I'm okay with a bit slower start time for the new instance, so that it can wait - instead of restart - until it's able to reach every component.

[edwarnicke]

@richardstone

@edwarnicke In my case everything started to work just fine, shortly after the unexpected restart of the forwarder.

Glad everything started to work just fine :) That's the behavior I would expect.

My concern is not about the time until we reach convergence in these cases, but the unexpected restart of the forwarder when it's unable to reach certain NSM components at the right time. I'm okay with a bit slower start time for the new instance, so that it can wait - instead of restart - until it's able to reach every component.

I tend to think about this in layers:

1. This is all working as designed... which is not to assert the design is correct :)
2. The restart only happens if the registry isn't available initially to the Forwarder, if the registry disappears after becoming initially available, the Forwarder should be fine.
3. Alarming users is not cool, even if things eventually end up being OK, so I'm very open to talking about less alarming ways to do this.

What would make this less alarming to you?

[richardstone]

@edwarnicke

The most alarming thing to me is that the forwarder restarted as soon as it faced a connection loss. If It could wait for a certain timeout (configurable) that would be the best, so if it get's connected to the other components within the specified timeout then everything is fine. In any other case it is free to restart, so the user gets notified about a - probably - bigger issue with the connection.

This should apply to every NSM component in my opinion.

[edwarnicke]

@richardstone Any chance you would be in the community meeting today (Tue) at 8am PT to discuss?

[denis-tingaikin]

Should be fixed with Implement 'retry' registry chain element networkservicemesh/sdk#1201

@richardstone Be free to reopen this issue if it will reproducible!