mirror watch doesn't recover when server is temporary down about mc

[znerol]

I'm observing that too. In my case it would be perfect if mc would exit with a non-zero status code if the connection to the bucket is lost.

Browsing through the code, I found the following (just a hypothesis, haven't tested my observations at all):

If I'm not mistaken, errors from the watcher are handled on line 754ff in mirror-main.go. If an error occurs, it is forwarded to the parallel manager (mj.parallel.queueTask()).

The parallel manager will handle that error task in one of its workers. If a worker encounters an error task, then it simply terminates itself (line 97ff in parallel-manager.go). But all the others seem to continue as if nothing happened.

[klauspost]

Not sure if this is something we want to fix. If a remote is down, we will not receive notifications and mirror will miss entries.

Restarting mc seems like the reasonable solution to re-enumerate the files on either side.

[harshavardhana]

That is what is expected from mc --watch not sure why it's failing.

[klauspost]

Well, the server isn't up when it is called, hence the connect: connection refused Isn't there just an mc ping -x miniohost missing?

[harshavardhana]

Well, the server isn't up when it is called, hence the connect: connection refused Isn't there just an mc ping -x miniohost missing?

No the idea was that even with that it perpetually keeps retrying. Looks like someone has changed the code.

[klauspost]

Well, that is a problem, if it changed then.

[znerol]

It seems that the code in parallel-manager.go doesn't actually handle the error, but just passes it back to the result channel. That one is polled in 535ff in mirror-main.go.

Maybe there should be a distinction between fatal errors (connection lost to bucket) and recoverable errors. In that case it would be possible to make a distinction at line 588ff