Shut down Volume Server due to duplicate volume directories about seaweedfs

[kmlebedev]

See #3059

[dyeldandi]

@kmlebedev thanks, but AFAIU #3630 has been already merged into version I'm running, so probably a different issue?

[kmlebedev]

@kmlebedev thanks, but AFAIU #3630 has been already merged into version I'm running, so probably a different issue?

You need to make sure that the contents of vol_dir.uuid are different and that you have not manually copied these files from one directory to another

[dyeldandi]

Yeah, they are definitely different.
1st is 9322c3e6-f8e4-4a53-9129-902e5b24bdb3
2nd is a9ced06f-a886-40d9-942e-73bf40cf5d74
the volume servers run fine for a week or two then one of them suddenly shuts down.

[kmlebedev]

It may be that you accidentally launch a second instance of weed volume at this time, which will erase the same folder with the file vol_dir.uuid

[dyeldandi]

I really doubt it. Nobody logged in on that server for a few months before it crashed. There is nothing in the logs that would indicate this case. Besides, every time it crashes like this there is a failed heartbeat just a few seconds before. E.g.

I0330 07:05:18.266249 volume_grpc_client_to_master.go:71 heartbeat to MASTER_IP_ADDRESS:9333 error: rpc error: code = Unavailable desc = error reading from server: EOF

[kmlebedev]

Heartbeat fails already in the process of stopping the revolutionary server.
Look in the logs on master server there is a message below:

volume: Duplicated volume directories were loaded
directory of %s on %s has been loaded

[dyeldandi]

There is no such message in the logs (I checked both volume and master servers)
Only this one from volume server that mentions duplicity:

E0330 07:05:25.166441 volume_grpc_client_to_master.go:130 Shut down Volume Server due to duplicate volume directories: [/VOLUME/DIRECTORY]

And master reports that directory has been loaded without mentioning duplicated volumes

I0330 07:05:25.135336 master_grpc_server.go:138 added volume server 1: VOLUMESERVER1_IP_ADDRESS:8071 [9322c3e6-f8e4-4a53-9129-902e5b24bdb3]
E0330 07:05:25.136146 master_grpc_server.go:40 directory of 9322c3e6-f8e4-4a53-9129-902e5b24bdb3 on VOLUMESERVER1_IP_ADDRESS:8071 has been loaded