how to deploy a high-availability cluster about seaweedfs

[britishboop]

Without seeing your error logs I can't be certain but it sounds like this might be the same issue I've run into. It appears that the filer continues to direct id assign requests to a given master even after another is elected.

I can't reproduce it in 3.55 - only 3.56 and I suspect 733db2b is the cause. I'm not overly familiar with it but it seems the new assign proxy logic might just "stick" to the master address returned when it's first called?

For some context, I'm seeing errors like these in the filer logs after the leading master is stopped (including after another leader is elected):

E0916 17:50:27.285278 filer_server_handlers_write.go:48 failing to assign a file id: StreamAssignSend: EOF
E0916 17:50:27.285290 filer_server_handlers_write_upload.go:203 upload error: StreamAssignSend: EOF
E0916 17:50:27.285296 s3api_object_handlers.go:519 upload to filer error: StreamAssignSend: EOF
I0916 17:50:27.285303 common.go:75 response method:PUT URL:/buckets/warp-benchmark-bucket/tnX0VlIw/1.Sc5TCVP67XKiBkW5.rnd with httpStatus:500 and JSON:{"erro  r":"StreamAssignSend: EOF"}
E0916 17:50:27.285435 s3api_object_handlers.go:519 upload to filer error: StreamAssignSend: EOF
E0916 17:50:27.368374 filer_server_handlers_write.go:48 failing to assign a file id: StreamAssignSend: EOF
E0916 17:50:27.368408 filer_server_handlers_write_upload.go:203 upload error: StreamAssignSend: EOF
I0916 17:50:27.368455 common.go:75 response method:PUT URL:/buckets/warp-benchmark-bucket/tnX0VlIw/1.Sc5TCVP67XKiBkW5.rnd with httpStatus:500 and JSON:{"erro  r":"StreamAssignSend: EOF"}
E0916 17:50:27.368730 s3api_object_handlers.go:519 upload to filer error: StreamAssignSend: EOF
E0916 17:50:27.388344 filer_server_handlers_write.go:48 failing to assign a file id: StreamAssignSend: EOF

Restarting the filer sorts the issue (until that master is no longer leader) and it's consistently reproducable. I did manage to have errors along the line of "raft.Server Not current leader" returned at one point as well but I'm struggling to reproduce that.

[Mikkitop]

I encountered the same Issue when upgrading to 3.56. I downgraded to 3.55 to resolve it as I couldn't get it to write at all, not even after restarting the filers:

Sep 21 00:22:02 seaweedfs2 weed@filer[1468898]: E0921 00:22:02.649214 filer_server_handlers_write.go:48 failing to assign a file id: StreamAssignSend: EOF
Sep 21 00:22:02 seaweedfs2 weed@filer[1468898]: E0921 00:22:02.649238 filer_server_handlers_write_upload.go:203 upload error: StreamAssignSend: EOF

Good to see it fixed in the next release.

[chrislusf]

@Mikkitop I want to reproduce this. how to trigger this StreamAssignSend: EOF error?

[Mikkitop]

For me it happened after an upgrade from 3.47 to 3.56. At first everything looked good after having restarted all 3 masters, 3 filers and 60 volumeservers. Reading was not an issue, however uploading a file was broken. It was right after I assume a new leader was elected:

ror":"rpc error: code = Unknown desc = raft.Server: Not current leader"}
Sep 21 00:17:03 seaweedfs3 weed@master[3680239]: E0921 00:17:03.029731 master_grpc_server_assign.go:28 StreamAssign failed to assign: raft.Server: Not current leader
Sep 21 00:17:03 seaweedfs3 weed@filer[3680191]: E0921 00:17:03.029907 filer_server_handlers_write.go:48 failing to assign a file id: rpc error: code = Unknown desc = raft.Server: Not current leader
Sep 21 00:17:03 seaweedfs3 weed@filer[3680191]: E0921 00:17:03.029944 filer_server_handlers_write_upload.go:203 upload error: rpc error: code = Unknown desc = raft.Server: Not current leader
Sep 21 00:17:03 seaweedfs3 weed@filer[3680191]: I0921 00:17:03.029981 common.go:75 response method:POST URL:/tmp/2023-09-20/b0909c9bb2c901a33b79ba1666a3d998650b6f5e907617.98363531/img_thumb.jpg?fsync=true with httpStatus:500 and JSON:{"error":"rpc error: code = Unknown desc = raft.Server: Not current leader"}
Sep 21 00:17:03 seaweedfs3 weed@filer[3680191]: E0921 00:17:03.033246 filer_server_handlers_write.go:48 failing to assign a file id: StreamAssignSend: EOF
Sep 21 00:17:03 seaweedfs3 weed@filer[3680191]: E0921 00:17:03.033269 filer_server_handlers_write_upload.go:203 upload error: StreamAssignSend: EOF
Sep 21 00:17:03 seaweedfs3 weed@filer[3680191]: I0921 00:17:03.033295 common.go:75 response method:POST URL:/tmp/2023-09-20/47bb850f215becb242f38c630d0727ce650b6f5e94a939.54416865/img_small.jpg?fsync=true with httpStatus:500 and JSON:{"error":"StreamAssignSend: EOF"}

After my post I noticed vacuum was not working and found there somehow was a corruption on one file which was also present before the upgrade:
E0921 00:23:16.536049 needle_read.go:45 /var/seaweedfs/dev314/211.dat read 3728 dataSize 9224 offset 59646566768 fileSize 150873988264: read /var/seaweedfs/dev314/211.dat: input/output error
E0921 00:23:18.204353 volume_grpc_vacuum.go:65 failed compact volume 211: cannot hydrate needle from file: read /var/seaweedfs/dev314/211.dat: input/output error

I fixed that by deleting and reuploading the 4 files that gave an error (detected through a volume.fsck and I have replication on 001)
failed to read 211:11f95b54 needle status of file /****: rpc error: code = Unknown desc = read /var/seaweedfs/dev314/211.dat: input/output error

I'll try to reproduce it in a test environment on 3.56. Anything in particular I should look at?

[chrislusf]

thanks for the quick reply and detailed context. It's helpful to know the issue is caused by the moved leader.

[kgalinurov]

Any progress here ?

[chrislusf]

this is fixed.

[kgalinurov]

in 3.57 or in next release ?

[chrislusf]

3.57