I/O errors reported by chunkserver after growing filesystem and reload about moosefs

[xandrus]

Hi,
Basically this is not good idea to do partition resize on a working file system and during read write operations.

By the way. Please check this option in /etc/mfs/mfshdd.cfg file.
~ at the beginning of the path means that significant change of total blocks count will not mark this drive as damaged. You can read more about it here.

[marcin-github]

But why error " hdd_io_begin ..." appeared?

[pkonopelko]

Cześć @marcin-github,

Such fragile modifications shouldn't be done when processes are performing I/O operations on a filesystem (of course including mfschunkserver, but I mean in general). The proper way to do grow the filesystem is as follows:

1. Enable maintenance mode for the specific Chunkserver via MFS CGI (tab Servers) or via MFSCLI: mfscli -CM1/ip_of_chunkserver/port_of_chunkserver
2. Stop mfschunkserver process: mfschunkserver stop or your system's specific command (service... / systemctl...)
3. Unmount the filesystem
4. Grow (resize) the filesystem
5. Start mfschunkserver process
6. Disable maintenance mode for the specific Chunkserver via MFS CGI (tab Servers) or via MFSCLI: mfscli -CM0/ip_of_chunkserver/port_of_chunkserver

You can read more about Maintenance mode in "MooseFS 3.0 User's Manual" - Chapter 5.3 "Maintenance mode", p. 48. This document can be obtained from here.

Of course in case of any further questions, please don't hesitate to ask! :)

Best regards / Pozdrawiam
Peter / MooseFS Team

[jamarat]

How about unnecessary / at the end of /var/lib/mfs/chunks/ what makes double // :
/var/lib/mfs/chunks//00/chunk_00000000000001B8_00000009.mfs

[pkonopelko]

@jamarat: yes, double / can be removed, but it actually shouldn't affect I/O operations.

Resizing the filesystem in a mentioned by @marcin-github way may be dangerous for data stored on a filesystem. Fortunately MooseFS is fault-tolerant, so if you have goal >= 2, MooseFS will automatically replicate eventual missing chunk(s) from another copy.

[marcin-github]

Cześć @OXide94 ,
I don't know what did you meant in "the currently resized block might be used by mfschunkserver". I resized filesystem many times while fs was under heavy load (postgres, mysql, or other services, on ext4 or xfs) and never ever I didn't had any issue with it. Only "issue" was frozed I/O for a while.

@jamarat double slash is because in mfshdd.cfg I have:
/var/lib/mfs/ - slash on the end.

[xandrus]

I would like to add that we added such a security - number of total blocks has been changed significantly in MooseFS chunkserver in case of hdd error.
We had such a situation when hard disk crashed and started to report 1PB HDD in system.

When you want to resize hard disk you have to add ~ at the beginning of hdd path.

[pkonopelko]

@marcin-github

I don't know what did you meant in "the currently resized block might be used by mfschunkserver".

Ok, sorry, I read everything once again. It is not true of course. I will edit and remove it from my previous reply not to mislead other users. I'm sorry.

Theoretically such resize (growing) of the filesystem should work properly on working, mounted FS. I just had a bad experience with it once in the past and now, when I have to do it, I always try to do it in the most safest way I can. Nevermind.

As @xandrus wrote:

When you want to resize hard disk you have to add ~ at the beginning of hdd path.

If you do so, mfschunkserver will not consider such HDD as damaged. (Of course after doing so, please don't forget to reload the Chunkserver process).

Peter / MooseFS Team

[marcin-github]

Thanks for answers. I think that we are not focusing on the same problem but nevermind. I didn't have any data loss nor other problems so if you don't see anything wrong with it then it's also ok for me:)

[pkonopelko]

@marcin-github

Messages you mentioned:

Feb 10 13:42:08 ch1 mfschunkserver[10411]: hdd_io_begin: file:/var/lib/mfs/chunks//00/chunk_00000000000001CC_00000002.mfs - open error: ENOENT (No such file or directory)
Feb 10 13:42:09 ch1 mfschunkserver[10411]: set_chunk_version: file:/var/lib/mfs/chunks//00/chunk_00000000000001B8_00000008.mfs->/var/lib/mfs/chunks//00/chunk_00000000000001B8_00000009.mfs - rename error: ENOENT (No such file or directory)
Feb 10 13:42:18 ch1 mfschunkserver[10411]: hdd_io_begin: file:/var/lib/mfs/chunks//00/chunk_00000000000001BC_0000000A.mfs - open error: ENOENT (No such file or directory)

actually mean, that Chunkserver wanted to open chunk 1CC (1st), increase chunk's 1B8 version (2nd) an open chunk 1BC (3rd) - and in each case failed, because for some reason these files were not present.

We'll check it out with our developers next week and get back to you. What filesystem exactly do you have? XFS or EXT4?

Peter

[marcin-github]

xfs, kernel 3.18. Thank you