mfsmaster -a restore hangs with 100% CPU usage about moosefs

[chogata]

Did you try WITHOUT -xx? Are there any error messages, or is the process just working and taking a lot of CPU? If it's the latter, how long did you wait for it to complete?

Yes, deleting files is a very intense operation (practically in every fs out there) and adding several hours worth of changelogs with mostly unlink operations may take a lot of time. If your system performs a lot of delete operations, then I would suggest you change your configuration for more frequent saves of metadata backups on metaloggers (even every hour), so there is less changelogs to apply in case of master failure.

[baby-gnu]

Did you try WITHOUT -xx? Are there any error messages, or is the process just working and taking a lot of CPU? If it's the latter, how long did you wait for it to complete?

No error, it's a test bed so I only have my /usr.

The start of the logs is:

loading metadata ...
found valid metadata file: metadata.mfs.empty (version: 1)
found valid metadata file: metadata_ml.mfs.back.1 (version: 3230313 ; id: 64414D6881386D89)
found invalid metadata file (wrong header): metadata.crc
found valid metadata file: metadata.mfs.back.1 (version: 3259697 ; id: 64414D6881386D89)
found valid metadata file: metadata_ml.mfs.back.2 (version: 2920563 ; id: 64414D6881386D89)
found valid metadata file: metadata_ml.mfs.back.3 (version: 2920563 ; id: 64414D6881386D89)
found valid metadata file: metadata_ml.mfs.back (version: 3259697 ; id: 64414D6881386D89)
found valid metadata file: metadata.mfs (version: 3259697 ; id: 64414D6881386D89)
chosen most recent metadata file: metadata.mfs.back.1 (version: 3259697 ; id: 64414D6881386D89)
loading sessions data ... ok (0.0000)
loading storage classes data ... ok (0.0000)
loading objects (files,directories,etc.) ... ok (0.1522)
loading names ... ok (0.2530)
loading deletion timestamps ... ok (0.0015)
loading quota definitions ... ok (0.0000)
loading xattr data ... ok (0.0000)
loading posix_acl data ... ok (0.0000)
loading open files data ... ok (0.0000)
loading flock_locks data ... ok (0.0000)
loading posix_locks data ... ok (0.0000)
loading chunkservers data ... ok (0.0000)
loading chunks data ... ok (0.1170)
checking filesystem consistency ... ok
connecting files and chunks ... ok
all inodes: 270896
directory inodes: 2
file inodes: 270894
chunks: 270568
skipping changelog file: changelog_ml_back.1.mfs (changes: ??? - ???)
using changelog file: changelog_ml.0.mfs (changes: 3259697 - 3564466)
skipping changelog file: changelog_ml.1.mfs (changes: 3244768 - 3259696)
skipping changelog file: changelog.4.mfs (changes: 2593709 - 2631558)
skipping changelog file: changelog_ml_back.0.mfs (changes: ??? - ???)
filename: changelog_ml.0.mfs ; current meta version: 3259697 ; previous changeid: 3259696 ; current changeid: 3259697 ; change data1682353320|CREATE(2,usr,2,511,18,0,0,0):326173

changelog_ml.0.mfs: change1682353320|CREATE(2,usr,2,511,18,0,0,0):326173
[...]

in my last test, the logs stop at a lines:

filename: changelog_ml.0.mfs ; current meta version: 3549085 ; previous changeid: 3549084 ; current changeid: 3549085 ; change data1682353481|UNLINK(346981,vendor-completions):359282

changelog_ml.0.mfs: change1682353481|UNLINK(346981,vendor-completions):359282

and the process eat 100% CPU without any new logs.

I started the process in a screen and I'll see tomorrow how things are going.

Thanks.

[baby-gnu]

I started the process in a screen and I'll see tomorrow how things are going.

Still the same, eating 100% CPU and no progress in the logs.

[chogata]

Since 3.0.115 we fixed one bug related to the restore process. It was a rare occurrence when it activated, nevertheless it's possible it is affecting you. Please, upgrade to the latest version (3.0.117) and try again.

[baby-gnu]

Ok, I tried the same configuration on Debian Bookworm using moosefs 3.0.117 and it's working fine.

Thanks.