How Can I get back pending deletion files about moosefs

[xandrus]

Hi,
When did you deleted this files?

Do you have old metadata file or metadata backup?
I mean files like metadata.mfs.back metadata.mfs.back.1 in /var/lib/mfs folder?
If yes:

1. Stop mfsmaster
2. BACKUP ALL METADATA FILES!
3. Move /var/lib/mfs/metadata.mfs file to other location.
4. Find the correct metadata file.
5. cp /var/lib/mfs/metadata.mfs.back.1 (or different older metadata file) /var/lib/mfs/metadata.mfs
6. Start master.

I'm looking forward to hearing from you.

[richarson]

Aren't those files (supposed to be) on the trash bin?

Check point 8.13 of the manual:
https://moosefs.com/Content/Downloads/moosefs-3-0-users-manual.pdf

[ruitingh]

thank for your reply.
I lost my files for one day.
the metadata.mfs.back.1 file is just one hour older than metadata.mfs.No other backup files.
I try your suggestions，but the result is same.

and I mount /mfs with -m as meta,but the trash dir is empty.

[whuii]

hi @xandrus
when all the chunkservers report wrong metaid, i delete all the chunkservers /path/from/mfshdd/.metaid file, i got all the chunks pending deletion from cgiserv report. I tried replace the master's meta file from metalogger's, but got the same result, and the trash is empty, how can i get these pending deletion chunks back, thank you !

[chogata]

First, I would like to explain about trash vs. chunks pending deletion, because there is a lot of misunderstanding here. Chunks pending deletion belong to files that don't exist in the filesystem anymore. Files in trash do still exist in the filesystem. You can think about trash as just another directory in mfs, although a special one, that is not visible, unless you specifically mount it. So as long as files exist in this special directory, their chunks will never be pending deletion.

Chunks pending deletion come from two sources only:

• a file has been deleted completely (it can no longer be found even in trash), but the system did not have time yet to physically delete the chunks belonging to that file; those chunks are marked as pending deletion and removed physically as soon as possible
• a chunkserver connects to master and reports chunks that the master cannot find in metadata (i.e. they do not belong to any file); those chunks are marked as pending deletion, but kept from deletion for a number of days (default: 7)

Now, if you unexpectedly end up with a lot of chunks pending deletion and no files in the system, that means that at some point your master server (re)started with wrong metadata. The master has a lot of safeguards to prevent metadata corruption, but can still be forced to start with empty metadata by user (system admin). In that case the only solution is to stop the master, backup and remove the current metadata.mfs file, and restore a previous metadata.mfs file (for example metadata.mfs.back or metadata.mfs.back.1 - those two are created with default mfs settings). In general, the older file should be significantly bigger than your current one. If you have a metalogger, it keeps more copies of metadata.mfs.back.N - you can try to use one of those, typically the biggest one is the right one to use.

Of course, if you deleted the .metaid files on chunkservers already, you will have to delete them again when you use a backup of metadata.mfs file. But you should never have deleted those in the first place. If the system reports metaid conflict, it means something is wrong with your metadata on the master server, not with .metaid files on chunkservers.

If you use the oldest backup metadata file and you still don't see your files, that means that your metadata is gone. Metadata cannot be restored if there is no backup. If your files were small (less than 64MB), there is still a way to recover some data, but you will not recover your directory structure.

But first try to find an older metadata.mfs.back.N file that still works. Let us know if you succeeded with that or not.