How to migrate data to new Chunk servers? about moosefs

[pkonopelko]

@aaafei123, @zcalusic:
Yes, it is one of possible options, but it may be slow.

Another one (but faster) is to add new Chunkservers to the existing instance, wait for everything to rebalance, then mark old servers for removal in /etc/mfs/mfshdd.cfg on every Chunkserver (add asterisk (*) at the beginning of every entry) and reload Chunkservers' processes.

Then, wait for chunks to be replicated (it may take quite long, but will definitely be faster than copying everything through one mount).

After chunks are replicated to new Chunkservers, you can safely disconnect old Chunkservers from the instance. At this point you should have new Chunservers connected to the old Master Server.

Next step is to set up new Master Server (don't run it yet). Then, stop old one and copy file /var/lib/mfs/metadata.mfs to new machine. Of course you have to change DNS name mfsmaster (or your custom) to a new IP address or use IP address of old Master Server.

At the end run new Master Server - it should properly load metadata.mfs file and start. Then Chunkservers will automatically connect to the new machine.

PS: It is a shortened description - if you need more info or have additional questions, please don't hesitate to ask! :)

Best regards,
Peter / MooseFS Team

[zcalusic]

Hello aaafei123,

As everything's new, the safest would probably be to build the new empty cluster from scratch, then mount both clusters on some machine and just copy/move data from the old instance to the new. After that reconfigure all the clients to point to the new cluster.

Any other procedure would be more complicated. You'd need to copy/move piece by piece of the old cluster to the new one. First master, then metaloggers, then chunkservers, then clients. And you'd need to be very careful during the move.

[pkonopelko]

Hi @aaafei123, I'm sorry for late reply.

1. When you stop your Master Server, it will dump metadata.mfs file. Unfortunately in such migration there is no possibility to avoid downtime, so you have to stop your MFS Master 1.6 first, then copy metadata.mfs to a new machine (and additionally to a safe place and keep for a few days just in case)

2. What does it mean: "mfscgiserv service cant works well."? Could you please provide more details?

3. Yes, you can. The procedure of upgrade MFS 1.6.27 to MFS 3.0 is the same like from 1.6.27 to 2.0 (see MooseFS 2.0 Upgrade Guide on this website - especially Paragraph 1.1 "Introduction / 1.1 Backing-up data and configuration files" and Chapter 3 "MooseFS 1.6.27-5 to 2.0.x CE
   upgrade". Please don't follow paragraph 1.2.2 "Repository Branches", because it is a bit outdated. Info about repositories is here). Please remember to do a backup of metadata.mfs after stopping your Master as I wrote above.

In case of any further questions, please don't hesitate to ask!

Best regards,
Peter / MooseFS Team

[pkonopelko]

Hi @aaafei123,

if all conditions below are met / fulfilled:

• mentioned Chunkserver has all HDDs marked for removal (their paths in /etc/mfs/mfshdd.cfg are preceded by an asterisk (*) sign) and
• all Chunks are replicated (i.e. in MFS CGI, on INFO tab after switching to regular view (see screenshot) in chunk state matrix there are neither undergoal, nor missing chunks)

then - yes, you can safely disconnect it.

Please upload a screenshot of your chunks state matrix (both all and regular) before disconnecting it, I can verify the situation.

By the way, we strongly recommend to use goal greater than 1, because if you use goal = 1, in case of any failure (disk failure, Chunkserver failure, ...) you loose your data...

Best regards,
Peter / MooseFS Team

[pkonopelko]

Hi @aaafei123,

as I can see on the last screenshot, you haven't marked HDDs for removal. If you want to disconnect a chunkserver from the instance safely, please first of all mark all HDDs on it "for removal". As I wrote a few comments above. To do so, please edit /etc/mfs/mfshdd.cfg (or /etc/mfshdd.cfg) and put and asterisk sign (*) at the beginning of all HDDs paths, so it will looks like:

*/mnt/chunk1
*/mnt/chunk2
*/mnt/chunk3
*...

and then reload the chunkserver process (mfschunkserver reload).

After doing so, your SERVERS tab in MFS CGI should show green bar in "marked for removal" column:

(here should be a green bar after reloading the chunkserver process).

Then you have to wait for chunks to replicate over the rest of Chunkservers.

After replications are finished ( = no undergoal or endangered chunks in "regular space matrix"), you can safely disconnect this chunkserver.

This is the only supported procedure. Please let me know if you want me to clarify anything.

Best regards,
Peter / MooseFS Team

[pkonopelko]

Hi @aaafei123,

I'm sorry for so late reply - I somehow missed your last message. Actually there should be no "endangered" chunks after disconnection (and after properly marking for removal HDDs on a Chunkserver which is going to be disconnected).

I hope the situation is stable now and all data is replicated, but probably it took very long (because of old MooseFS 1.6 replication limit and mechanisms.

Anyway - I strongly recommend to upgrade your instance to MooseFS 2.0 or MooseFS 3.0 (you don't have to migrate anything, you can do an upgrade in-place on existing servers).

Best regards,
Peter / MooseFS Team

[aaafei123]

but when I disconnect every chunkserver, and metalogger server. I can justs find / var / mfs /:changelog.*.mfs,metadata.mfs.back,metadata.mfs.back.1,metadata。 mfs.empty，sessions.mfs。
I can't find metadata.mfs

you must stop your mfs master server firstly(#mfsmaster stop), and then you can see the metadata.mfs file.

[aaafei123]

hi zcalusic,

thanks for your reply:) I will try it following your advice. thanks a lot.

[aaafei123]

@OXide94 thank you very much:)

My MFS current version is v1.6.27, and it has been running for two years.
Here also some quetions need your help:

1. When set up new Master Server, copy file var/lib/mfs/metadata.mfs to new machine. but on my MFS just have the following files on $MFS_HOME/var/mfs/:
   changelog.*.mfs, metadata.mfs.back, metadata.mfs.back.1, metadata.mfs.empty, sessions.mfs and stats.mfs.
   Need I copy all files to new machine(new Master Server)?
2. I started the mfscgiserv service on Master Server($MFS_HOME/sbin/mfscgiserv start ), and visit the URL http://IP:9425 to monitor, but seems the mfscgiserv service cant works well. how to fix it?
3. Can I upgrade the MFS from v1.6.27 to v3.0 when I migrate the old Master to new Master Server? from the docs, I just find the content from v2.0 to v3.0. Here any advices?

Thanks again.

Best regards,
Fei

[aaafei123]

@OXide94 Peter, thank you very much!

#2 mfscgiserv service works fine now.

I will follow your advices to upgrade our MFS. I will email you if there are any issues during upgrading. Hmmm...Can you give me your email address?

Thanks a lot.

Best regards,
Fei

[pkonopelko]

@aaafei123 Please write at [email protected]. It will create a support ticket then. I think it is the best way :)

[aaafei123]

@OXide94 Hi Peter, disturb you again.

if my MFS goal=1, can I REMOVE the old ChunkServer?

thanks.

Best regards,
Fei

[aaafei123]

@OXide94 Hi Peter,
pls check the screenshot below, thanks!

[pkonopelko]

@aaafei123: Could you please switch the view to "regular" and then send the screenshot again?

Best regards,
Peter / MooseFS Team

[aaafei123]

@OXide94 ok, pls check below, thanks a lot:)

[pkonopelko]

@aaafei123 Ok, so now, if you have all HDDs (on the chunkserver you want to disconnect) marked for removal and situation as on both screenshots above - yes, you can safely disconnect this specific Chunkserver. I can finally confirm it if you could send me last screenshot from "SERVERS" tab - on the list, next to the specific Chunkserver you want to disconnect, it's whole space should be listed in table's last column ('marked for removal' hdd space).

Best regards,
Peter / MooseFS Team

[aaafei123]

@OXide94 The “SERVER”tab screenshot:

[aaafei123]

@OXide94 According to the steps(mark hdds for removal and reload the chunkserver), now the status is:

info tab:

servers tab:

disks tab:

there any issues?
thanks a lot~

Best regards,
Fei

[pkonopelko]

@aaafei123: Could you please also send a current screenshot of "regular" view of INFO tab?

Best regards,
Peter / MooseFS Team

[aaafei123]

@OXide94 ok, pls check below, thanks:)

[pkonopelko]

Hi @aaafei123,
if situation looks like on presented screenshots, chunks from this marked for removal Chunkserver are replicated and you can safely stop the Chunkserver process on it.

Best regards,
Peter / MooseFS Team

[aaafei123]

@OXide94 Hi Peter, I have stopped the chunkserver which marked for removal. And I upload the screenshots for reference, and pls help to check everything are ok, thanks again.

info tab(all):

info(regular):

servers:

disks:

[aaafei123]

@OXide94 Hi peter, thanks a lot, everything is ok now:)
Next step, I plan to upgrade the MFS to v3.0
if there any issues during upgrading, I will disturb you again. thanks again.

Best regards,
Fei

[pkonopelko]

Hi @aaafei123,

great to hear that! :) No problem, ask anything you need!

For now I'm closing this issue, of course feel free to create a new one in case of any issues during the upgrade. Please follow the upgrade guide I mentioned in point 3 of this comment: #34 (comment).

Best regards,
Peter / MooseFS Team

[GYUMO]

@OXide94 thank you very much:)

My MFS current version is v1.6.27, and it has been running for two years.
Here also some quetions need your help:

1. When set up new Master Server, copy file var/lib/mfs/metadata.mfs to new machine. but on my MFS just have the following files on $MFS_HOME/var/mfs/:
   changelog.*.mfs, metadata.mfs.back, metadata.mfs.back.1, metadata.mfs.empty, sessions.mfs and stats.mfs.
   Need I copy all files to new machine(new Master Server)?
2. I started the mfscgiserv service on Master Server($MFS_HOME/sbin/mfscgiserv start ), and visit the URL http://IP:9425 to monitor, but seems the mfscgiserv service cant works well. how to fix it?
3. Can I upgrade the MFS from v1.6.27 to v3.0 when I migrate the old Master to new Master Server? from the docs, I just find the content from v2.0 to v3.0. Here any advices?

Thanks again.

Best regards,
Fei

hai. I have a problem, when i change the master server. should I mark all chunk server to be disconnected. I mean, should I disconnect all the chunk server firstly, then copy metadata.mfs to new master server. Finally,change the configuration of all the chunk server to link to the new master node.
Thank you

[aaafei123]

@OXide94 thank you very much:)
My MFS current version is v1.6.27, and it has been running for two years.
Here also some quetions need your help:

1. When set up new Master Server, copy file var/lib/mfs/metadata.mfs to new machine. but on my MFS just have the following files on $MFS_HOME/var/mfs/:
   changelog.*.mfs, metadata.mfs.back, metadata.mfs.back.1, metadata.mfs.empty, sessions.mfs and stats.mfs.
   Need I copy all files to new machine(new Master Server)?
2. I started the mfscgiserv service on Master Server($MFS_HOME/sbin/mfscgiserv start ), and visit the URL http://IP:9425 to monitor, but seems the mfscgiserv service cant works well. how to fix it?
3. Can I upgrade the MFS from v1.6.27 to v3.0 when I migrate the old Master to new Master Server? from the docs, I just find the content from v2.0 to v3.0. Here any advices?

Thanks again.
Best regards,
Fei

hai. I have a problem, when i change the master server. should I mark all chunk server to be disconnected. I mean, should I disconnect all the chunk server firstly, then copy metadata.mfs to new master server. Finally,change the configuration of all the chunk server to link to the new master node.
Thank you

yes, you must disconnect all the chunk servers firstly.

[GYUMO]

Thank you very much

…

------------------ 原始邮件 ------------------ 发件人: "isharing"<[email protected]>; 发送时间: 2019年6月30日(星期天) 下午5:02 收件人: "moosefs/moosefs"<[email protected]>; 抄送: "2994899095"<[email protected]>; "Comment"<[email protected]>; 主题: Re: [moosefs/moosefs] How to migrate data to new Chunk servers? (#34) @OXide94 thank you very much:) My MFS current version is v1.6.27, and it has been running for two years. Here also some quetions need your help: When set up new Master Server, copy file var/lib/mfs/metadata.mfs to new machine. but on my MFS just have the following files on $MFS_HOME/var/mfs/: changelog.*.mfs, metadata.mfs.back, metadata.mfs.back.1, metadata.mfs.empty, sessions.mfs and stats.mfs. Need I copy all files to new machine(new Master Server)? I started the mfscgiserv service on Master Server($MFS_HOME/sbin/mfscgiserv start ), and visit the URL http://IP:9425 to monitor, but seems the mfscgiserv service cant works well. how to fix it? Can I upgrade the MFS from v1.6.27 to v3.0 when I migrate the old Master to new Master Server? from the docs, I just find the content from v2.0 to v3.0. Here any advices? Thanks again. Best regards, Fei hai. I have a problem, when i change the master server. should I mark all chunk server to be disconnected. I mean, should I disconnect all the chunk server firstly, then copy metadata.mfs to new master server. Finally,change the configuration of all the chunk server to link to the new master node. Thank you yes, you must disconnect all the chunk servers firstly. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[GYUMO]

but when I disconnect every chunkserver, and metalogger server. I can justs find / var / mfs /:changelog.*.mfs,metadata.mfs.back,metadata.mfs.back.1,metadata。 mfs.empty，sessions.mfs。
I can't find metadata.mfs