ibm-cloud-architecture / icp-backup Goto Github PK

Hi Eduardo,

I was following your description on this topic on GitHub:

https://github.com/ibm-cloud-architecture/icp-backup/blob/3e7fc551bd5f246849fddbb353c37b1567cf5db8/docs/mongodb.md

Hoping to have done it correctly I am nevertheless now stuck with the following issue: The deployed bitnami/mongodb cannot connect to the icp-mongodb service.

Upfront: I am no ICP/kubernetes expert... so hopefully it is nothing obvious/stupid :-)

Here is what I am deploying:

apiVersion: batch/v1
kind: Job
metadata:
name: icp-backup-mongotest
namespace: kube-system
spec:
template:
spec:
containers:
- name: icp-backup-mongotest
image: vmh-lh-icp-master-test.dcx.dlh.de:8500/kube-system/mongodb:4.1.3
command: ["mongodump", "--host", "icp-mongodb", "--password", "password", "--out", "/dump", "--verbose"]
volumeMounts:
- mountPath: "/dump"
name: icp-backup-mongotest
volumes:
- name: icp-backup-mongotest
persistentVolumeClaim:
claimName: pvc-icp-backup-mongotest
restartPolicy: Never
backoffLimit: 4

This brings up the pods/containers. But they fail. Trying to troubleshoot this I did a "Deployment" of the image instead of the "Job" and then tried to execute the mongodump manually:

$ kubectl exec -it icp-backup-mongotest-69b874f9b8-6ttrh --namespace=kube-system -- /bin/bash
I have no name!@icp-backup-mongotest-69b874f9b8-6ttrh:/$ mongodump --host icp-mongodb --password password --out /dump --verbose
2018-09-16T11:00:47.342+0000 Failed: error connecting to db server: no reachable servers
I have no name!@icp-backup-mongotest-69b874f9b8-6ttrh:/$ command terminated with exit code 137

So somehow the service is not reachable (both being in kube-system namespace though). And the service icp-mongodb appears to be running to me:

$ kubectl get svc --namespace=kube-system | grep mongo
icp-mongodb ClusterIP None 27017/TCP 12d
mongodb NodePort 10.0.0.44 27017:30419/TCP 12d

$ kubectl describe svc icp-mongodb --namespace=kube-system
Name: icp-mongodb
Namespace: kube-system
Labels: app=icp-mongodb
chart=icp-mongodb-2.2.1
heritage=Tiller
release=mongodb
Annotations: service.alpha.kubernetes.io/tolerate-unready-endpoints=true
Selector: app=icp-mongodb,release=mongodb
Type: ClusterIP
IP: None
Port: peer 27017/TCP
TargetPort: 27017/TCP
Endpoints: 198.18.162.250:27017,198.18.23.67:27017,198.18.47.189:27017
Session Affinity: None
Events:

Any ideas how to solve this our troubleshoot this further?

Mit freundlichen Grüßen / Kind Regards

Hi,
There is an error running backupEtcd.sh script:

[root@icp-master1 scripts]# ./backupEtcd.sh
Current Time : 2019.03.04-10.27.26
Back up to the following file: /data/etcd.2019.03.04-10.27.26.db
Unable to find image 'ibmcom/etcd:v3.2.24' locally
docker: Error response from daemon: manifest for ibmcom/etcd:v3.2.24 not found.
See 'docker run --help'.

Loocking at the etcd.sh script you can see that the Tag is defined as "v3.2.24", but the actual Tag is "3.2.4" whitout the "v". If you change the Tag definition in the script, the script works ok and the snapshot is created.

Thanks for your time, Lissette

in the mean time I've taken the etcd.sh and backupEtcd.sh scripts from github and put them on master1 however they don't complete succesfully

When there is a mgmt node, the procedure described at https://github.com/ibm-cloud-architecture/icp-backup/blob/master/docs/etcd_restore_multi.md fails, at some key components (particularly kube-dns running on the mgmt node) gets confused with the new etcd data.

We need to describe how to properly restore the environment if there is a mgmt node.

when running this command it fails:

root@kub-ams03-01-master01:/var/lib/registry# kubectl run mongodb-mongodb-client --rm --tty -i --image bitnami/mongodb --command -- mongo --host mongodb-mongodb -p password
If you don't see a command prompt, try pressing enter.
Error attaching, falling back to logs:
MongoDB shell version v4.0.3
connecting to: mongodb://mongodb-mongodb:27017/
2018-10-17T09:02:34.979+0000 E QUERY [js] Error: couldn't connect to server mongodb-mongodb:27017, connection attempt failed: HostNotFound: Could not find address for mongodb-mongodb:27017: SocketException: Host not found (non-authoritative), try again later :
connect@src/mongo/shell/mongo.js:257:13
@(connect):1:6
exception: connect failed

Hi Eduardo. Jonathon Goldsworthy said I should reach out to you directly to ask about your ICP Backup/Restore stuff on Github here . I’m using some of the commands and have questions about the MongoDB backup. I am trying to run the job you defined and I noticed that the command has arguments in the wrong order
This
command: ["mongodump", "--host", "mongodb-mongodb", "--password", "password", "--out", "--verbose", "/dump"]
Should be
command: ["mongodump", "--host", "mongodb-mongodb", "--password", "password", "--out", "/dump", "--verbose"]

And lastly, running the job results in logs stating that a valid database server could not be found. Looking at the mongodb secrets in ICP, the password value you have there also doesn’t match and there is no service that uses mongodb-mongdo, only icp-mongodb and mongodb. Do you have any other documentation that deals specifically for backup/restore data from MongoDB in both single and multi-master ICP clusters?

I’m pulling together the commands for a Backup/Restore strategy at ABN AMRO and they will absolutely need (same with other large customers) this defined and I have not yet seen anything definitive to use as a reference.

Thanks for your time

Etcd won't start after restoring in ICP3.1.1,
I found that at this icp version etcd is configured with dedicated wal directory(--wal-dir=/var/lib/etcd-wal) the etcd snapshot restore puts both the wal and snap files in the same directory(/var/lib/etcd) so etcd starting with the old wal files and crashing. my solution is to move the restored wal files from /var/lib/etcd/member/wal to /var/lib/etcd-wal/wal
I made a fork to this repository with my fixes at https://github.com/roitevethibm/icp-backup/blob/master/docs/etcd_restore_multi.md
can you please validate it? if my solution gets validated I'll be glad to make a pull request from my fork.

Thank you
Roi Teveth

I think official image can do the same functionality and you take it from the official source.
You can also add an specific version(tag) to the image of your job:
https://raw.githubusercontent.com/patrocinio/mongodb/master/deploy/mongodump_job.yaml
ICP uses Mongo v4.0.0 in my case...

This applies to both icp-mongodb-mongodump-job.yaml and icp-mongodb-mongorestore-job.yaml .

1. Mounted secret permissions are 400 and cant be read by the nobody user running the newer mongodb image - this can be resolved by changing the secrets to defaultMode: 0444.

2. The pod command uses cat to combine the certs and place them in /, however, the user can't write to /, changing to a writable dir such as /tmp fixes this

3. The mongorestore command flag --host should be rs0/mongodb:$MONGODB_SERVICE_PORT, otherwise you receive an error Failed: no reachable servers

Next define an environment variable endpoint that points to a Master Node:

This is not right to backup the ETCD...it should be, the endpoint is the node where etcd is installed

Hi Eduardo,

We were following your description on this topic on GitHub to take the ICP Mongo DB backup.

https://github.com/ibm-cloud-architecture/icp-backup/blob/3e7fc551bd5f246849fddbb353c37b1567cf5db8/docs/mongodb.md

But it is not getting succeeded. It is throwing error,we are getting connection error as below while connecting to Mongo DB.

[root@dev-icp-mstr-01 ~]# kubectl get pods -nkube-system | grep mongo
mongoutil-mongodump-sbmzp 1/1 Running 0 10s
[root@dev-icp-mstr-01 ~]#
[root@dev-icp-mstr-01 ~]# kubectl logs mongoutil-mongodump-4fdk5 -nkube-system
2019-01-21T11:08:24.419+0000 Failed: error connecting to db server: no reachable servers

Are the steps mentioned in github is correct or do you have any updated steps to perform this activity but not included in git.

Please do the needful we are unable to configure the ICP Mongodb backup in customer environment.

Hi,

ICP dash board is down with 504 gateway timeout error and in the master node noticed maria db pod is not started because /var/lib/mysql got overwrote by other process. Is there any way I can create maria db pod from initial ICP installation or any other way to recover. We don't have much work happened after ICP installation, we are ok to restore with initial one.

MongoDB in the newest fixpack has been updated to 4.0.20. Seems the --ssl flag for mongodump has been removed.
Removing the flags from the script cause "Failed: error connecting to db server: no reachable servers"

Any workarounds?

Hi John,

Thanks for sharing the Github link(https://github.com/ibm-cloud-architecture/icp-backup/blob/master/docs/mongodb.md).

I have already tried with this steps and facing issues during the backup. Please look into below points and provide the suggestion.

1. Since our client servers are not exposed to internet, facing issue on the pulling the images on scheduled nodes.

2. The bitnami/mongodb image- containers default doesn't have privilege to create a mount point/directory on the container & in PVC's.

3. Our ICP-mongodb provisioned with secured SSL & Username/Password. We are facing the issue on connecting the ICP-mongdb with SSL.

4. Not sure about the DB name inside the ICP-MongoDB to take a backup.

5. Not sure about the the ICP-MongoDb secret user(masked by secret on statefulSet) having the privilege to take the dump & restore.

Referred link: https://github.com/ibm-cloud-architecture/icp-backup/blob/master/docs/mongodb.md
https://medium.com/@epatro/how-to-back-up-a-mongodb-database-deployed-to-a-kubernetes-environment-ed681f4f7146
https://docs.mongodb.com/manual/tutorial/backup-and-restore-tools/
https://docs.mongodb.com/manual/core/backups/

Thanks,
G.Senthilkumar