Try new br with full backup.
Here is the cluster:
[[email protected] tidb-cluster]$kubectl get pod -n dan5
NAME READY STATUS RESTARTS AGE
dan5-pd-0 1/1 Running 0 112m
dan5-pd-1 1/1 Running 0 113m
dan5-pd-2 1/1 Running 0 114m
dan5-tidb-0 2/2 Running 0 42m
dan5-tidb-1 2/2 Running 0 43m
dan5-tikv-0 1/1 Running 0 111m
dan5-tikv-1 1/1 Running 0 112m
dan5-tikv-2 1/1 Running 0 112m
./br backup full --pd 10.233.40.168:2379 --storage local:///var/lib/tikv/backup
[[email protected] tidb-cluster]$kubectl exec -it dan5-tikv-0 -n dan5 -- sh -c "ls -lartR /var/lib/tikv/backup"
/var/lib/tikv/backup:
total 20324
drwxr-xr-x 7 root root 102 Nov 18 09:17 ..
drwxr-xr-x 2 root root 6 Nov 18 09:17 localtmp
-rw-r--r-- 1 root root 1731074 Nov 18 09:17 4_1005_22_49_write.sst
-rw-r--r-- 1 root root 19077269 Nov 18 09:17 4_1005_22_49_default.sst
drwxr-xr-x 3 root root 84 Nov 18 09:17 .
/var/lib/tikv/backup/localtmp:
total 0
drwxr-xr-x 3 root root 84 Nov 18 09:17 ..
drwxr-xr-x 2 root root 6 Nov 18 09:17 .
[[email protected] tidb-cluster]$kubectl exec -it dan5-tikv-2 -n dan5 -- sh -c "ls -lartR /var/lib/tikv/backup"
/var/lib/tikv/backup:
total 20328
drwxr-xr-x 7 root root 112 Nov 18 08:21 ..
drwxr-xr-x 2 root root 6 Nov 18 09:17 localtmp
-rw-r--r-- 1 root root 4589 Nov 18 09:17 backupmeta
-rw-r--r-- 1 root root 1726923 Nov 18 09:17 1_2_22_51_write.sst
-rw-r--r-- 1 root root 19076989 Nov 18 09:17 1_2_22_51_default.sst
drwxr-xr-x 3 root root 96 Nov 18 09:17 .
/var/lib/tikv/backup/localtmp:
total 0
drwxr-xr-x 3 root root 96 Nov 18 09:17 ..
drwxr-xr-x 2 root root 6 Nov 18 09:17 .
[[email protected] tidb-cluster]$kubectl exec -it dan5-tikv-1 -n dan5 -- sh -c "ls -lartR /var/lib/tikv/backup"
/var/lib/tikv/backup:
total 0
drwxr-xr-x 2 root root 6 Nov 18 09:17 localtmp
drwxr-xr-x 7 root root 102 Nov 18 09:17 ..
drwxr-xr-x 3 root root 22 Nov 18 09:17 .
/var/lib/tikv/backup/localtmp:
total 0
drwxr-xr-x 3 root root 22 Nov 18 09:17 ..
drwxr-xr-x 2 root root 6 Nov 18 09:17 .
However, there is no backup file on tikv-1 even if there are 20 region leaders of 22 regions totally on it:
[[email protected] test]$curl http://10.233.40.168:2379/pd/api/v1/store/5
{
"store": {
"id": 5,
"address": "dan5-tikv-1.dan5-tikv-peer.dan5.svc:20160",
"labels": [
{
"key": "host",
"value": "172.16.4.249"
}
],
"version": "3.0.3",
"state_name": "Up"
},
"status": {
"capacity": "440.9GiB",
"available": "427.7GiB",
"leader_count": 20,
"leader_weight": 1,
"leader_score": 20,
"leader_size": 20,
"region_count": 22,
"region_weight": 1,
"region_score": 112,
"region_size": 112,
"start_ts": "2019-11-18T08:01:55Z",
"last_heartbeat_ts": "2019-11-18T09:59:56.491943663Z",
"uptime": "1h58m1.491943663s"
}
}
Please help confirm if this is expected behavior since per design, the sst files should be created by region leaders.
BTW, checksum fails because br wants to check the files on remote tikv, not sure if I run the command missing some options:
/var/lib/tikv # ./br meta checksum -s local:///var/lib/tikv/backup/
[2019/11/18 09:47:46.458 +00:00] [INFO] [version.go:22] ["Welcome to Backup & Restore (BR)"]
[2019/11/18 09:47:46.458 +00:00] [INFO] [version.go:23] [BR] [release-version=]
[2019/11/18 09:47:46.458 +00:00] [INFO] [version.go:24] [BR] [git-hash=bbd137c28cefc595ba8a3c8c98db378028a7b553]
[2019/11/18 09:47:46.459 +00:00] [INFO] [version.go:25] [BR] [git-branch=master]
[2019/11/18 09:47:46.459 +00:00] [INFO] [version.go:26] [BR] [utc-build-time="2019-11-18 08:12:51"]
[2019/11/18 09:47:46.459 +00:00] [INFO] [version.go:43] [arguments] [ca=] [cert=] [help=false] [key=] [log-file=] [log-level=info] [pd=127.0.0.1:2379] [slow-log-file=] [status-addr=] [storage=local:///var/lib/tikv/backup/]
[2019/11/18 09:47:46.472 +00:00] [INFO] [meta.go:85] ["file info"] [table=sbtest1] [file=4_1005_22_49_default.sst] [crc64xor=8739028299443176440] [totalKvs=100000] [totalBytes=21200000] [startVersion=412632649740320770] [endVersion=412632649740320770] [startKey=dIAAAAAAAAAx] [endKey=dIAAAAAAAAAy]
Error: open /var/lib/tikv/backup/4_1005_22_49_default.sst: no such file or directory
open /var/lib/tikv/backup/4_1005_22_49_default.sst: no such file or directory
github.com/pingcap/errors.AddStack
/root/go/pkg/mod/github.com/pingcap/[email protected]/errors.go:174
github.com/pingcap/errors.Trace
/root/go/pkg/mod/github.com/pingcap/[email protected]/juju_adaptor.go:15
github.com/pingcap/br/cmd.NewMetaCommand.func2
/home/dev/br/cmd/meta.go:99
github.com/spf13/cobra.(*Command).execute
/root/go/pkg/mod/github.com/spf13/[email protected]/command.go:826
github.com/spf13/cobra.(*Command).ExecuteC
/root/go/pkg/mod/github.com/spf13/[email protected]/command.go:914
github.com/spf13/cobra.(*Command).Execute
/root/go/pkg/mod/github.com/spf13/[email protected]/command.go:864
main.main
/home/dev/br/main.go:56
runtime.main
/usr/local/go/src/runtime/proc.go:203
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1357