cloudfoundry / backup-and-restore-sdk-release

Backup and Restore SDK BOSH release

The Backup and Restore SDK BOSH release is used for two distinct things:

1. enabling release authors to incorporate database backup & restore functionality in their releases (See Database Backup and Restore)
2. enabling operators to configure their deployments which use external blobstores to be backed up and restored by BBR (See Blobstore Backup and Restore)

Slack: #bbr on https://slack.cloudfoundry.org

Pivotal Tracker: https://www.pivotaltracker.com/n/projects/1662777

CI Status

Backup and Restore SDK Release status

Database Backup and Restore

Name	Versions
MariaDB	10.2.x
MySQL	5.7.x, 8.0.x
Postgres	9.6.x, 10.x, 11.x

The SDK can use used against Postgres 9.4, but is not supported upstream by the Postgres community.

CI Status:

• GCP:
• AWS (RDS):
• Bosh Deployed:

Blobstore Backup and Restore

Supported Blobstores

CI Status:

Name
S3-Compatible
Azure
Google Cloud Storage

Developing

This repository using develop as the main branch, tested releases are tagged with their versions, and master branch represents the latest release.

Testing

Unit tests

The unit tests make use of ruby, go, and ginkgo. The easiest way to run them is to use our docker files, which provide versions of these dependencies that we've already tested. You can do this with:

make unit

If you want to run the tests using your local development tools, without using docker, you can run:

make local-unit

Individual targets exist for individual unit tests, like make docker-unit-blobstore-gcs and make local-unit-blobstore-gcs. Check the Makefile for all available targets

Contract tests

To run the Blobstore contract tests, you'll need to export the environment variables the particular test requires. Check the sdk-unit-blobstore pipeline task for details.

System tests

To run the system tests, you'll need to export the necessary environment variables that the particular test requires.

See the sdk-system-blobstore pipeline task and sdk-system-db pipeline task for more details.

You'll also need a bosh director with a few particular bosh releases deployed in it. Given a bosh director exists, the infrastructure pipeline can be set to deploy the necessary releases.

backup-and-restore-sdk-release's Issues

Failed to compile with jammy stemcell

I tried to use this release on jammy stemcell, did not compile.
manifest:

# db on vsphere
# bosh deploy -d db db.yml
---
name: db

releases:
  - name: "backup-and-restore-sdk"
    version: "1.18.36"
    url: "https://bosh.io/d/github.com/cloudfoundry-incubator/backup-and-restore-sdk-release?v=1.18.36"
    sha1: "7af40256f493476bce1463654bf655d2214b0d53"

stemcells:
  - alias: "default"
    os: "ubuntu-jammy"
    version: "0.0.7"

update:
  canaries: 2
  max_in_flight: 1
  canary_watch_time: 5000-60000
  update_watch_time: 5000-60000

instance_groups:
  - name: db
    instances: 1
    jobs:
      - consumes: {}
        name: database-backup-restorer
        provides: {}
        release: backup-and-restore-sdk
    vm_type: medium.disk
    stemcell: default
    persistent_disk_type: "10240"
    azs:
      - az-0
    networks:
      - default:
          - dns
          - gateway
        name: deployment-network

features:
  use_dns_addresses: true

Logs:

Task 345

Task 345 | 06:53:36 | Preparing deployment: Preparing deployment (00:00:04)
Task 345 | 06:53:40 | Preparing deployment: Rendering templates (00:00:01)
Task 345 | 06:53:41 | Preparing package compilation: Finding packages to compile (00:00:00)
Task 345 | 06:53:41 | Compiling packages: golang-1-linux/1a6489b2b33f55547b95cc524b932a0cfffb11a8f6c24479fe102d3494a4f030
Task 345 | 06:53:41 | Compiling packages: golang-1-linux/ad6aaba3771663154e241ba072646296fb4fa4edc2b7c55702f3fc67c1576378
Task 345 | 06:53:41 | Compiling packages: database-backup-restorer-mysql-5.6/01bf18f19277261bcccac9736d7634b49eb184a93cd6549b78f4e1d75eabe35a
Task 345 | 06:53:41 | Compiling packages: database-backup-restorer-mysql-5.7/782d833de90f48f0867dae4145696c77de2b7acf6bc9ce60cd7ec249a75cefac
Task 345 | 06:58:17 | Compiling packages: golang-1-linux/1a6489b2b33f55547b95cc524b932a0cfffb11a8f6c24479fe102d3494a4f030 (00:04:36)
Task 345 | 06:58:17 | Compiling packages: libpcre2/0d89cc293293ba02edb6389db603b1ac6e15d52d9f6fbc22e8c34b3100762587
Task 345 | 06:58:20 | Compiling packages: golang-1-linux/ad6aaba3771663154e241ba072646296fb4fa4edc2b7c55702f3fc67c1576378 (00:04:39)
Task 345 | 06:58:20 | Compiling packages: database-backup-restorer-postgres-13/c3effde85ab71ee41f368c5fa74ad663a04df7414a73c7d87d2be649dc1c223e
Task 345 | 06:58:47 | Compiling packages: database-backup-restorer-mysql-5.7/782d833de90f48f0867dae4145696c77de2b7acf6bc9ce60cd7ec249a75cefac (00:05:06)
                    L Error: Action Failed get_task: Task 364711f6-56f6-4875-532b-016de4568c66 result: Compiling package database-backup-restorer-mysql-5.7: Running packaging script: Running packaging script: Command exited with 1; Truncated stdout: -- Looking for include file strings.h - found
-- Looking for include file sys/ioctl.h
-- Looking for include file sys/ioctl.h - found
-- Looking for include file sys/mman.h
-- Looking for include file sys/mman.h - found
-- Looking for include file sys/resource.h
-- Looking for include file sys/resource.h - found
-- Looking for include file sys/select.h
-- Looking for include file sys/select.h - found
-- Looking for include file sys/socket.h
-- Looking for include file sys/socket.h - found
-- Looking for include files curses.h, term.h
-- Looking for include files curses.h, term.h - found
-- Looking for include file termios.h
-- Looking for include file termios.h - found
-- Looking for include file termio.h
-- Looking for include file termio.h - found
-- Looking for include file unistd.h
-- Looking for include file unistd.h - found
-- Looking for include file sys/wait.h
-- Looking for include file sys/wait.h - found
-- Looking for include file sys/param.h
-- Looking for include file sys/param.h - found
-- Looking for include file fnmatch.h
-- Looking for include file fnmatch.h - found
-- Looking for include file sys/un.h
-- Looking for include file sys/un.h - found
-- Looking for include file sys/devpoll.h
-- Looking for include file sys/devpoll.h - not found
-- Looking for include file sys/epoll.h
-- Looking for include file sys/epoll.h - found
-- Looking for TAILQ_FOREACH
-- Looking for TAILQ_FOREACH - found
-- Looking for _aligned_malloc
-- Looking for _aligned_malloc - not found
-- Looking for backtrace
-- Looking for backtrace - found
-- Looking for printstack
-- Looking for printstack - not found
-- Looking for index
-- Looking for index - found
-- Looking for clock_gettime
-- Looking for clock_gettime - found
-- Looking for cuserid
-- Looking for cuserid - found
-- Looking for directio
-- Looking for directio - not found
-- Looking for ftruncate
-- Looking for ftruncate - found
-- Looking for compress
-- Looking for compress - not found
-- Looking for crypt
-- Looking for crypt - found
-- Looking for dlopen
-- Looking for dlopen - found
-- Looking for fchmod
-- Looking for fchmod - found
-- Looking for fcntl
-- Looking for fcntl - found
-- Looking for fdatasync
-- Looking for fdatasync - found
-- Looking for fdatasync
-- Looking for fdatasync - found
-- Looking for fedisableexcept
-- Looking for fedisableexcept - found
-- Looking for fseeko
-- Looking for fseeko - found
-- Looking for fsync
-- Looking for fsync - found
-- Looking for gethostbyaddr_r
-- Looking for gethostbyaddr_r - found
-- Looking for gethrtime
-- Looking for gethrtime - not found
-- Looking for getnameinfo
-- Looking for getnameinfo - found
-- Looking for getpass
-- Looking for getpass - found
-- Looking for getpassphrase
-- Looking for getpassphrase - not found
-- Looking for getpwnam
-- Looking for getpwnam - found
-- Looking for getpwuid
-- Looking for getpwuid - found
-- Looking for getrlimit
-- Looking for getrlimit - found
-- Looking for getrusage
-- Looking for getrusage - found
-- Looking for initgroups
-- Looking for initgroups - found
-- Looking for issetugid
-- Looking for issetugid - not found
-- Looking for getuid
-- Looking for getuid - found
-- Looking for geteuid
-- Looking for geteuid - found
-- Looking for getgid
-- Looking for getgid - found
-- Looking for getegid
-- Looking for getegid - found
-- Looking for lstat
-- Looking for lstat - found
-- Looking for madvise
-- Looking for madvise - found
-- Looking for malloc_info
-- Looking for malloc_info - found
-- Looking for memrchr
-- Looking for memrchr - found
-- Looking for mlock
-- Looking for mlock - found
-- Looking for mlockall
-- Looking for mlockall - found
-- Looking for mmap64
-- Looking for mmap64 - found
-- Looking for poll
-- Looking for poll - found
-- Looking for posix_fallocate
-- Looking for posix_fallocate - found
-- Looking for posix_memalign
-- Looking for posix_memalign - found
-- Looking for pread
-- Looking for pread - found
-- Looking for pthread_condattr_setclock
-- Looking for pthread_condattr_setclock - found
-- Looking for pthread_sigmask
-- Looking for pthread_sigmask - found
-- Looking for readlink
-- Looking for readlink - found
-- Looking for realpath
-- Looking for realpath - found
-- Looking for setfd
-- Looking for setfd - not found
-- Looking for sigaction
-- Looking for sigaction - found
-- Looking for sleep
-- Looking for sleep - found
-- Looking for stpcpy
-- Looking for stpcpy - found
-- Looking for stpncpy
-- Looking for stpncpy - found
-- Looking for strlcpy
-- Looking for strlcpy - not found
-- Looking for strndup
-- Looking for strndup - found
-- Looking for strnlen
-- Looking for strnlen - found
-- Looking for strlcat
-- Looking for strlcat - not found
-- Looking for strsignal
-- Looking for strsignal - found
-- Looking for fgetln
-- Looking for fgetln - not found
-- Looking for strsep
-- Looking for strsep - found
-- Looking for tell
-- Looking for tell - not found
-- Looking for vasprintf
-- Looking for vasprintf - found
-- Looking for memalign
-- Looking for memalign - found
-- Looking for nl_langinfo
-- Looking for nl_langinfo - found
-- Looking for ntohll
-- Looking for ntohll - not found
-- Looking for clock_gettime
-- Looking for clock_gettime - found
-- Looking for epoll_create
-- Looking for epoll_create - found
-- Looking for inet_ntop
-- Looking for inet_ntop - found
-- Looking for kqueue
-- Looking for kqueue - not found
-- Looking for timeradd
-- Looking for timeradd - found
-- Looking for timerclear
-- Looking for timerclear - found
-- Looking for timercmp
-- Looking for timercmp - found
-- Looking for timerisset
-- Looking for timerisset - found
-- Looking for include file sys/time.h
-- Looking for include file sys/time.h - found
-- Looking for include file sys/times.h
-- Looking for include file sys/times.h - found
-- Looking for times
-- Looking for times - found
-- Looking for gettimeofday
-- Looking for gettimeofday - found
-- Looking for lrand48
-- Looking for lrand48 - found
-- Looking for TIOCGWINSZ
-- Looking for TIOCGWINSZ - found
-- Looking for FIONREAD
-- Looking for FIONREAD - found
-- Looking for FIONREAD
-- Looking for FIONREAD - not found
-- Looking for SIGEV_THREAD_ID
-- Looking for SIGEV_THREAD_ID - found
-- Looking for SIGEV_PORT
-- Looking for SIGEV_PORT - not found
-- Looking for log2
-- Looking for log2 - found
-- Looking for isinf
-- Looking for isinf - found
-- Performing Test HAVE_CXX_ISINF
-- Performing Test HAVE_CXX_ISINF - Success
-- Looking for timer_create
-- Looking for timer_create - found
-- Looking for timer_settime
-- Looking for timer_settime - found
-- Looking for kqueue
-- Looking for kqueue - not found
-- Looking for EVFILT_TIMER
-- Looking for EVFILT_TIMER - not found
-- Check size of char *
-- Check size of char * - done
-- Check size of long
-- Check size of long - done
-- Check size of short
-- Check size of short - done
-- Check size of int
-- Check size of int - done
-- Check size of long long
-- Check size of long long - done
-- Check size of off_t
-- Check size of off_t - done
-- Check size of time_t
-- Check size of time_t - done
-- Check size of struct timespec
-- Check size of struct timespec - done
-- Check size of uint
-- Check size of uint - done
-- Check size of ulong
-- Check size of ulong - done
-- Check size of u_int32_t
-- Check size of u_int32_t - done
-- Performing Test HAVE_VISIBILITY_HIDDEN
-- Performing Test HAVE_VISIBILITY_HIDDEN - Success
-- Checking stack direction : -1
-- Looking for include files time.h, sys/time.h
-- Looking for include files time.h, sys/time.h - found
-- Looking for O_NONBLOCK
-- Looking for O_NONBLOCK - found
-- Performing Test HAVE_PAUSE_INSTRUCTION
-- Performing Test HAVE_PAUSE_INSTRUCTION - Success
-- Performing Test HAVE_BUILTIN_UNREACHABLE
-- Performing Test HAVE_BUILTIN_UNREACHABLE - Success
-- Performing Test HAVE_BUILTIN_EXPECT
-- Performing Test HAVE_BUILTIN_EXPECT - Success
-- Performing Test HAVE_BUILTIN_STPCPY
-- Performing Test HAVE_BUILTIN_STPCPY - Success
-- Performing Test HAVE_GCC_ATOMIC_BUILTINS
-- Performing Test HAVE_GCC_ATOMIC_BUILTINS - Success
-- Performing Test HAVE_GCC_SYNC_BUILTINS
-- Performing Test HAVE_GCC_SYNC_BUILTINS - Success
-- Looking for netinet/in6.h
-- Looking for netinet/in6.h - not found
-- Check size of struct sockaddr_in6
-- Check size of struct sockaddr_in6 - done
-- Check size of struct in6_addr
-- Check size of struct in6_addr - done
-- Performing Test HAVE_SOCKADDR_STORAGE_SS_FAMILY
-- Performing Test HAVE_SOCKADDR_STORAGE_SS_FAMILY - Success
-- Performing Test HAVE_SOCKADDR_IN_SIN_LEN
-- Performing Test HAVE_SOCKADDR_IN_SIN_LEN - Failed
-- Performing Test HAVE_SOCKADDR_IN6_SIN6_LEN
-- Performing Test HAVE_SOCKADDR_IN6_SIN6_LEN - Failed
-- Performing Test HAVE_PTHREAD_THREADID_NP
-- Performing Test HAVE_PTHREAD_THREADID_NP - Failed
-- Performing Test HAVE_IMPLICIT_DEPENDENT_NAME_TYPING
-- Performing Test HAVE_IMPLICIT_DEPENDENT_NAME_TYPING - Failed
-- Looking for chown
-- Looking for chown - found
-- Looking for include file numa.h
-- Looking for include file numa.h - not found
-- Looking for include file numaif.h
-- Looking for include file numaif.h - not found
-- NUMA library missing or required version not available
-- Check size of socklen_t
-- Check size of socklen_t - failed
-- Check size of off64_t
-- Check size of off64_t - done
-- Looking for unistd.h
-- Looking for unistd.h - found
-- OPENSSL_INCLUDE_DIR = /usr/include
-- OPENSSL_LIBRARY = /usr/lib/x86_64-linux-gnu/libssl.so
-- CRYPTO_LIBRARY = /usr/lib/x86_64-linux-gnu/libcrypto.so
-- OPENSSL_MAJOR_VERSION = 
-- OPENSSL_MINOR_VERSION = 
-- OPENSSL_FIX_VERSION = 
-- Looking for SHA512_DIGEST_LENGTH
-- Looking for SHA512_DIGEST_LENGTH - found
-- 
Cannot find appropriate system libraries for WITH_SSL=system.
Make sure you have specified a supported SSL version. 
Valid options are : 
system (use the OS openssl library), 
yes (synonym for system), 
</path/to/custom/openssl/installation>

-- Configuring incomplete, errors occurred!
See also "/var/vcap/data/compile/database-backup-restorer-mysql-5.7/mysql-5.7.36/bld/CMakeFiles/CMakeOutput.log".
See also "/var/vcap/data/compile/database-backup-restorer-mysql-5.7/mysql-5.7.36/bld/CMakeFiles/CMakeError.log".
, Stderr: + MYSQL_VERSION=5.7.36
++ pwd
+ MY_DIR=/var/vcap/data/compile/database-backup-restorer-mysql-5.7
++ mktemp
+ patch_file=/var/vcap/data/tmp/tmp.x5jQemIMyf
+ echo '--- mysys/mf_iocache2.c 2019-08-05 09:42:13.000000000 +0000
+++ mysys/mf_iocache2.c.patched 2019-08-05 09:41:53.000000000 +0000
@@ -17,6 +17,7 @@
   More functions to be used with IO_CACHE files
 */

+#include "mysql/psi/mysql_file.h"
 #include "mysys_priv.h"
 #include "my_sys.h"
 #include <m_string.h>
'
+ tar xzf mysql/mysql-5.7.36.tar.gz
+ set -e
+ cd mysql-5.7.36
+ patch mysys/mf_iocache2.c /var/vcap/data/tmp/tmp.x5jQemIMyf
+ mkdir bld
+ cd bld
+ cmake .. -DCMAKE_INSTALL_PREFIX=/var/vcap/packages/database-backup-restorer-mysql-5.7 -DWITHOUT_SERVER=ON -DWITH_SSL=system -DWITH_WSREP=ON -DWITH_INNODB_DISALLOW_WRITES=1 -DWITH_PCRE=bundled -DWITH_BOOST=/var/vcap/packages/database-backup-restorer-boost/boost_1_59_0
CMake Deprecation Warning at CMakeLists.txt:30 (CMAKE_MINIMUM_REQUIRED):
  Compatibility with CMake < 2.8.12 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.


CMake Deprecation Warning at CMakeLists.txt:35 (CMAKE_POLICY):
  The OLD behavior for policy CMP0018 will be removed from a future version
  of CMake.

  The cmake-policies(7) manual explains that the OLD behaviors of all
  policies are deprecated and that a policy should be set to OLD only under
  specific short-term circumstances.  Projects should be ported to the NEW
  behavior and not rely on setting a policy to OLD.


CMake Deprecation Warning at CMakeLists.txt:41 (CMAKE_POLICY):
  The OLD behavior for policy CMP0022 will be removed from a future version
  of CMake.

  The cmake-policies(7) manual explains that the OLD behaviors of all
  policies are deprecated and that a policy should be set to OLD only under
  specific short-term circumstances.  Projects should be ported to the NEW
  behavior and not rely on setting a policy to OLD.


CMake Deprecation Warning at CMakeLists.txt:48 (CMAKE_POLICY):
  The OLD behavior for policy CMP0045 will be removed from a future version
  of CMake.

  The cmake-policies(7) manual explains that the OLD behaviors of all
  policies are deprecated and that a policy should be set to OLD only under
  specific short-term circumstances.  Projects should be ported to the NEW
  behavior and not rely on setting a policy to OLD.


CMake Deprecation Warning at CMakeLists.txt:49 (CMAKE_POLICY):
  The OLD behavior for policy CMP0042 will be removed from a future version
  of CMake.

  The cmake-policies(7) manual explains that the OLD behaviors of all
  policies are deprecated and that a policy should be set to OLD only under
  specific short-term circumstances.  Projects should be ported to the NEW
  behavior and not rely on setting a policy to OLD.


CMake Deprecation Warning at CMakeLists.txt:62 (CMAKE_POLICY):
  The OLD behavior for policy CMP0075 will be removed from a future version
  of CMake.

  The cmake-policies(7) manual explains that the OLD behaviors of all
  policies are deprecated and that a policy should be set to OLD only under
  specific short-term circumstances.  Projects should be ported to the NEW
  behavior and not rely on setting a policy to OLD.


CMake Deprecation Warning at zlib/CMakeLists.txt:24 (cmake_minimum_required):
  Compatibility with CMake < 2.8.12 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.


CMake Error at cmake/ssl.cmake:63 (MESSAGE):
  Please install the appropriate openssl developer package.

Call Stack (most recent call first):
  cmake/ssl.cmake:280 (FATAL_SSL_NOT_FOUND_ERROR)
  CMakeLists.txt:568 (MYSQL_CHECK_SSL)

test issue - to test tracker integration

Can't dump Mysql >= 8.0.32 and Mysql >= 5.7.40

https://bugs.launchpad.net/ubuntu/+source/mysql-5.7/+bug/2003866/comments/9

Using --single-transaction with mysqldump version 8.0.32 required either the RELOAD or FLUSH_TABLES privilege. This requirement now applies only when both gtid_mode=ON (default OFF) and with --set-gtid-purged = ON|AUTO (default AUTO).

This should have started failing when MySQL Docker tag 8.0-debian got updated 20 days ago to point to 8.0.32-debian:
https://hub.docker.com/layers/library/mysql/8.0.32-debian/images/sha256-3e54e01ca605ee64672bf588cd1782a669cb387954593ccd1c9dcd551598d1df?context=explore
By that time GHActions were already failing so we never noticed.

It seems bumping mysql blob from mysql-8.0.32 to mysql-8.0.33 fixes this issues (untested)

A bump is also needed for mysql-5.7.40 to 5.7.42

[Feature Request] Support Azure Blob Storage in Azure sovereign clouds

Firstly, thanks for supporting azure blob storage. It's awesome. However, it seems that only the Azure public cloud blob storage is supported.
In other sovereign clouds (Azure Germany Cloud, Azure China Cloud, Azure US Gov), the storage endpoint postfix is different. E.g. blob.core.cloudapi.de for Azure Germany Cloud.
Is it on the road map to support the blob storage in Azure sovereign clouds? Thanks.

bbr 1.18.53 failed at gettig blob

Hi, we noticed a bosh deployment failure due to newer version of bbr sdk v1.18.53.

Task 28153 | 15:01:22 | Error: Action Failed get_task: Task fadc4046-e2c7-4ee1-40c7-62b4d2ce5a20 result: Preparing apply spec: Preparing package database-backup-restorer-postgres-9.6: Fetching package blob: Getting blob from inner blobstore: Getting blob from inner blobstore: Shelling out to bosh-blobstore-dav cli: Running command: 'bosh-blobstore-dav -c /var/vcap/bosh/etc/blobstore-dav.json get f9a1e8f0-f265-4754-50e5-c8aaee913ca0 /var/vcap/data/tmp/bosh-blobstore-externalBlobstore-Get3179005383', stdout: 'Error running app - Getting dav blob f9a1e8f0-f265-4754-50e5-c8aaee913ca0: Wrong response code: 404; body: <html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>
', stderr: '': exit status 1

Bosh Backup & Restore SDK Cannot Restore Vanilla RDS Backup

The BOSH team recently began having BBR tests fail in our pipeline due to a
MySQL RDS backup/restore test that failed with the error message "ERROR 1227
(42000) at line 18: Access denied; you need (at least one of) the SUPER
privilege(s) for this operation ". Careful investigation showed that the line
that was causing the failure was trying to unset GTID for the restore session
by running SET @@SESSION.SQL_LOG_BIN=0;. It turns out that AWS (recently
enabled GTID for RDS)[https://forums.aws.amazon.com/ann.jspa?annID=6203] in
early October of 2018 which is likely what began our test failures. The
implication of this change is that a vanilla MySQL RDS is created with a user
that has permission to run mysqldump but cannot use that SQL dump file to
restore since it needs SUPER privileges to unset GTID for the restore session.
There are four ways to work around this that we have:

1. Change the BBR SDK to use --set-gtid-purged=OFF [1]
2. Use an older version of MySQL that does not support GTID for our own tests (5.5 seems to work)
3. Change our RDS parameter group to set gtid-mode to OFF (default with no DB replica and no backups for us is OFF_PERMISSIVE which results in this failure anyway)
4. Switch from using RDS to using a VM running MySQL and use the root user to run BBR which should be able to run this sort of command anyway.

-- @pivotal-mp && @jaresty from the BOSH team

BBR Unable to backup Bosh Director using mysql 8.0.x

Hi,

I am getting the error below when backing up Bosh director using external mysql db version 8.0.x.
From the Readme, I understand 8.0.X is supported?
Appreciate any guidance in this matter.

Error attempting to run backup for job director on bosh/0: 2022/08/10 07:42:18 MYSQL server version 8.0.28
2022/08/10 07:42:18 unsupported version of mysql: 8.0 - exit code 1

MySQL BBR restore doesn't print any output during a successful run

On a successful run of the BBR restore script with a MySQL database, no output is printed to STDOUT/ERR. The only way to tell it succeeded is to check the exit code. Would be nice to see confirmation that the restore went as expected.

Also, while the backup script does print plenty of output, it is not obvious whether it was successful or not.

Thanks!

`cf-deployment` opsfile link in README is broken

The actual link should be https://github.com/cloudfoundry/cf-deployment/blob/master/operations/experimental/enable-backup-restore.yml

Unable to backup Postgres 11.1 due to parsing errors

Hey,

@YoussB and I were trying to perform backups of Concourse (using Postgres 11.1 - release v33) and running bbr deploy [...] backup [...] returned the following error:

2018/12/27 18:05:39 Unable to check version of Postgres: could not parse semver: "11.1" - exit code 1

Our guess is that there's a parsing issue happening under version_detector.go:

https://github.com/cloudfoundry-incubator/backup-and-restore-sdk-release/blob/c735374dd73f26932afc39510bc0156902c1ad36/src/github.com/cloudfoundry-incubator/database-backup-restore/postgres/version_detector.go#L28-L30

Looking at ci, there's no v33 (pg 11.1) there, so I guess that's because it's supposed to not be supported yet - is that right?

Please let me know if there's any way that we can help 😁

Thanks very much!

Support Bionic stemcell

Hi all,

we tried to compile the backup-and-restore-sdk release for Ubuntu Bionic and ran into the following error:

'TLSv1_client_method' is deprecated [-Wdeprecated-declarations]
                                          TLSv1_client_method() :
                                          ^~~~~~~~~~~~~~~~~~~
In file included from /usr/include/openssl/e_os2.h:13:0,
                 from /usr/include/openssl/ssl.h:15,

It seems that the openssl version of Bionic is not compatible with mysql 5.6 (tested with 1.18.3 & 1.18.5).

Thanks!
cc @StefanWutz

---

To reproduce:

bosh -e bosh export-release -d compilation-workspace backup-and-restore-sdk/1.18.3 ubuntu-bionic/0.18

name: compilation-workspace

releases:
- name: backup-and-restore-sdk
  version: "1.18.3"

stemcells:
- alias: default
  os: ubuntu-bionic
  version: latest

instance_groups: []

update:
  canaries: 1
  max_in_flight: 1
  canary_watch_time: 1000-90000
  update_watch_time: 1000-90000
~

---

Full error log:

Task XXXX | 13:52:48 | Compiling packages: database-backup-restorer-mysql-5.6/1510ee0c2b36d03063e9d34cf4e82a30666cb0787802c2b80ec79d363cfec7a4 (00:06:01)
                      L Error: Action Failed get_task: Task e421a611-8de3-422d-505f-16aff1c25546 result: Compiling package database-backup-restorer-mysql-5.6: Running packaging script: Running packaging script: Command exited with 2; Truncated stdout: -- Looking for timeradd - not found
[...]
-- Build files have been written to: /var/vcap/data/compile/database-backup-restorer-mysql-5.6/mysql-5.6.45/bld
/var/vcap/data/compile/database-backup-restorer-mysql-5.6/mysql-5.6.45/cmd-line-utils/libedit/terminal.c: In function 'terminal_alloc':
/var/vcap/data/compile/database-backup-restorer-mysql-5.6/mysql-5.6.45/cmd-line-utils/libedit/terminal.c:366:28: warning: comparison between pointer and zero character constant [-Wpointer-compare]
   if (*tmp != NULL && *tmp != '\0' && *tmp != *str) {
                            ^~
/var/vcap/data/compile/database-backup-restorer-mysql-5.6/mysql-5.6.45/cmd-line-utils/libedit/terminal.c:366:23: note: did you mean to dereference the pointer?
   if (*tmp != NULL && *tmp != '\0' && *tmp != *str) {
                       ^
/var/vcap/data/compile/database-backup-restorer-mysql-5.6/mysql-5.6.45/vio/vio.c: In function 'vio_end':
/var/vcap/data/compile/database-backup-restorer-mysql-5.6/mysql-5.6.45/vio/vio.c:387:3: warning: 'ERR_remove_state' is deprecated [-Wdeprecated-declarations]
   ERR_remove_state(0);
   ^~~~~~~~~~~~~~~~
In file included from /usr/include/openssl/e_os2.h:13:0,
                 from /usr/include/openssl/ssl.h:15,
                 from /var/vcap/data/compile/database-backup-restorer-mysql-5.6/mysql-5.6.45/include/violite.h:143,
                 from /var/vcap/data/compile/database-backup-restorer-mysql-5.6/mysql-5.6.45/vio/vio_priv.h:26,
                 from /var/vcap/data/compile/database-backup-restorer-mysql-5.6/mysql-5.6.45/vio/vio.c:23:
/usr/include/openssl/err.h:260:1: note: declared here
 DEPRECATEDIN_1_0_0(void ERR_remove_state(unsigned long pid))
 ^
/var/vcap/data/compile/database-backup-restorer-mysql-5.6/mysql-5.6.45/vio/viosslfactories.c: In function 'get_dh2048':
/var/vcap/data/compile/database-backup-restorer-mysql-5.6/mysql-5.6.45/vio/viosslfactories.c:71:7: error: dereferencing pointer to incomplete type 'DH {aka struct dh_st}'
     dh->p=BN_bin2bn(dh2048_p,sizeof(dh2048_p),NULL);
       ^~
/var/vcap/data/compile/database-backup-restorer-mysql-5.6/mysql-5.6.45/vio/viosslfactories.c: In function 'new_VioSSLFd':
/var/vcap/data/compile/database-backup-restorer-mysql-5.6/mysql-5.6.45/vio/viosslfactories.c:233:42: warning: 'TLSv1_client_method' is deprecated [-Wdeprecated-declarations]
                                          TLSv1_client_method() :
                                          ^~~~~~~~~~~~~~~~~~~
In file included from /usr/include/openssl/e_os2.h:13:0,
                 from /usr/include/openssl/ssl.h:15,
                 from /var/vcap/data/compile/database-backup-restorer-mysql-5.6/mysql-5.6.45/include/violite.h:143,
                 from /var/vcap/data/compile/database-backup-restorer-mysql-5.6/mysql-5.6.45/vio/vio_priv.h:26,
                 from /var/vcap/data/compile/database-backup-restorer-mysql-5.6/mysql-5.6.45/vio/viosslfactories.c:16:
/usr/include/openssl/ssl.h:1854:1: note: declared here
 DEPRECATEDIN_1_1_0(__owur const SSL_METHOD *TLSv1_client_method(void))
 ^
/var/vcap/data/compile/database-backup-restorer-mysql-5.6/mysql-5.6.45/vio/viosslfactories.c:234:42: warning: 'TLSv1_server_method' is deprecated [-Wdeprecated-declarations]
                                          TLSv1_server_method())))
                                          ^~~~~~~~~~~~~~~~~~~
In file included from /usr/include/openssl/e_os2.h:13:0,
                 from /usr/include/openssl/ssl.h:15,
                 from /var/vcap/data/compile/database-backup-restorer-mysql-5.6/mysql-5.6.45/include/violite.h:143,
                 from /var/vcap/data/compile/database-backup-restorer-mysql-5.6/mysql-5.6.45/vio/vio_priv.h:26,
                 from /var/vcap/data/compile/database-backup-restorer-mysql-5.6/mysql-5.6.45/vio/viosslfactories.c:16:
/usr/include/openssl/ssl.h:1853:1: note: declared here
 DEPRECATEDIN_1_1_0(__owur const SSL_METHOD *TLSv1_server_method(void))
 ^
make[2]: *** [vio/CMakeFiles/vio.dir/viosslfactories.c.o] Error 1
make[1]: *** [vio/CMakeFiles/vio.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
make: *** [all] Error 2
, Stderr: + MYSQL_VERSION=5.6.45
++ pwd
+ MY_DIR=/var/vcap/data/compile/database-backup-restorer-mysql-5.6
+ tar xzf mysql/mysql-5.6.45.tar.gz
+ set -e
+ cd mysql-5.6.45
+ mkdir bld
+ cd bld
+ cmake .. -DCMAKE_INSTALL_PREFIX=/var/vcap/packages/database-backup-restorer-mysql-5.6 -DWITHOUT_SERVER=ON -DWITH_SSL=system -DWITH_WSREP=ON -DWITH_INNODB_DISALLOW_WRITES=1 -DWITH_PCRE=bundled

Does not support postgresql 16

Error as of version 1.19.12:

  + /var/vcap/jobs/database-backup-restorer/bin/backup --config /var/vcap/jobs/bbr-credhubdb/config/bbr.json --artifact-file /var/vcap/store/bbr-backup/bbr-credhubdb//credhubdb_dump
  2024/05/14 20:21:26 unsupported version of postgresql: 16.3 - exit code 1

Does the latest version support postgresql 16? If so, the README needs to be updated. Also: Currently the README says it supports postgres "9.6.x, 10.x, 11.x", but 1.19.12 is working with postgres 15 too.

1.7.0 release fails when used as a github release in Concourse

We are using the following Concourse resource

- name: backup-and-restore-sdk-release
  type: github-release
  source:
    owner: cloudfoundry-incubator
    repository: backup-and-restore-sdk-release
    access_token: ((token))

When using the v1.7.0 tag the artefact downloaded fails to process as it is an invalid tarball.

Upon inspecting the tarball. We see the contents is a follows:

<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Key>88870798/f6f2ff46-5aa4-11e8-90a0-2f9beb9bbc85</Key><RequestId>1BB4DD73A15A1E5F</RequestId><HostId>XEevxyEqRTYvsCmEuPdrGXa5Lay/pbvMGG/YEpycByCICwE50LVGZkcFBPqkEg/C4K5RrCm0Ifs=</HostId></Error>root@14b34d60-f1c8-4ab4-705e-def0c4472b47

Fail to compile v1.18.37

Hello,

My CI has trouble compiling backup-and-restore-sdk-release v1.18.37 against default stemcell xenial/1.x ubuntu-bionic/1.x

I got the following error:

+ go build -mod vendor -o /var/vcap/packages/s3-blobstore-backup-restorer/bin/s3-blobstore-backup-restore ./cmd/s3-blobstore-backup-restore
go: inconsistent vendoring in /var/vcap/data/compile/s3-blobstore-backup-restorer/s3-blobstore-backup-restore:
	s3-blobstore-backup-restore: is marked as replaced in vendor/modules.txt, but not replaced in go.mod

	To ignore the vendor directory, use -mod=readonly or -mod=mod.
	To sync the vendor directory, run:
		go mod vendor

Restore script on postgres errors restoring Cloud Controller DB

When we run the restore script as the non-admin cloud_controller Postgres user, we get a variety of permissions-related errors when it tries to drop/create extensions and the "public" schema.

Since these extensions already exist and the rest of the restore succeeds, the restore is successful despite the script exiting non-zero.

These were the steps we took:

1. Deployed a fresh CF with Postgres
2. Took a back up of the cloud_controller database using the bbr backup script. We did not see any errors during this process.
3. Stopped Cloud Controller bosh jobs (api, cc-clock, cc-worker)
4. Dropped the cloud_controller database and did a monit restart postgres to have the Postgres job recreate the database/user with the appropriate permissions/extensions.
5. Started the api bosh job to get Cloud Controller to run its migrations and restore the schema (since this is required?)
6. Ran the bbr restore script using the cloud_controller Postgres user

This returned the following errors:

pg_restore: dropping COMMENT EXTENSION pgcrypto
pg_restore: dropping EXTENSION pgcrypto
pg_restore: [archiver (db)] Error while PROCESSING TOC:
pg_restore: [archiver (db)] Error from TOC entry 2; 3079 16475 EXTENSION pgcrypto
pg_restore: [archiver (db)] could not execute query: ERROR:  must be owner of extension pgcrypto
    Command was: DROP EXTENSION pgcrypto;

pg_restore: dropping COMMENT EXTENSION citext
pg_restore: dropping EXTENSION citext
pg_restore: [archiver (db)] Error from TOC entry 3; 3079 16391 EXTENSION citext
pg_restore: [archiver (db)] could not execute query: ERROR:  must be owner of extension citext
    Command was: DROP EXTENSION citext;

pg_restore: dropping COMMENT EXTENSION plpgsql
pg_restore: dropping EXTENSION plpgsql
pg_restore: [archiver (db)] Error from TOC entry 1; 3079 12393 EXTENSION plpgsql
pg_restore: [archiver (db)] could not execute query: ERROR:  must be owner of extension plpgsql
    Command was: DROP EXTENSION plpgsql;

pg_restore: dropping COMMENT SCHEMA public
pg_restore: dropping SCHEMA public
pg_restore: [archiver (db)] Error from TOC entry 5; 2615 2200 SCHEMA public vcap
pg_restore: [archiver (db)] could not execute query: ERROR:  must be owner of schema public
    Command was: DROP SCHEMA public;

pg_restore: creating SCHEMA public
pg_restore: [archiver (db)] could not execute query: ERROR:  permission denied for database cloud_controller
    Command was: CREATE SCHEMA public;



pg_restore: creating COMMENT SCHEMA public
pg_restore: [archiver (db)] Error from TOC entry 3152; 0 0 COMMENT SCHEMA public vcap
pg_restore: [archiver (db)] could not execute query: ERROR:  must be owner of schema public
    Command was: COMMENT ON SCHEMA public IS 'standard public schema';



pg_restore: creating EXTENSION plpgsql
pg_restore: creating COMMENT EXTENSION plpgsql
pg_restore: [archiver (db)] Error from TOC entry 3153; 0 0 COMMENT EXTENSION plpgsql
pg_restore: [archiver (db)] could not execute query: ERROR:  must be owner of extension plpgsql
    Command was: COMMENT ON EXTENSION plpgsql IS 'PL/pgSQL procedural language';



pg_restore: creating EXTENSION citext
pg_restore: creating COMMENT EXTENSION citext
pg_restore: [archiver (db)] Error from TOC entry 3154; 0 0 COMMENT EXTENSION citext
pg_restore: [archiver (db)] could not execute query: ERROR:  must be owner of extension citext
    Command was: COMMENT ON EXTENSION citext IS 'data type for case-insensitive character strings';



pg_restore: creating EXTENSION pgcrypto
pg_restore: creating COMMENT EXTENSION pgcrypto
pg_restore: [archiver (db)] Error from TOC entry 3155; 0 0 COMMENT EXTENSION pgcrypto
pg_restore: [archiver (db)] could not execute query: ERROR:  must be owner of extension pgcrypto
    Command was: COMMENT ON EXTENSION pgcrypto IS 'cryptographic functions';

1.7.0 release fails when used as a github release in Concourse

We are using the following Concourse resource

- name: backup-and-restore-sdk-release
  type: github-release
  source:
    owner: cloudfoundry-incubator
    repository: backup-and-restore-sdk-release
    access_token: ((token))

When using the v1.7.0 tag the artefact downloaded fails to process as it is an invalid tarball.

Upon inspecting the tarball. We see the contents is a follows:

<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Key>88870798/f6f2ff46-5aa4-11e8-90a0-2f9beb9bbc85</Key><RequestId>1BB4DD73A15A1E5F</RequestId><HostId>XEevxyEqRTYvsCmEuPdrGXa5Lay/pbvMGG/YEpycByCICwE50LVGZkcFBPqkEg/C4K5RrCm0Ifs=</HostId></Error>root@14b34d60-f1c8-4ab4-705e-def0c4472b47

Could not compile release v1.18.12: ftp.pcre.org unreachable

Hello,

I'm having trouble compiling latest (v1.18.12) version of backup-and-restore-sdk-release in my environment.

In package database-backup-restorer-mariadb, while compiling mariadb v10.6.3, the cmake tries to fetch pcre2 library from ftp.pcre.org. My compilation vms do not have direct internet access thus the process fails.

Cmake uses libcurl internally therefore should be compatible with proxy environment settings. Does anyone know a way to set proxy settings to bosh compilation vms ?

Alternatively, would it be possible to ship the release with libpcre2 pre-installed by another package and with mariadb compiled with -DWITH_PRCE=system ?

Thanks a lot,

scripts/autobump-libpcre2.sh might be simplified

It seems possible that this function is totally unneeded to

https://github.com/cloudfoundry-incubator/backup-and-restore-sdk-release/blob/1f315b6ebdcd92592056f78841b9afb1196a0893/scripts/autobump-libpcre2.sh#L19-L22

I think so because when the function is not defined it fallbacks to this implementation

https://github.com/pivotal/cryogenics-concourse-tasks/blob/9944f6ba4e57c67b4672a837c90d2b4b54aef862/deps-automation/bump-package/task.sh#L32-L34

The fallback implementation is not equivalent, since it allows more combinations to be valid.
But I think that every possible combination accepted by scripts/autobump-libpcre2.sh is also accepted by the fallback implementation.

The fallback implementation seems to be a superset of the implementation in scripts/autobump-libpcre2.sh

Please configure GITBOT

Pivotal uses GITBOT to synchronize Github issues and pull requests with Pivotal Tracker.
Please add your new repo to the GITBOT config-production.yml in the Gitbot configuration repo.
If you don't have access you can send an ask ticket to the CF admins. We prefer teams to submit their changes via a pull request.

Steps:

• Fork this repo: cfgitbot-config
• Add your project to config-production.yml file
• Submit a PR

If there are any questions, please reach out to [email protected].