orange-cloudfoundry / mongodb-boshrelease

Mongodb Bosh Release

Contents

• Purpose
• What should the Release do
• Packages versions summary
• Installation
  • Clone the repository
  • Deployment manifests
    • Variables
  • Operation files
  • Deployment
• Broker
  • Mongodb Broker
  • Mongodb Broker Smoke Tests
• Configuring CF to use Mongodb service
  • Available Plans
  • Broker registration
  • Service provisioning
  • Service binding
  • Service unbinding
  • Service deprovisioning
• Contributing
  • Ruby Env Setup

Purpose

This project is a Mongodb Bosh release. The blobs are the provided ones from the mongodb community and are not compiled anymore. So the release can now only be deployed on an ubuntu stemcell.

This version exclude the rocksdb engine, which is not supported anymore.

What should the Release do

• Configure a standalone or a set of standalone servers
• Configure a replica set
• Configure a sharded cluster including config server and mongos
• Complete requirements for mongodb servers (production notes)
• Install mongodb component (shell / tools / mongod)
• Authentification using bosh/credhub generated passwords (could be disable)

Packages versions summary

• Mongodb database and modules version

Package	Version	Note
mongodb	3.6.12
mongo-rocks	3.4.7	Not supported anymore
mongo-tools	3.6.12
rocksdb	3.4.7	Not supported anymore

Installation

Clone the repository

git clone --recursive https://github.com/orange-cloudfoundry/mongodb-boshrelease.git

Deployment manifests

Two different base manifests are provided for single replicaset or sharded deployment and can be found in the manifests directory

Variables

Release include a deployment-vars-template.yml file, which includes all the needed variables for the deployment. Just copy and fill the variables for your needs.

Operation files

The release provides a set of operation files to enable or disable features. Operation files are located in the operationsdirectory. This folder contains commons opsfiles and two subdirectories for sharding and replicaset

Ops file	feature	needed variable	dependecies
rename-azs.yml	use specific azs	azs-list
use-specific-mongodb-release.yml	use a named uploaded mongodb release version instead of the latest one	mongodb-release-version
use_mmapv1.yml	use mmapv1 engine instead of wiredtiger default
use-trusty.yml	use an ubuntu trusty stemcell instead of the xenial default
use-specific-stemcell.yml	Use a specifically named stemcell version instead of the latest one	stemcell-version
enable-mongodb-acceptance-test.yml	Deploy the acceptance tests errand	accept_vm_type
enable-mongodb-broker.yml		broker_vm_type broker_persistent_disk_type broker_catalog_yml
enable-mongodb-broker-route-registrar.yml		cf.nats_host cf.nats_password cf.system_domain	enable-mongodb-broker.yml
enable-mongodb-broker-smoke-tests.yml			enable-mongodb-broker.yml enable-mongodb-broker-route-registrar.yml
rename-broker-network.yml	use a specific network for the broker instead of the default one		enable-mongodb-broker.yml
enable-prometheus-exporter.yml	deploy the prometheus mongodb exporter from prometheus-addons bosh release	clustermonitor_username

Note that operations directory include some others opsfiles like ssl ones that are not fully tested yet and should not be use

Deployment

bosh create-release
bosh upload-release
bosh -d [deployment name] -n deploy manifests/manifest[rs|shard].yml <-o operations/[operation file name] -o ...> -l <deployment-vars-file> <--vars-store=credentials.yml >

--vars-store=credentials.yml is uneeded if you are using credhub

Broker

Mongodb Broker (broker job)

The mongodb broker implements the 5 REST endpoints required by Cloud Foundry to write V2 services :

• Catalog management in order to register the broker to the platform
• Provisioning in order to create resource in the mongodb server
• Deprovisioning in order to release resource previously allocated
• Binding (credentials type) in order to provide application with a set of information required to use the allocated service
• Unbinding in order to delete credentials resources previously allocated

Mongodb Broker Smoke Tests (broker-smoke-tests job)

The mongodb broker smoke test acts as an end user developper who wants to host its application in a cloud foundry.

For that, it relies on a sample mongodb application : https://github.com/JCL38-ORANGE/cf-mongodb-example-app

The following steps are performed by the smoke tests job :

• Authentication on Cloud Foundry by targeting org and space (cf auth and cf target)
• Deployment of the sample mongodb application (cf push)
• Provisioning of the service (cf create-service)
• Binding of the service (cf bind-service)
• Restaging of the sample mongodb application (cf restage)
• Table creation in the mongodb cluster (HTTP POST command to the sample mongodb application)
• Table deletion in the mongodb cluster (HTTP DELETE command to the sample mongodb application)

Configuring CF to use Mongodb service

Available Plans

For the moment, only 1 default plan available for shared Mongodb.

Broker registration

The broker uses HTTP basic authentication to authenticate clients. The cf create-service-broker command expects the credentials for the cloud controller to authenticate itself to the broker.

cf create-service-broker p-mongodb-broker <user> <password> <url> 
cf enable-service-access mongodb

Service provisioning

cf create-service mongodb default mongodb-instance

Service binding

cf bind-service mongodb-example-app mongodb-instance

Service unbinding

cf unbind-service mongodb-example-app mongodb-instance

Service deprovisioning

cf delete-service mongodb-instance

Contributing

Ruby Env Setup

This my setup:

brew install ruby-build chruby
ruby-build 2.4.2 --install-dir ~/.rubies/ruby-2.4.2
gem update --system
gem install bundler
bundle install

mongodb-boshrelease's Issues

quota enforcer uses only one node to connect to cluster

Expected behavior

• As a paas-templates operator
• In order to limit space usage
• I need to have quota-enforcer process working

Observed behavior

When the master node, is not on the first mondb (index 0), the update user process is not working.
We need to have all the mongodb topology in the context, so the program can use the real master node

Add a dedicated repository for manifests (collection of BOSH manifests)

Unable to git clone

Hi Team,

I'm unable to clone the repo. it is failing at the submodule mongodb-quota-enforcer

Please look into it.

ubuntu-bionic compatibility

trying bosh release 12 with bionic stemcell 19.
mongod prestart fails with;

mongod/53c60889-23e9-4966-b087-7d2dc1cf6687:/var/vcap/sys/log/mongod# vi pre-start.stderr.log 
mongod/53c60889-23e9-4966-b087-7d2dc1cf6687:/var/vcap/sys/log/mongod# uname -a
Linux 502ffd6b-306b-4ddc-aa90-77f59313fcf7 5.4.0-70-generic #78~18.04.1-Ubuntu SMP Sat Mar 20 14:10:07 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
mongod/53c60889-23e9-4966-b087-7d2dc1cf6687:/var/vcap/sys/log/mongod# cat pre-start.stderr.log 
+ export JOBS_COMMON_DIR=/var/vcap/packages/mongodb-jobs-common
+ JOBS_COMMON_DIR=/var/vcap/packages/mongodb-jobs-common
+ source /var/vcap/packages/mongodb-jobs-common/bin/setenv mongod
++ ulimit -n 64000
++ ulimit -u 64000
++ ulimit -f unlimited
++ ulimit -t unlimited
++ ulimit -v unlimited
++ ulimit -m unlimited
++ export LANG=en_US.UTF-8
++ LANG=en_US.UTF-8
++ export JOB_NAME=mongod
++ JOB_NAME=mongod
++ output_label=mongod
++ export JOB_DIR=/var/vcap/jobs/mongod
++ JOB_DIR=/var/vcap/jobs/mongod
++ export JOBS_COMMON_DIR=/var/vcap/packages/mongodb-jobs-common
++ JOBS_COMMON_DIR=/var/vcap/packages/mongodb-jobs-common
++ chmod 755 /var/vcap/jobs/mongod
++ source /var/vcap/packages/mongodb-jobs-common/helpers/ctl_utils.sh
+++ find /var/vcap/packages -follow -mindepth 2 -path '*bin' -type d
++ for package_bin_dir in $(find  /var/vcap/packages -follow -mindepth 2 -path '*bin' -type d)
++ export PATH=/var/vcap/packages/node_exporter/bin:/usr/sbin:/usr/bin:/sbin:/bin
++ PATH=/var/vcap/packages/node_exporter/bin:/usr/sbin:/usr/bin:/sbin:/bin
++ for package_bin_dir in $(find  /var/vcap/packages -follow -mindepth 2 -path '*bin' -type d)
++ export PATH=/var/vcap/packages/bosh-dns/bin:/var/vcap/packages/node_exporter/bin:/usr/sbin:/usr/bin:/sbin:/bin
++ PATH=/var/vcap/packages/bosh-dns/bin:/var/vcap/packages/node_exporter/bin:/usr/sbin:/usr/bin:/sbin:/bin
++ for package_bin_dir in $(find  /var/vcap/packages -follow -mindepth 2 -path '*bin' -type d)
++ export PATH=/var/vcap/packages/bpm/lib/libseccomp/bin:/var/vcap/packages/bosh-dns/bin:/var/vcap/packages/node_exporter/bin:/usr/sbin:/usr/bin:/sbin:/bin
++ PATH=/var/vcap/packages/bpm/lib/libseccomp/bin:/var/vcap/packages/bosh-dns/bin:/var/vcap/packages/node_exporter/bin:/usr/sbin:/usr/bin:/sbin:/bin
++ for package_bin_dir in $(find  /var/vcap/packages -follow -mindepth 2 -path '*bin' -type d)
++ export PATH=/var/vcap/packages/bpm/bin:/var/vcap/packages/bpm/lib/libseccomp/bin:/var/vcap/packages/bosh-dns/bin:/var/vcap/packages/node_exporter/bin:/usr/sbin:/usr/bin:/sbin:/bin
++ PATH=/var/vcap/packages/bpm/bin:/var/vcap/packages/bpm/lib/libseccomp/bin:/var/vcap/packages/bosh-dns/bin:/var/vcap/packages/node_exporter/bin:/usr/sbin:/usr/bin:/sbin:/bin
++ for package_bin_dir in $(find  /var/vcap/packages -follow -mindepth 2 -path '*bin' -type d)
++ export PATH=/var/vcap/packages/mongodb_exporter/bin:/var/vcap/packages/bpm/bin:/var/vcap/packages/bpm/lib/libseccomp/bin:/var/vcap/packages/bosh-dns/bin:/var/vcap/packages/node_exporter/bin:/usr/sbin:/usr/bin:/sbin:/bin
++ PATH=/var/vcap/packages/mongodb_exporter/bin:/var/vcap/packages/bpm/bin:/var/vcap/packages/bpm/lib/libseccomp/bin:/var/vcap/packages/bosh-dns/bin:/var/vcap/packages/node_exporter/bin:/usr/sbin:/usr/bin:/sbin:/bin
++ for package_bin_dir in $(find  /var/vcap/packages -follow -mindepth 2 -path '*bin' -type d)
++ export PATH=/var/vcap/packages/shield/bin:/var/vcap/packages/mongodb_exporter/bin:/var/vcap/packages/bpm/bin:/var/vcap/packages/bpm/lib/libseccomp/bin:/var/vcap/packages/bosh-dns/bin:/var/vcap/packages/node_exporter/bin:/usr/sbin:/usr/bin:/sbin:/bin
++ PATH=/var/vcap/packages/shield/bin:/var/vcap/packages/mongodb_exporter/bin:/var/vcap/packages/bpm/bin:/var/vcap/packages/bpm/lib/libseccomp/bin:/var/vcap/packages/bosh-dns/bin:/var/vcap/packages/node_exporter/bin:/usr/sbin:/usr/bin:/sbin:/bin
++ for package_bin_dir in $(find  /var/vcap/packages -follow -mindepth 2 -path '*bin' -type d)
++ export PATH=/var/vcap/packages/blackbox/bin:/var/vcap/packages/shield/bin:/var/vcap/packages/mongodb_exporter/bin:/var/vcap/packages/bpm/bin:/var/vcap/packages/bpm/lib/libseccomp/bin:/var/vcap/packages/bosh-dns/bin:/var/vcap/packages/node_exporter/bin:/usr/sbin:/usr/bin:/sbin:/bin
++ PATH=/var/vcap/packages/blackbox/bin:/var/vcap/packages/shield/bin:/var/vcap/packages/mongodb_exporter/bin:/var/vcap/packages/bpm/bin:/var/vcap/packages/bpm/lib/libseccomp/bin:/var/vcap/packages/bosh-dns/bin:/var/vcap/packages/node_exporter/bin:/usr/sbin:/usr/bin:/sbin:/bin
++ for package_bin_dir in $(find  /var/vcap/packages -follow -mindepth 2 -path '*bin' -type d)
++ export PATH=/var/vcap/packages/mongodb/bin:/var/vcap/packages/blackbox/bin:/var/vcap/packages/shield/bin:/var/vcap/packages/mongodb_exporter/bin:/var/vcap/packages/bpm/bin:/var/vcap/packages/bpm/lib/libseccomp/bin:/var/vcap/packages/bosh-dns/bin:/var/vcap/packages/node_exporter/bin:/usr/sbin:/usr/bin:/sbin:/bin
++ PATH=/var/vcap/packages/mongodb/bin:/var/vcap/packages/blackbox/bin:/var/vcap/packages/shield/bin:/var/vcap/packages/mongodb_exporter/bin:/var/vcap/packages/bpm/bin:/var/vcap/packages/bpm/lib/libseccomp/bin:/var/vcap/packages/bosh-dns/bin:/var/vcap/packages/node_exporter/bin:/usr/sbin:/usr/bin:/sbin:/bin
++ for package_bin_dir in $(find  /var/vcap/packages -follow -mindepth 2 -path '*bin' -type d)
++ export PATH=/var/vcap/packages/mongodb-jobs-common/bin:/var/vcap/packages/mongodb/bin:/var/vcap/packages/blackbox/bin:/var/vcap/packages/shield/bin:/var/vcap/packages/mongodb_exporter/bin:/var/vcap/packages/bpm/bin:/var/vcap/packages/bpm/lib/libseccomp/bin:/var/vcap/packages/bosh-dns/bin:/var/vcap/packages/node_exporter/bin:/usr/sbin:/usr/bin:/sbin:/bin
++ PATH=/var/vcap/packages/mongodb-jobs-common/bin:/var/vcap/packages/mongodb/bin:/var/vcap/packages/blackbox/bin:/var/vcap/packages/shield/bin:/var/vcap/packages/mongodb_exporter/bin:/var/vcap/packages/bpm/bin:/var/vcap/packages/bpm/lib/libseccomp/bin:/var/vcap/packages/bosh-dns/bin:/var/vcap/packages/node_exporter/bin:/usr/sbin:/usr/bin:/sbin:/bin
++ LD_LIBRARY_PATH=
+++ find /var/vcap/packages -follow -maxdepth 2 -mindepth 2 -type d -name lib -o -name lib64
++ for package_lib_dir in $(find  /var/vcap/packages -follow -maxdepth 2 -mindepth 2 -type d -name 'lib' -o -name 'lib64')
++ LD_LIBRARY_PATH=/var/vcap/packages/bpm/lib:
++ export LD_LIBRARY_PATH
++ export RUN_DIR=/var/vcap/sys/run/mongod
++ RUN_DIR=/var/vcap/sys/run/mongod
++ export LOG_DIR=/var/vcap/sys/log/mongod
++ LOG_DIR=/var/vcap/sys/log/mongod
++ export TMP_DIR=/var/vcap/sys/tmp/mongod
++ TMP_DIR=/var/vcap/sys/tmp/mongod
++ export STORE_DIR=/var/vcap/store/mongod
++ STORE_DIR=/var/vcap/store/mongod
++ export DATA_DIR=/var/vcap/store/mongod/data
++ DATA_DIR=/var/vcap/store/mongod/data
++ export MONGODB_BIN=/var/vcap/packages/mongodb/bin
++ MONGODB_BIN=/var/vcap/packages/mongodb/bin
++ export MONGODB_CONF=/var/vcap/jobs/mongod/config
++ MONGODB_CONF=/var/vcap/jobs/mongod/config
++ export MONGODB_SSL=/var/vcap/jobs/mongod/ssl
++ MONGODB_SSL=/var/vcap/jobs/mongod/ssl
++ PIDFILE=/var/vcap/sys/run/mongod/mongod.pid
+ source /var/vcap/packages/mongodb-jobs-common/bin/mdb-functions
++ source /var/vcap/jobs/mongod/bin/mdb-variables.sh
+++ property_require_ssl=0
+++ property_mongod_listen_port=27017
+++ property_root_username=MongoRoot
+++ property_root_password=HyjlgPWHWPDk5kYY3ayShyhNMt0t4x
+++ property_admin_username=admin
+++ property_admin_password=2aitX15MrCEFFKSiaFiGdlISf3sDEY
+++ property_replication_enable=1
+++ property_replica_set_name=rs0
+++ property_use_arbiter=0
+++ property_engine=wiredTiger
+++ deployment_current_ip=192.168.31.163
+++ deployment_rs_config=192.168.31.163:27017,192.168.31.170:27017,192.168.31.169:27017
+ source /var/vcap/jobs/mongod/bin/mdb-variables.sh
++ property_require_ssl=0
++ property_mongod_listen_port=27017
++ property_root_username=MongoRoot
++ property_root_password=HyjlgPWHWPDk5kYY3ayShyhNMt0t4x
++ property_admin_username=admin
++ property_admin_password=2aitX15MrCEFFKSiaFiGdlISf3sDEY
++ property_replication_enable=1
++ property_replica_set_name=rs0
++ property_use_arbiter=0
++ property_engine=wiredTiger
++ deployment_current_ip=192.168.31.163
++ deployment_rs_config=192.168.31.163:27017,192.168.31.170:27017,192.168.31.169:27017
++ check_ssl
++ echo 0
+ '[' 0 -eq 0 ']'
+ export MONGO_CMD=/var/vcap/packages/mongodb-jobs-common/bin/mongo.sh
+ MONGO_CMD=/var/vcap/packages/mongodb-jobs-common/bin/mongo.sh
+ system_update
+ '[' -d /sys/kernel/mm/transparent_hugepage ']'
+ thp_path=/sys/kernel/mm/transparent_hugepage
+ echo never
+ echo never
+ unset thp_path
+ rights_and_directories
+ for dir in $RUN_DIR $LOG_DIR $TMP_DIR $STORE_DIR $DATA_DIR
+ mkdir -p /var/vcap/sys/run/mongod
+ chown vcap:vcap /var/vcap/sys/run/mongod
+ chmod 775 /var/vcap/sys/run/mongod
+ for dir in $RUN_DIR $LOG_DIR $TMP_DIR $STORE_DIR $DATA_DIR
+ mkdir -p /var/vcap/sys/log/mongod
+ chown vcap:vcap /var/vcap/sys/log/mongod
+ chmod 775 /var/vcap/sys/log/mongod
+ for dir in $RUN_DIR $LOG_DIR $TMP_DIR $STORE_DIR $DATA_DIR
+ mkdir -p /var/vcap/sys/tmp/mongod
+ chown vcap:vcap /var/vcap/sys/tmp/mongod
+ chmod 775 /var/vcap/sys/tmp/mongod
+ for dir in $RUN_DIR $LOG_DIR $TMP_DIR $STORE_DIR $DATA_DIR
+ mkdir -p /var/vcap/store/mongod
+ chown vcap:vcap /var/vcap/store/mongod
+ chmod 775 /var/vcap/store/mongod
+ for dir in $RUN_DIR $LOG_DIR $TMP_DIR $STORE_DIR $DATA_DIR
+ mkdir -p /var/vcap/store/mongod/data
+ chown vcap:vcap /var/vcap/store/mongod/data
+ chmod 775 /var/vcap/store/mongod/data
+ keyfile=/var/vcap/jobs/mongod/config/keyfile
+ chmod 600 /var/vcap/jobs/mongod/config/keyfile
+ chown vcap:vcap /var/vcap/jobs/mongod/config/keyfile
+ export MONGO_CMD_PARAM=--quiet
+ MONGO_CMD_PARAM=--quiet
++ check_master
++ source /var/vcap/jobs/mongod/bin/mdb-variables.sh
+++ property_require_ssl=0
+++ property_mongod_listen_port=27017
+++ property_root_username=MongoRoot
+++ property_root_password=HyjlgPWHWPDk5kYY3ayShyhNMt0t4x
+++ property_admin_username=admin
+++ property_admin_password=2aitX15MrCEFFKSiaFiGdlISf3sDEY
+++ property_replication_enable=1
+++ property_replica_set_name=rs0
+++ property_use_arbiter=0
+++ property_engine=wiredTiger
+++ deployment_current_ip=192.168.31.163
+++ deployment_rs_config=192.168.31.163:27017,192.168.31.170:27017,192.168.31.169:27017
++ Master=
+++ tr -s , ' '
+++ echo 192.168.31.163:27017,192.168.31.170:27017,192.168.31.169:27017
++ for Server in $(echo ${deployment_rs_config}|tr -s "," " ")
++ '[' '' == '' ']'
+++ /var/vcap/packages/mongodb-jobs-common/bin/mongo.sh --quiet mongodb://192.168.31.163:27017 --eval 'db.isMaster().ismaster'
++ '[' '' == true ']'
+++ /var/vcap/packages/mongodb-jobs-common/bin/mongo.sh --quiet mongodb://192.168.31.163:27017 -u MongoRoot -p HyjlgPWHWPDk5kYY3ayShyhNMt0t4x --authenticationDatabase admin --eval 'db.isMaster().ismaster'
++ '[' '' == true ']'
++ for Server in $(echo ${deployment_rs_config}|tr -s "," " ")
++ '[' '' == '' ']'
+++ /var/vcap/packages/mongodb-jobs-common/bin/mongo.sh --quiet mongodb://192.168.31.170:27017 --eval 'db.isMaster().ismaster'
++ '[' '' == true ']'
+++ /var/vcap/packages/mongodb-jobs-common/bin/mongo.sh --quiet mongodb://192.168.31.170:27017 -u MongoRoot -p HyjlgPWHWPDk5kYY3ayShyhNMt0t4x --authenticationDatabase admin --eval 'db.isMaster().ismaster'
++ '[' '' == true ']'
++ for Server in $(echo ${deployment_rs_config}|tr -s "," " ")
++ '[' '' == '' ']'
+++ /var/vcap/packages/mongodb-jobs-common/bin/mongo.sh --quiet mongodb://192.168.31.169:27017 --eval 'db.isMaster().ismaster'
++ '[' '' == true ']'
+++ /var/vcap/packages/mongodb-jobs-common/bin/mongo.sh --quiet mongodb://192.168.31.169:27017 -u MongoRoot -p HyjlgPWHWPDk5kYY3ayShyhNMt0t4x --authenticationDatabase admin --eval 'db.isMaster().ismaster'
++ '[' '' == true ']'
++ echo
+ '[' '' '!=' '' ']'
++ check_password
++ return=0
++ '[' -f /var/vcap/store/mongod/data/mongo_ident.conf ']'
++ '[' -f /var/vcap/store/mongod/data/mongo_private.pem ']'
++ '[' -f /var/vcap/store/mongod/data/mongo_public.pem ']'
+++ openssl rsautl -decrypt -inkey /var/vcap/store/mongod/data/mongo_private.pem -in /var/vcap/store/mongod/data/mongo_ident.conf
++ line='MongoRoot HyjlgPWHWPDk5kYY3ayShyhNMt0t4x'
+++ echo MongoRoot HyjlgPWHWPDk5kYY3ayShyhNMt0t4x
+++ cut '-d ' -f1
++ _root_username=MongoRoot
+++ cut '-d ' -f2
+++ echo MongoRoot HyjlgPWHWPDk5kYY3ayShyhNMt0t4x
++ _root_password=HyjlgPWHWPDk5kYY3ayShyhNMt0t4x
++ '[' MongoRoot '!=' MongoRoot -o HyjlgPWHWPDk5kYY3ayShyhNMt0t4x '!=' HyjlgPWHWPDk5kYY3ayShyhNMt0t4x ']'
++ echo 0
+ '[' '!' -f /var/vcap/sys/run/mongod/mongodb_initialized -o 0 -eq 1 ']'
+ start_mongod_noauth
+ '[' -f /var/vcap/sys/run/mongod/mongod.pid ']'
+ pid_guard /var/vcap/sys/run/mongod/mongod.pid mongod
+ pidfile=/var/vcap/sys/run/mongod/mongod.pid
+ name=mongod
+ '[' -f /var/vcap/sys/run/mongod/mongod.pid ']'
+ '[' -f /tmp/mongod_bs.conf ']'
+ sed -e '/^[[:space:]]*keyFile:/d' -e '/^[[:space:]]*clusterAuthMode:[[:space:]]*x509.*/d' /var/vcap/jobs/mongod/config/mongod.conf
+ chmod a+r /tmp/mongod_bs.conf
+ chpst -u vcap:vcap /var/vcap/packages/mongodb/bin/mongod --config /tmp/mongod_bs.conf --noauth --pidfilepath /var/vcap/sys/run/mongod/mongod.pid
/var/vcap/packages/mongodb/bin/mongod: /usr/lib/x86_64-linux-gnu/libcurl.so.4: version `CURL_OPENSSL_3' not found (required by /var/vcap/packages/mongodb/bin/mongod)

Support airgapped CF deployments

As a mongodb service operator, in order to run smoke tests in a CF deployment which does not have internet access to pull ruby gems dependencies, I need the app dependencies to be vendored bundle package --all before the cf push is launched. This would require the job to accepts new http_proxy/https_proxy/no_proxy job properties

https://github.com/orange-cloudfoundry/mongodb-boshrelease/blob/master/jobs/broker-smoke-tests/spec#L12-L13

mongodb-boshrelease/jobs/broker-smoke-tests/templates/bin/run

Lines 51 to 53 in e14c7f9

	# Push mongodb-example-app
	cf push
	echo 'End push'

See more details about Vendor App Dependencies in https://docs.cloudfoundry.org/buildpacks/ruby/index.html#vendoring

This may remove need for orange-cloudfoundry/cf-mongodb-example-app#2 and enable ruby security dependencies to be fixed

Package sample application as a submodule instead of a blob

Today, the sample application is used to achieve the broker smoke tests and is packaged as a blob.
An enhancement is to package the sample application as a submodule (https://github.com/JCL38-ORANGE/cf-mongodb-example-app)

missing releases/mongodb-services/mongodb-services-7.yml V7

Hi all,

i try running v7 from source, but in releases/mongodb-services missing yml file mongodb-services-7.yml

support arbitrator job

As a platform runner, i want to use the mongodb bosh release on 3 azs, and be able to only put az on one az

cc @regismarg @ccambien

Update example-manifest to include broker and broker-smoke-tests errand

This would be useful to reduce configuration for people wanting to use this bosh release

Missing LICENSE file

Please add a license file, likely the apache V2 license.

remove internal database in broker which holds binding credentials

Improve broker smoke test

add an hamode parameter on broker in order to return direct connection to replica set (primary and seconday uris)

Expected behavior

• As a paas-templates operator
• In order to use both normal and ha mode during binding
• I need to have a parameter in broker job to choose the appropriate mode

Observed behavior

Today, the broker connects to the mongodb server thanks to IPs given in the application.yml file. Once connected, the topology is discovered dynamically and sent to the client (outside) at binding time.
This implementation works fine in the normal mode (mongod nodes communication based on publics IPs reachable by a client).
This implementation doesn't work when mongod nodes have private IPs with haproxy in front of mongo nodes. In this case, the topology is based on private IPs (discovered dynamically by the broker) and they are not reachable by a client outside the platform.

Proposed solution

Add an hamode parameter in the broker job.
When hamode is set to false (default), the IPs returned at binding time are those discovered dynamically by the broker with a replica set uri format.
When hamode is set to true :

• the IPs returned at binding time are those given in the application.yml file
• the uri connection string is set to direct connection (target the primary node of the replica set)
• a second uri is supplied in case of failover.

Usage of BOSH Vendoring Packages (Java)

https://bosh.io/docs/package-vendoring.html
blocked
cloudfoundry/bosh-package-java-release#1

Replace groupid in the broker

Replace groupid in the java broker from org.springframework.cloud to com.orange.oss.osb.mongodb

Release not compatible with CentOS stemcell

Compiled mongod binary (/var/vcap/packages/mongodb/bin/mongod) not linked to required library/

On CentOS 7.4 stemcell:

# for package_lib_dir in $(ls -d /var/vcap/packages/*/lib /var/vcap/packages/*/lib64)
> do
> export LD_LIBRARY_PATH=${package_lib_dir}:$LD_LIBRARY_PATH
> done

# ldd /var/vcap/packages/mongodb/bin/mongod
/var/vcap/packages/mongodb/bin/mongod: /lib64/libc.so.6: version `GLIBC_2.18' not found (required by /var/vcap/packages/mongodb/lib64/libstdc++.so.6)
        linux-vdso.so.1 =>  (0x00007ffcc3b6e000)
        libsnappy.so.1.1.5 => /var/vcap/packages/mongodb/lib/libsnappy.so.1.1.5 (0x00007f5246fa1000)
        liblz4.so.1 => /var/vcap/packages/mongodb/lib/liblz4.so.1 (0x00007f5246d86000)
        libzstd.so.1 => /var/vcap/packages/mongodb/lib/libzstd.so.1 (0x00007f5246b04000)
        libz.so.1 => /var/vcap/packages/mongodb/lib/libz.so.1 (0x00007f52468e9000)
        libssl.so.1.0.0 => not found
        libcrypto.so.1.0.0 => not found
        librt.so.1 => /lib64/librt.so.1 (0x00007f52466d7000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f52464d2000)
        libstdc++.so.6 => /var/vcap/packages/mongodb/lib64/libstdc++.so.6 (0x00007f5246158000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f5245e56000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f5245c3f000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f5245a23000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f5245660000)
        /lib64/ld-linux-x86-64.so.2 (0x000055c1f6aa6000)

libssl.so.1.0.0 and libcrypto.so.1.0.0 are missing.
On CentOS 7.4, libssl and libcrypto are located in /usr/lib64 and version 1.0.2k

Example on a working deployment (Ubuntu trusty stemcell):

# for package_lib_dir in $(ls -d /var/vcap/packages/*/lib /var/vcap/packages/*/lib64)
> do
> export LD_LIBRARY_PATH=${package_lib_dir}:$LD_LIBRARY_PATH
> done

# ldd /var/vcap/packages/mongodb/bin/mongod
        linux-vdso.so.1 =>  (0x00007ffc8cd0e000)
        libsnappy.so.1.1.5 => /var/vcap/packages/mongodb/lib/libsnappy.so.1.1.5 (0x00007f7e80c5a000)
        liblz4.so.1 => /var/vcap/packages/mongodb/lib/liblz4.so.1 (0x00007f7e80a3f000)
        libzstd.so.1 => /var/vcap/packages/mongodb/lib/libzstd.so.1 (0x00007f7e807bd000)
        libz.so.1 => /var/vcap/packages/mongodb/lib/libz.so.1 (0x00007f7e805a2000)
        libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007f7e8033d000)
        libcrypto.so.1.0.0 => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 (0x00007f7e7ff60000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f7e7fd58000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f7e7fb54000)
        libstdc++.so.6 => /var/vcap/packages/mongodb/lib64/libstdc++.so.6 (0x00007f7e7f7d9000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f7e7f4d3000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f7e7f2bd000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f7e7f09e000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f7e7ecd5000)
        /lib64/ld-linux-x86-64.so.2 (0x000055d520704000)

libssl.so.1.0.0 and libcrypto.so.1.0.0 are located in /lib/x86_64-linux-gnu on Ubuntu Trustry.

Add the possibility to change externally the catalog service broker labels (plans and description)

enhance broker binding credentials

adding hostname, username, password to returned binding credentials

The dashboard_url field must return a string value instead of a field

With recent versions of cloudfoundry the null value is not more allowed for the dashboard_url field.
This problem appears with the bump of capi-version 1.61 (https://bosh.io/releases/github.com/cloudfoundry/capi-release?version=1.61.0)
The problem is referenced there :

• https://www.pivotaltracker.com/n/projects/2105761/stories/157601813
• https://github.com/openservicebrokerapi/servicebroker/pull/437/files
  The broker code must be updated to return an empty string instead of a null value for dashboard_url field.

Support root password rotation

Expected behavior

As a mongodb service operator, in order to protect my platform, I need sometimes to perform password regeneration.

Observed behavior

credhub regenerate -n "path to mongodb root password"
After regeneration, neither deploy, neither recreate can be done on deployments.
The following error is raised :
Task 2359451 | 12:29:06 | Updating instance mongod: mongod/48713cb5-ac31-41ea-b451-ed087a63a432 (0) (canary) (00:00:41)
14:29:47

                 L Error: Action Failed get_task: Task 63022b42-c6c7-4b1c-4fe4-0f2146d83e89 result: 1 of 6 pre-start scripts failed. Failed Jobs: mongod. Successful Jobs: login_banner, bpm, bosh-dns, syslog_forwarder, ca_certs.

14:29:47

Task 2359451 | 12:29:47 | Error: Action Failed get_task: Task 63022b42-c6c7-4b1c-4fe4-0f2146d83e89 result: 1 of 6 pre-start scripts failed. Failed Jobs: mongod. Successful Jobs: login_banner, bpm, bosh-dns, syslog_forwarder, ca_certs.
14:29:47

Affected release

Reproduced on version 6 and 7

Stand-alone mode only - The v1 bosh-release doesn't not support bosh deploy --recreate

The BOSH message is :
...
Using deployment 'mongodb'

Task 1020779

16:10:35 | Preparing deployment: Preparing deployment (00:00:01)
16:10:38 | Preparing package compilation: Finding packages to compile (00:00:00)
16:10:38 | Updating instance mongodb-broker: mongodb-broker/76e11b9d-9e14-4788-88fa-8ead0f28267f (0) (canary)
16:10:38 | Updating instance mongod: mongod/201cd9f6-84e7-483d-b978-136afad86d17 (0) (canary)
16:13:16 | Updating instance mongodb-broker: mongodb-broker/76e11b9d-9e14-4788-88fa-8ead0f28267f (0) (canary) (00:02:38)
17:13:48 | Updating instance mongod: mongod/201cd9f6-84e7-483d-b978-136afad86d17 (0) (canary) (01:03:10)
L Error: Action Failed get_task: Task 13482112-ace9-4174-5373-7bab3952615b result: 1 of 4 pre-start scripts failed. Failed Jobs: mongod. Successful Jobs: login_banner, syslog_forwarder, ca_certs.

17:13:48 | Error: Action Failed get_task: Task 13482112-ace9-4174-5373-7bab3952615b result: 1 of 4 pre-start scripts failed. Failed Jobs: mongod. Successful Jobs: login_banner, syslog_forwarder, ca_certs.

Started Fri Dec 1 16:10:35 UTC 2017
Finished Fri Dec 1 17:13:48 UTC 2017
Duration 01:03:13

Task 1020779 error

Changing state:
Expected task '1020779' to succeed but state is 'error'

Exit code 1
...

The deployment is configured as standalone.
The problem seems to be located in the pre-start script of the mongod job.
There is an infinite loop in the function "create_admin_users".

Add the possibility to configure the listen port used by the broker

Update MongoDB & Tools from 3.6.12 to 3.6.18

Due to a security issue in 3.6.12 () the package should be updated to 3.6.18 (http://downloads.mongodb.org/linux/mongodb-linux-x86_64-3.6.18.tgz)

https://jira.mongodb.org/browse/SERVER-45472
CVE-2020-7921

We have already updated the release internally but since we don't have access to the underlying blobstore I don't think it would be possible to create a pull request for this:

diff --git a/README.md b/README.md
index 4fb0844..ed43d0c 100644
--- a/README.md
+++ b/README.md
@@ -47,9 +47,9 @@ This version exclude the rocksdb engine, which is not supported anymore.

 | Package         | Version     | Note                  |
 | --------------- | ----------- | --------------------- |
-| mongodb         | `3.6.12`    |                       |
+| mongodb         | `3.6.18`    |                       |
 | ~~mongo-rocks~~ | ~~`3.4.7`~~ | Not supported anymore |
-| mongo-tools     | `3.6.12`    |                       |
+| mongo-tools     | `3.6.18`    |                       |
 | ~~rocksdb~~     | ~~`3.4.7`~~ | Not supported anymore |


diff --git a/config/blobs.yml b/config/blobs.yml
index 0e81202..9518f8f 100644
--- a/config/blobs.yml
+++ b/config/blobs.yml
@@ -2,10 +2,9 @@ cf-cli/cf-cli_6.30.0_linux_x86-64.tgz:
   size: 5909858
   object_id: e57130b7-9c58-447b-43b2-bd83770093cf
   sha: 10a795927fd6f03f1b7212f21ac508e6278270fd
-mongodb/mongodb-linux-x86_64-3.6.12.tar.gz:
-  size: 100307521
-  object_id: 4f4284a7-9755-4340-714a-70291712c2b9
-  sha: 5a145cccd202aab0764b326f1e7503f246b8d506
+mongodb/mongodb-linux-x86_64-3.6.18.tar.gz:
+  size: 100717642
+  sha: sha256:aedd36ba22a81e2fed6e2b3b38ee38976bde027d9e323fef82127424fc1ebb7e
 openjdk/openjdk.tar.gz:
   size: 126476497
   object_id: eb5177af-ce3b-4dfd-657d-ae9b84a255a7
diff --git a/mongodb_version b/mongodb_version
index 81c117c..c241e96 100644
--- a/mongodb_version
+++ b/mongodb_version
@@ -1 +1 @@
-3.6.12
\ No newline at end of file
+3.6.18

Thanks!

Florian.

quota enforcement

As a platform operator, using the bosh release to offer a shared servic, i need to have an optional quota enforcement component

cc @JCL38-ORANGE