shieldproject / shield-boshrelease Goto Github PK
View Code? Open in Web Editor NEWBOSH Release for shield
License: MIT License
BOSH Release for shield
License: MIT License
- Unable to render jobs for instance group 'shield'. Errors are:
- Unable to render templates for job 'shield-agent'. Errors are:
- Error filling in template 'agent.conf' (line 4: no implicit conversion of nil into String)```
https://github.com/starkandwayne/shield-boshrelease/blob/master/jobs/shield-agent/templates/config/agent.conf#L4
Do we really need az? Can we leave it out and hope name and index is ok?
Rentenions, schedules and stores can be specified in the manifest. Can we also add support for targets and jobs?
On the shield/0
VM, the /var/vcap/jobs/shield-agent/bin/ctl
startup script has an issue producing this error:
# tail /var/vcap/data/sys/log/monit/shield-agent.log
/var/vcap/jobs/shield-agent/bin/ctl: line 22: syntax error near unexpected token `}'
This is due to both if_p("shield.agent.autoprovision")
and if_p("shield.agent.daemon_public_key")
evaluating to false
and not rendering their respective blocks. Even when these are set in the deployment properties, which is strange. Anyway, this issue could easily be circumvented with a mere call to /bin/true
within the curly braces so that the block is never empty from a Bash syntax point of view.
(This issue was originally part of issue #48.)
In the post-start script the uuid isn't being passed in to the shield edit-xxx
command.
If the Agent and the Daemon are co-located on the same host, the agent will create a blank "authorized_keys" file if it starts before the daemon. The agent should not be allowed to create a blank file and should fail, so that it can try to start again once Monit successfully starts the daemon.
(migrated here from shieldproject/shield#244)
Out-of-the-box, SHIELD should just use a local SQLite3 database - no need for fancy PostgreSQL or MySQL configuration.
Need postgres server running to do migration but job doesn't exist in release in v8. Maybe use https://github.com/cloudfoundry/postgres-release as stop gap for upgrade to v8?
Hey
It would be great if the agent-pgtools
would come with a newer version of postgres
. Something like 9.6.x
would be fine, since this is latest stable.
An other option would be to split the package into multiple versions like it’s done for mongo
.
I’m happy to do so (and also kinda working on it), but it would be cool to agree on how to do it and then do it :)
cheers
The property descriptions are a little vague in some areas.
Also, the README should be updated to include some common configurations, and perhaps more in-depth explanation of the properties, jobs, and how they interrelate.
Looks like it fails to download a tarball:
Building release tarball from manifest '/mnt/tmp/downloader-GitDownloader598508896/releases/shield/shield-2.1.yml': Running bosh create release: Running command: 'bosh create release /mnt/tmp/downloader-GitDownloader598508896/releases/shield/shield-2.1.yml', stdout: 'Recreating release from the manifest
Copying packages
----------------
bsdtar (fe279cae5da0aa7b9cbf07238dad8afba6c8edc7)
common (e401816a4748292163679fafcbd8f818ed8154a5)
elasticsearch-cloud-aws-plugin (a8701cdd6895f248d4d51fd918e4d71daa678b4f)
generated_agent_key (d130726fd279b4cfb8ef211daad7b64e62bb888d)
generated_daemon_key (d130726fd279b4cfb8ef211daad7b64e62bb888d)
golang (201283451505af9fc84d23998b96584a84a45e57)
Downloading from blobstore (id=890933cd-bf31-4d77-a44c-54dcc0d8b030)...
', stderr: 'Blobstore error: sha1 mismatch expected=fd14409a79e30a47d04b4ab15bf96798fd7f3b37 actual=20d09c7867ecf27a8aefd614dc737266d405e45b': exit status 1
Hello, I'v been testing shiled on my local bosh-lite.
command : shield run [jobs name] -k
result :
shieldd log :
2017-01-23 04:07:35.545134238 +0000 UTC shieldd: INFO: schedule adhoc backup job
2017-01-23 04:07:35.545204147 +0000 UTC shieldd: INFO: scheduling immediate (ad hoc) execution of job shield [625738c4-db29-46fc-8fd7-af5ee6d11e15]
2017-01-23 04:07:35.547435938 +0000 UTC shieldd: INFO: schedule task dab38689-15f2-40aa-861a-9f1c256316db with deadline {{63620784455 547401390 0x1157da0}}
2017/01/23 04:07:35 http: multiple response.WriteHeader calls
2017-01-23 04:07:35.966399278 +0000 UTC shieldd: INFO: sent a task to a worker
2017-01-23 04:07:35.966703638 +0000 UTC shieldd: INFO: dab38689-15f2-40aa-861a-9f1c256316db> TASK FAILED!! shield worker 5 unable to connect to 10.244.204.2:443 (ssh: handshake failed: ssh: invalid packet length, packet too large)
2017-01-23 04:07:35.975731706 +0000 UTC shieldd: WARNING: dab38689-15f2-40aa-861a-9f1c256316db: task failed!
2017-01-23 05:30:40.733353999 +0000 UTC shieldd: INFO: schedule adhoc backup job
2017-01-23 05:30:40.73347915 +0000 UTC shieldd: INFO: scheduling immediate (ad hoc) execution of job shield [625738c4-db29-46fc-8fd7-af5ee6d11e15]
2017-01-23 05:30:40.735746302 +0000 UTC shieldd: INFO: schedule task 52110f7e-636f-43e8-9a97-4d876cab765a with deadline {{63620789440 735728122 0x1157da0}}
2017/01/23 05:30:40 http: multiple response.WriteHeader calls
2017-01-23 05:30:40.963795183 +0000 UTC shieldd: INFO: sent a task to a worker
my bosh vms, below
+-------------------------------------------------+---------+-----+----------+--------------+
| VM | State | AZ | VM Type | IPs |
+-------------------------------------------------+---------+-----+----------+--------------+
| shield/0 | running | n/a | small_z1 | 10.244.204.2 |
+-------------------------------------------------+---------+-----+----------+--------------+
+---------------------------------------------------------------+---------+-----+----------+-------------+
| VM | State | AZ | VM Type | IPs |
+---------------------------------------------------------------+---------+-----+----------+-------------+
| postgresql_docker_z1/0 | running | n/a | small_z1 | 10.244.30.6 |
+---------------------------------------------------------------+---------+-----+----------+-------------+
+---------------------------------------------------------------+---------+-----+----------+-------------+
| VM | State | AZ | VM Type | IPs |
+---------------------------------------------------------------+---------+-----+----------+-------------+
| postgresql_docker_z1/0 | running | n/a | small_z1 | 10.244.40.6 |
+---------------------------------------------------------------+---------+-----+----------+-------------+
my job
Name: shield
Paused: N
Retention Policy: default
Expires in: 100 days
Schedule Policy: default
Target: postgres
Target Endpoint: {"pg_user":"target", "pg_password":"target", "pg_host":"10.244.40.6", "pg_port":"5432"}
Remote IP: 10.244.204.2:5524
Store: postgres
Store Endpoint: {"pg_user": "store", "pg_password": "store", "pg_host": "10.244.30.6", "pg_port": "5432"}
Notes: shield
IP & Port of Target & Store is correct. I saw init.script of test-bed.
I've thought that i don't understand shield.
I wait your reply.
The xtrabackup
binary version 2.2.10
shipped by the cf-mysql release v26
doesn't support the --move-back
option on which the xtrabackup SHIELD plugin relies.
In such situation, the xtrabackup
job of the shield-boshrelease is supposed to help. Unfortunately, it brings an xtrabackup
package whose name conflicts with the one already installed by the mysql
job of the cf-mysql release.
A solution for the SHIELD release would be to rename the xtrabackup
package into xtrabackup-2.4.5
so that it doesn't conflict with the default xtrabackup
package shipped by the mysql
job of the cf-mysql release.
Hello,
I think that the log-level.yml log-level.yml
operator file contain a typo.
Line 3 : path: /instance_groups/name=shield/job/name=shield-daemon/properties/log_level?
Must be : path: /instance_groups/name=shield/jobs/name=shield-daemon/properties/log_level?
Line 7 : path: /instance_groups/name=shield/job/name=shield-agent/properties/log_level?
Must be : path: /instance_groups/name=shield/jobs/name=shield-agent/properties/log_level?
We have to use jobs instead of job
Best regards,
Greetings,
we are trying to deploy the new version of SHIELD (7.0.1), and our deployment manifest seems broken. The error we are getting is the following
Started preparing deployment > Preparing deployment. Failed: Unable to process links for deployment. Errors are:
- Can't find link with type 'sql' for job 'shield' in deployment 'shield-yyy' (00:00:00)
The difference in our manifest is that we are adding the database url directly instead of using links and that there no deployment offering such a link. The manifest looks like that
- name: shield-daemon
properties:
shield:
daemon:
auth:
api_keys:
autoprovision: XXXXX
basic_password: YYYYY
basic_user: admin
database:
db: shielddb
host: 10.x.y.z
password: zzzzzzzz
port: 3306
type: mysql
username: shieldadmin
ssh_private_key: |
-----BEGIN RSA PRIVATE KEY-----
What do you think of having the sql link optional in the spec file?
Also shield-daemon is starting even if the database access is not established (monit summary give running), is that on purpose?
Thanks!
https://github.com/starkandwayne/shield-boshrelease/blob/master/manifests/shield.yml#L43-L44
by switching from rsa to ssh we can get the proper format for the public_key this would allow us to no longer supply private keys to the agent (currently the public_key is calculated during startup).
bosh int <(bosh int <(echo -e "foo: ((foo))\nvariables:\n- name: foo\n type: ssh") --vars-store=/dev/null) --path /foo/public_key
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDG+KD6nt10/hK+byUtJyX0HPbLbXtsf0CjrWSCP+HIm/aCvFzpOyefYoivg1TIpVcAyHUx5bfVavlaVUCYaQOMIH6gmG9oModKly4wOEe572ewE0GbJyCT6L+aSo1nOI6+1jNnKchsT/WLNqwA0s+6NrJXwOyqKmz+JMCWodjKzh3eLbM6xK8KH9q3ouzfjsBqpO1fCbkE3Lptoz7iiPfjA8T//VrdMZ/gH58TerQEDD2E7aksXWFkL96IQMRIy/KHcWDFbypZufmifPtHzbu6DAMa/8hZeDOg1jRNPHqLu4r+F6saSK7cMWG9xqkTp+1ZktKg3lQiymWn7TApgzv9
Hi,
I recently upgraded from release v6.3.6 to v6.4.0, and I'm facing 3 issues.
shield/0
VM, postgres startup produces 1.7 GB of logs which is quite a lot.# du -sh /var/vcap/store/postgres-9.4/pg_log/startup.log
1.7G /var/vcap/store/postgres-9.4/pg_log/startup.log
By the way, shouldn't these startup logs be located inside the /var/vcap/data/sys/log/postgres
directory?
shield/0
VM, the postgres job produces 27+ GB of logs each day. This fills my production disk very quickly.For example, I recreated the shield/0
VM yesterday and here is what I have:
# du -sh /var/vcap/data/sys/log/postgres/postgresql.log
27G /var/vcap/data/sys/log/postgres/postgresql.log
I didn't find a way to control the verbosity of these logs in the postgres-9.4
job spec. Is there any other way?
Would it be possible for the BOSH release to implement some rolling policies on these log files? Or is it up to the deployer to implement some BOSH add-on that does the trick?
shield/0
VM, the /var/vcap/jobs/shield-agent/bin/ctl
startup script has an issue producing this error:# tail /var/vcap/data/sys/log/monit/shield-agent.log
/var/vcap/jobs/shield-agent/bin/ctl: line 22: syntax error near unexpected token `}'
This is due to both if_p("shield.agent.autoprovision")
and if_p("shield.agent.daemon_public_key")
evaluating to false
and not rendering their respective blocks. Even when these are set in the deployment properties, which is strange. Anyway, this issue could easily be circumvented with a mere call to /bin/true
within the curly braces so that the block is never empty from a Bash syntax point of view.
Hi,
since we are using bosh, we need everything to be deployed through the bosh deployment manifest and not have any manual operation on the API. Currently it is only possible to add a single store (or schedule or retention policy). We would like to extend that.
For example to have something like that in the deployment manifests
shield:
schedules:
daily: daily 2am
monthly: 2st sunday at 0am
weekly: sundays 9am
which will create
shield list schedules
Using https://10.244.2.2:443 (default) as SHIELD backend
Name Summary Frequency / Interval (UTC)
==== ======= ==========================
daily daily 2am
monthly 2st sunday at 0am
weekly sundays 9am
Is it something you would like to integrate ?
Hi there,
the current bosh release does not include a manifest for testing SHIELD with mariadb.
Testing fast with mariadb in a bosh-lite would great help to find fast potential issues.
Thanks
Hello,
I am using the shield-boshrelease to backup the Cloud Foundry blobstore.
The release are using bzip2 to compress backups so it may take a long time (for huge amount of data).
My question is : what about to have a new feature that enable to change the compress algorithm (user can choose between tar,bzip2,..), so for example having it as a parameter in the bosh-release manifest.
Best regards,
I cannot find where these properties are being referenced.
It look like the logic on the post-start script has changed to expect the shield.agent.autoprovision
property to be the server endpoint.
Hello,
I am trying to use the s3 plugin behind a http proxy.
Only a socks5 proxy is allowed by the s3 configuration plugin.
Have you any idea how to run the s3 plugin behind a http proxy?
Best regards,
Latest master builds of shieldd
sport a new /v1/meta/pubkey
endpoint that returns the SSH public key, suitable for inclusion into an agent's authorized_keys file.
It would be great if we had a new property to enable this, like shield.agent.autoprovision
which could be set to the base URL of the shieldd to get the key from. Absense (nullification / empty-stringitude) of this key would then signify "don't autoprovision"
every purge task has the following output:
TASK FAILED!! shield worker 4 unable to connect to localhost:5444 (ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain)
I assume its just a missconfiguration. Posting it here for feedback.
We should add errands, or auto-provisioning tasks that are capable of analyzing the bosh manifest + current deployment to instantiate jobs/targets/retention policies/stores/schedules with little end-user effort.
The binary path for the database binaries are hard coded into the code. It would cleaner and easier to change if they were made into a constant and that constant is referenced.
If the username of the postgres user for the shield database is changed, the shield daemon will fail to start because it won't be able to access any of the existing relations with the new users. We need to add GRANT statements for the relations so that the new user can take over.
Hey
This is a feature-request
.
It would be great if the shield-boshdeployment
would support bosh
links which would allow to install nginx
and shield
in a dedicated vm and stuff like shield.daemon.database.host
would not have to be set statically.
(I hope to find some time to implement this on my own.)
cheers
We are thinking about deploying the shield-agent within the director vm using bosh-init in order to provide automated bosh director deployments to other people. It should work, right?
So we tried to deploy a bosh director instance with the shield-agent release included ... but currently bosh-init is not able to render networking properties like my_ip
defined in https://github.com/starkandwayne/shield-boshrelease/blob/master/jobs/shield-agent/templates/bin/post-start.erb#L25 and it fails.
Is it worth defining an alternative property to assign the IP address: if the new property is defined then take the value, otherwise fall back on the current implementation. What do you think about that?
Hey
This is a feature request.
Can you add xtrabackup
as package/job (probably best as new job)? There was a plugin added to shield
and it would be awesome if this could be used with this boshrelease out of the box.
(I’d do this on my own but I currently don’t understand where the code needs to be. According to the bosh
documentation I’d expect something like a blob
directory, where the xtrabackup
code/application lives but there is none)
cheers
$ shield status -k
Using http://52.40.115.221 (proto) as SHIELD backend
Name: acf-global- S.H.I.E.L.D
API Version: (development)
We have shield deployed to multiple environments and configure the agent job at site level as follows:
properties:
shield:
job:
name: (( grab name))
<etc>
Now that only plural version of jobs
key is supported we cannot dynamically determin the job name when merging the templates via spruce / genesis.
Hi guys,
We've seen that these parameters seem to be unique among multiple shield agents but can be easily overriden. Let me clarify :)
I have 2 shield agents on 2 separate deployments that communicate to a shield daemon. We'll call them node_1 and node_2. Shield agent is configured this way :
properties:
shield:
agent:
autoprovision: https://xxxx/
provisioning_key: xxxxx
targets:
node1-data-dir:
plugin: fs
config:
base_dir: /var/vcap/data
schedules:
daily: daily 3am
retention-policies:
weekly: 7d
jobs:
backup_data_dir:
schedule: daily
retention: weekly
target: node1-data-dir
store: shield-local-storage
properties:
shield:
agent:
autoprovision: https://xxxx/
provisioning_key: xxxxx
targets:
node2-data-dir:
plugin: fs
config:
base_dir: /var/vcap/data
schedules:
daily: daily 4am
retention-policies:
weekly: 7d
jobs:
backup_data_dir:
schedule: daily
retention: weekly
target: node2-data-dir
store: shield-local-storage
When I deploy both nodes, it results getting only 1 job and 1 schedule with deployment success. I understand that each keys is meant to be unique, it's a proper behaviour.
Problem is that one shield agent can override another one's configuration, it can result in canceled backups and data loss.
What do you guys think ?
best
Hello! I've been using shield on bosh.
I finished about simple test that backup & restore of postgresql-docker.
So, I've tested backup & restore of cloud foundry.
Below is my plan
first, CCDB
second, UAA
...
Applcation of blobstore
...
Well.. I'm blocked by restore of CCDB. I did only backup,
But, It is not definitely. bcuz restore is not working.
my scenario is very simple.
task log is very long. So, I attached txt file.
It is return exiting 0.
cf restore log (it is not working).txt
Below is my settings.
I set setting that is same success of postgresql docker.
I wait your reply :D
Build a pipeline for building, vetting and testing this BOSH release.
When using the shield ui I see the following error in /var/vcap/sys/log/shield/shieldd.log
:
2018-01-04 11:41:58.643334246 +000 UTC /var/vcap/packages/shield/bin/shieldd: ERROR: GET /v2/tenants/0ba8c0b6-3976-44eb-b8f5-10a36a82c04e/health errored: no such column: j.fixed_key
Each click in the ui generates the above log line. I'm using v8.3.0. And have just one agent and an empty database.
Hey
Apparently it is currently not possible to run postgres
in another vm/instance
then nginx+shield-daemon
.
Having a bosh2.0
style manifest which puts everything onto one VM:
---
name: shield
director_uuid: <uuid>
releases:
- name: shield
version: latest
stemcells:
- alias: trusty
name: bosh-vsphere-esxi-ubuntu-trusty-go_agent
version: latest
instance_groups:
- azs:
- az1
name: shield
instances: 1
vm_type: 2cpu_4gbram_20gbdisk
stemcell: trusty
networks:
- name: static
static_ips: 10.35.96.66
jobs:
- name: nginx
release: shield
- name: shield-daemon
release: shield
- name: shield-agent
release: shield
- name: postgres
release: shield
properties:
shield:
daemon:
domain: 10.35.96.66
ssh_private_key: |+
<private-key-stuff>
database:
type: postgres
host: 10.35.96.66
port: 5524
username: shieldadmin
password: admin
db: shielddb
agent:
autoprovision: "https://10.35.96.66"
databases:
address: 10.35.96.66
databases:
- citext: true
name: shielddb
tag: shield
- citext: true
name: sessionsdb
tag: sessions
db_scheme: postgres
port: 5524
roles:
- name: shieldadmin
password: admin
tag: admin
update:
canaries: 1
max_in_flight: 50
canary_watch_time: 5000-120000
update_watch_time: 5000-120000
This is working fine.
When trying to give postgres
a dedicated vm:
---
name: shield-multi-vms
director_uuid: <uuid>
releases:
- name: shield
version: latest
stemcells:
- alias: trusty
name: bosh-vsphere-esxi-ubuntu-trusty-go_agent
version: latest
instance_groups:
- azs:
- az1
name: shield
instances: 1
vm_type: 2cpu_4gbram_20gbdisk
stemcell: trusty
networks:
- name: static
static_ips: 10.35.96.67
jobs:
- name: nginx
release: shield
- name: shield-daemon
release: shield
- name: shield-agent
release: shield
properties:
shield:
daemon:
domain: 10.35.96.67
ssh_private_key: |+
<private-key>
database:
type: postgres
host: 10.35.96.68
port: 5524
username: shieldadmin
password: admin
db: shielddb
agent:
autoprovision: "https://10.35.96.67"
- azs:
- az1
name: postgres
instances: 1
vm_type: 2cpu_4gbram_20gbdisk
stemcell: trusty
networks:
- name: static
static_ips: 10.35.96.68
jobs:
- name: postgres
release: shield
properties:
databases:
address: 10.35.96.68
databases:
- citext: true
name: shielddb
tag: shield
- citext: true
name: sessionsdb
tag: sessions
db_scheme: postgres
port: 5524
roles:
- name: shieldadmin
password: admin
tag: admin
update:
canaries: 1
max_in_flight: 50
canary_watch_time: 5000-120000
update_watch_time: 5000-120000
fails with:
Director task 481
Started preparing deployment > Preparing deployment. Done (00:00:00)
Started preparing package compilation > Finding packages to compile. Done (00:00:00)
Started creating missing vms
Started creating missing vms > shield/e636d5bc-4e27-4800-8d1c-4a9286207f5a (0)
Started creating missing vms > postgres/d241e01b-1483-429e-8da8-1c9137a66663 (0)
Done creating missing vms > shield/e636d5bc-4e27-4800-8d1c-4a9286207f5a (0) (00:00:52)
Done creating missing vms > postgres/d241e01b-1483-429e-8da8-1c9137a66663 (0) (00:00:54)
Done creating missing vms (00:00:54)
Started updating instance shield > shield/e636d5bc-4e27-4800-8d1c-4a9286207f5a (0) (canary). Failed: 'shield/0 (e636d5bc-4e27-4800-8d1c-4a9286207f5a)' is not running after update. Review logs for failed jobs: shield-daemon (00:02:23)
Error 400007: 'shield/0 (e636d5bc-4e27-4800-8d1c-4a9286207f5a)' is not running after update. Review logs for failed jobs: shield-daemon
Task 481 error
The shield-daemon
fails because it cannot connect to postgres
, because there is not even a postgres
running:
/d241e01b-1483-429e-8da8-1c9137a66663:/var/vcap/sys/log$ ps aux |grep postgres
/d241e01b-1483-429e-8da8-1c9137a66663:/var/vcap/sys/log$ netstat -tulpen
(No info could be read for "-p": geteuid()=1001 but you should be root.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State User Inode PID/Program name
tcp 0 0 0.0.0.0:32785 0.0.0.0:* LISTEN 103 10455 -
tcp 0 0 127.0.0.1:33331 0.0.0.0:* LISTEN 0 12301 -
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 0 11300 -
tcp 0 0 127.0.0.1:2822 0.0.0.0:* LISTEN 0 12300 -
tcp 0 0 127.0.0.1:2825 0.0.0.0:* LISTEN 0 11465 -
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 0 10405 -
tcp6 0 0 :::22 :::* LISTEN 0 11302 -
tcp6 0 0 :::47455 :::* LISTEN 103 10459 -
tcp6 0 0 :::111 :::* LISTEN 0 10408 -
udp 0 0 0.0.0.0:37550 0.0.0.0:* 103 10452 -
udp 0 0 0.0.0.0:735 0.0.0.0:* 0 10404 -
udp 0 0 127.0.0.1:756 0.0.0.0:* 0 10436 -
udp 0 0 0.0.0.0:111 0.0.0.0:* 0 10399 -
udp6 0 0 :::735 :::* 0 10407 -
udp6 0 0 :::111 :::* 0 10406 -
udp6 0 0 :::53597 :::* 103 10457 -
Can we somehow make it possible to separate the postgres
instance from shield-daemon
?
cheers
UAA based oauth requires a provider_url parameter in order to work. We should add a property for this so people can use UAA based oauth.
Hello,
I am using the shield-boshrelease to backup the Cloud Foundry blobstore but i got an error "timeout".
The size of the blobstore that i want to backup is 12 GB.
Running backup task (using bzip2 compression)
=============================================
Get http://mystorage.com/mybucket/?location=: dial tcp X.X.X.X:80: i/o timeout
bsdtar: Write error
TASK FAILED!! shield worker 2 failed to execute the command against the remote agent XX.XX.XX.XX:5444 (Process exited with: 4. Reason was: ())
Unable to exec '/var/vcap/packages/bsdtar/bin/bsdtar': exit status 1
EXITING 4
TASK FAILED!! No restore key detected in worker 2. Cowardly refusing to create an archive record
I am using the "fs" plugin as target and "scality" as store.
I tested the same configurations with an other folder that contains less data and it worked well.
Is it about file size ?
Best regards,
Getting following error while rendering templates:
Deploying:
Building state for instance 'bosh/0':
Rendering job templates for instance 'bosh/0':
Rendering templates for job 'shield-agent/8ad3e3f29dcd1b8d7a8810f45fd4c929b7928d27':
Rendering template src: config/agent.conf, dst: config/agent.conf:
Rendering template src: /home/vcap/.bosh/installations/df1091ff-fe5b-4955-7e1a-cb30c4890b7a/tmp/bosh-release-job341768806/templates/config/agent.conf, dst: /home/vcap/.bosh/installations/df1091ff-fe5b-4955-7e1a-cb30c4890b7a/tmp/
rendered-jobs252948898/config/agent.conf:
Running ruby to render templates:
Running command: 'ruby /home/vcap/.bosh/installations/df1091ff-fe5b-4955-7e1a-cb30c4890b7a/tmp/erb-renderer047888025/erb-render.rb /home/vcap/.bosh/installations/df1091ff-fe5b-4955-7e1a-cb30c4890b7a/tmp/erb-renderer047888025
/erb-context.json /home/vcap/.bosh/installations/df1091ff-fe5b-4955-7e1a-cb30c4890b7a/tmp/bosh-release-job341768806/templates/config/agent.conf /home/vcap/.bosh/installations/df1091ff-fe5b-4955-7e1a-cb30c4890b7a/tmp/rendered-jobs252948898
/config/agent.conf', stdout: '', stderr: '/home/vcap/.bosh/installations/df1091ff-fe5b-4955-7e1a-cb30c4890b7a/tmp/erb-renderer047888025/erb-render.rb:189:in `rescue in render': Error filling in template '/home/vcap/.bosh/installations/df1
091ff-fe5b-4955-7e1a-cb30c4890b7a/tmp/bosh-release-job341768806/templates/config/agent.conf' for shield-agent/0 (line 3: #<TypeError: no implicit conversion of nil into String>) (RuntimeError)
from /home/vcap/.bosh/installations/df1091ff-fe5b-4955-7e1a-cb30c4890b7a/tmp/erb-renderer047888025/erb-render.rb:175:in `render'
from /home/vcap/.bosh/installations/df1091ff-fe5b-4955-7e1a-cb30c4890b7a/tmp/erb-renderer047888025/erb-render.rb:200:in `<main>'
':
exit status 1
Exit code 1
Which trace back to .gsub('(name)', spec.name)
, apparently spec.name
is nill because bosh create-env does not set it here. It does set spec.job.name
.
In our case we are specifying an agent name without any placeholders so defaulting spec.name
to ''
as is done for spec.az
would be sufficient.
Would a PR for this change be merged?
This "convenience" causes more problems than it solves.
Instead, we should provide a static key for people wanting to experiment with BOSH-lite, and require a value for the public/private RSA keypair (in SSH format) on real deployments.
Then we can get rid of the packages altogether.
When using a floating IP in Openstack, HTTP does not redirect to HTTPS it hangs, attempts to redirect to the internal IP, and then fails.
e.g. http://172.x.x.114
-> http://10.4.1.32
not https://172.x.x.114
On the shield/0
VM, the postgres job produces 27+ GB of logs each day. This fills my production disk very quickly.
For example, I recreated the shield/0
VM yesterday and here is what I have:
# du -sh /var/vcap/data/sys/log/postgres/postgresql.log
27G /var/vcap/data/sys/log/postgres/postgresql.log
I didn't find a way to control the verbosity of these logs in the postgres-9.4
job spec. Is there any other way?
Would it be possible for the BOSH release to implement some rolling policies on these log files? Or is it up to the deployer to implement some BOSH add-on that does the trick?
(This issue was originally part of issue #48.)
We deployed a bosh with bosh-init and included the shield agent with target and job sections configured. When registering with the shield daemon the 'Remote IP:Port' was configured as :5444
. It looks like this line where the ip is being determined isn't interpreted with bosh-init the same way as with regular bosh.
Hello,
I found some bugs under the import errand .
The current monit file content is wrong , it will try to run the wrong job (store) :
check process store
with pidfile /var/vcap/sys/run/shield/store.pid
start program "/var/vcap/jobs/store/bin/nginx start"
stop program "/var/vcap/jobs/store/bin/nginx stop"
group vcap
==>The monit file should be empty for errand jobs, otherwise bosh will try to restart the job after executing the run.sh script.
The current run.sh script contain an error + need more things to able to establish connection with the shield core
a) Line 11 :
the script is using a wrong name to call the shield cli
/var/vcap/packages/shield/bin/buckler
we have to use
/var/vcap/packages/shield/bin/shield
b) Environment Variables:
the run.sh script use the shield cli to import backups to the shield core , we need to provide some environment variables:
SHIELD_CORE_TOKEN
SHIELD_CORE
SHIELD_API
C)Login to the shield core
After setting the environment variables, we need to login to the shield core .
We may need to use :
/var/vcap/packages/shield/bin/shield api $SHIELD_API shield -k
/var/vcap/packages/shield/bin/shield login
Best regards,
Once its cut, I'll look at upgrading https://github.com/genesis-community/shield-genesis-kit
Hello,
The #58 PR aim to ensure that the postgres plugin finds the bin_dir by default.
Let's take a look to the following code :
https://github.com/starkandwayne/shield-boshrelease/blob/master/jobs/postgres/templates/bin/postgres_start.sh
PACKAGE_DIR=/var/vcap/packages/postgres-9.4
PACKAGE_DIR_OLD=/var/vcap/packages/postgres
Line 26 ==>
# Ensure that default postgres bin_dir of the postgres plugin is able to locate the bin_dir
if [ ! -d $PACKAGE_DIR ]; then
ln -s $PACKAGE_DIR $PACKAGE_DIR_OLD
fi
I guess that the test must be :
if [ ! -d $PACKAGE_DIR_OLD]; then
ln -s $PACKAGE_DIR $PACKAGE_DIR_OLD
fi
Best regards
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.