shieldproject / shield-boshrelease Goto Github PK

Rentenions, schedules and stores can be specified in the manifest. Can we also add support for targets and jobs?

On the shield/0 VM, the /var/vcap/jobs/shield-agent/bin/ctl startup script has an issue producing this error:

# tail  /var/vcap/data/sys/log/monit/shield-agent.log
/var/vcap/jobs/shield-agent/bin/ctl: line 22: syntax error near unexpected token `}'

This is due to both if_p("shield.agent.autoprovision") and if_p("shield.agent.daemon_public_key") evaluating to false and not rendering their respective blocks. Even when these are set in the deployment properties, which is strange. Anyway, this issue could easily be circumvented with a mere call to /bin/true within the curly braces so that the block is never empty from a Bash syntax point of view.

(This issue was originally part of issue #48.)

In the post-start script the uuid isn't being passed in to the shield edit-xxx command.

If the Agent and the Daemon are co-located on the same host, the agent will create a blank "authorized_keys" file if it starts before the daemon. The agent should not be allowed to create a blank file and should fail, so that it can try to start again once Monit successfully starts the daemon.

(migrated here from shieldproject/shield#244)

Out-of-the-box, SHIELD should just use a local SQLite3 database - no need for fancy PostgreSQL or MySQL configuration.

Need postgres server running to do migration but job doesn't exist in release in v8. Maybe use https://github.com/cloudfoundry/postgres-release as stop gap for upgrade to v8?

Hey

It would be great if the agent-pgtools would come with a newer version of postgres. Something like 9.6.x would be fine, since this is latest stable.

An other option would be to split the package into multiple versions like it’s done for mongo.

I’m happy to do so (and also kinda working on it), but it would be cool to agree on how to do it and then do it :)

cheers

The property descriptions are a little vague in some areas.

Also, the README should be updated to include some common configurations, and perhaps more in-depth explanation of the properties, jobs, and how they interrelate.

Looks like it fails to download a tarball:

Building release tarball from manifest '/mnt/tmp/downloader-GitDownloader598508896/releases/shield/shield-2.1.yml': Running bosh create release: Running command: 'bosh create release /mnt/tmp/downloader-GitDownloader598508896/releases/shield/shield-2.1.yml', stdout: 'Recreating release from the manifest

Copying packages
----------------
bsdtar (fe279cae5da0aa7b9cbf07238dad8afba6c8edc7) 
common (e401816a4748292163679fafcbd8f818ed8154a5) 
elasticsearch-cloud-aws-plugin (a8701cdd6895f248d4d51fd918e4d71daa678b4f) 
generated_agent_key (d130726fd279b4cfb8ef211daad7b64e62bb888d) 
generated_daemon_key (d130726fd279b4cfb8ef211daad7b64e62bb888d) 
golang (201283451505af9fc84d23998b96584a84a45e57) 
Downloading from blobstore (id=890933cd-bf31-4d77-a44c-54dcc0d8b030)...

', stderr: 'Blobstore error: sha1 mismatch expected=fd14409a79e30a47d04b4ab15bf96798fd7f3b37 actual=20d09c7867ecf27a8aefd614dc737266d405e45b': exit status 1

Hello, I'v been testing shiled on my local bosh-lite.

command : shield run [jobs name] -k
result :

shieldd log :
2017-01-23 04:07:35.545134238 +0000 UTC shieldd: INFO: schedule adhoc backup job
2017-01-23 04:07:35.545204147 +0000 UTC shieldd: INFO: scheduling immediate (ad hoc) execution of job shield [625738c4-db29-46fc-8fd7-af5ee6d11e15]
2017-01-23 04:07:35.547435938 +0000 UTC shieldd: INFO: schedule task dab38689-15f2-40aa-861a-9f1c256316db with deadline {{63620784455 547401390 0x1157da0}}
2017/01/23 04:07:35 http: multiple response.WriteHeader calls
2017-01-23 04:07:35.966399278 +0000 UTC shieldd: INFO: sent a task to a worker
2017-01-23 04:07:35.966703638 +0000 UTC shieldd: INFO: dab38689-15f2-40aa-861a-9f1c256316db> TASK FAILED!! shield worker 5 unable to connect to 10.244.204.2:443 (ssh: handshake failed: ssh: invalid packet length, packet too large)
2017-01-23 04:07:35.975731706 +0000 UTC shieldd: WARNING: dab38689-15f2-40aa-861a-9f1c256316db: task failed!
2017-01-23 05:30:40.733353999 +0000 UTC shieldd: INFO: schedule adhoc backup job
2017-01-23 05:30:40.73347915 +0000 UTC shieldd: INFO: scheduling immediate (ad hoc) execution of job shield [625738c4-db29-46fc-8fd7-af5ee6d11e15]
2017-01-23 05:30:40.735746302 +0000 UTC shieldd: INFO: schedule task 52110f7e-636f-43e8-9a97-4d876cab765a with deadline {{63620789440 735728122 0x1157da0}}
2017/01/23 05:30:40 http: multiple response.WriteHeader calls
2017-01-23 05:30:40.963795183 +0000 UTC shieldd: INFO: sent a task to a worker

my bosh vms, below
+-------------------------------------------------+---------+-----+----------+--------------+
| VM | State | AZ | VM Type | IPs |
+-------------------------------------------------+---------+-----+----------+--------------+
| shield/0 | running | n/a | small_z1 | 10.244.204.2 |
+-------------------------------------------------+---------+-----+----------+--------------+
+---------------------------------------------------------------+---------+-----+----------+-------------+
| VM | State | AZ | VM Type | IPs |
+---------------------------------------------------------------+---------+-----+----------+-------------+
| postgresql_docker_z1/0 | running | n/a | small_z1 | 10.244.30.6 |
+---------------------------------------------------------------+---------+-----+----------+-------------+
+---------------------------------------------------------------+---------+-----+----------+-------------+
| VM | State | AZ | VM Type | IPs |
+---------------------------------------------------------------+---------+-----+----------+-------------+
| postgresql_docker_z1/0 | running | n/a | small_z1 | 10.244.40.6 |
+---------------------------------------------------------------+---------+-----+----------+-------------+

my job
Name: shield
Paused: N
Retention Policy: default
Expires in: 100 days
Schedule Policy: default
Target: postgres
Target Endpoint: {"pg_user":"target", "pg_password":"target", "pg_host":"10.244.40.6", "pg_port":"5432"}
Remote IP: 10.244.204.2:5524
Store: postgres
Store Endpoint: {"pg_user": "store", "pg_password": "store", "pg_host": "10.244.30.6", "pg_port": "5432"}
Notes: shield

IP & Port of Target & Store is correct. I saw init.script of test-bed.
I've thought that i don't understand shield.

I wait your reply.

The xtrabackup binary version 2.2.10 shipped by the cf-mysql release v26 doesn't support the --move-back option on which the xtrabackup SHIELD plugin relies.

In such situation, the xtrabackup job of the shield-boshrelease is supposed to help. Unfortunately, it brings an xtrabackup package whose name conflicts with the one already installed by the mysql job of the cf-mysql release.

A solution for the SHIELD release would be to rename the xtrabackup package into xtrabackup-2.4.5 so that it doesn't conflict with the default xtrabackup package shipped by the mysql job of the cf-mysql release.

Hello,

I think that the log-level.yml log-level.yml operator file contain a typo.

Line 3 : path: /instance_groups/name=shield/job/name=shield-daemon/properties/log_level?
Must be : path: /instance_groups/name=shield/jobs/name=shield-daemon/properties/log_level?

Line 7 : path: /instance_groups/name=shield/job/name=shield-agent/properties/log_level?
Must be : path: /instance_groups/name=shield/jobs/name=shield-agent/properties/log_level?

We have to use jobs instead of job

Best regards,

Greetings,
we are trying to deploy the new version of SHIELD (7.0.1), and our deployment manifest seems broken. The error we are getting is the following

 Started preparing deployment > Preparing deployment. Failed: Unable to process links for deployment. Errors are:
  - Can't find link with type 'sql' for job 'shield' in deployment 'shield-yyy' (00:00:00)

The difference in our manifest is that we are adding the database url directly instead of using links and that there no deployment offering such a link. The manifest looks like that

  - name: shield-daemon
    properties:
      shield:
        daemon:
          auth:
            api_keys:
              autoprovision: XXXXX
            basic_password: YYYYY
            basic_user: admin
          database:
            db: shielddb
            host: 10.x.y.z
            password: zzzzzzzz
            port: 3306
            type: mysql
            username: shieldadmin
          ssh_private_key: |
            -----BEGIN RSA PRIVATE KEY-----

What do you think of having the sql link optional in the spec file?
Also shield-daemon is starting even if the database access is not established (monit summary give running), is that on purpose?

Thanks!

https://github.com/starkandwayne/shield-boshrelease/blob/master/manifests/shield.yml#L43-L44

by switching from rsa to ssh we can get the proper format for the public_key this would allow us to no longer supply private keys to the agent (currently the public_key is calculated during startup).

bosh int <(bosh int <(echo -e "foo: ((foo))\nvariables:\n- name: foo\n  type: ssh") --vars-store=/dev/null) --path /foo/public_key
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDG+KD6nt10/hK+byUtJyX0HPbLbXtsf0CjrWSCP+HIm/aCvFzpOyefYoivg1TIpVcAyHUx5bfVavlaVUCYaQOMIH6gmG9oModKly4wOEe572ewE0GbJyCT6L+aSo1nOI6+1jNnKchsT/WLNqwA0s+6NrJXwOyqKmz+JMCWodjKzh3eLbM6xK8KH9q3ouzfjsBqpO1fCbkE3Lptoz7iiPfjA8T//VrdMZ/gH58TerQEDD2E7aksXWFkL96IQMRIy/KHcWDFbypZufmifPtHzbu6DAMa/8hZeDOg1jRNPHqLu4r+F6saSK7cMWG9xqkTp+1ZktKg3lQiymWn7TApgzv9

Hi,

I recently upgraded from release v6.3.6 to v6.4.0, and I'm facing 3 issues.

1. On the shield/0 VM, postgres startup produces 1.7 GB of logs which is quite a lot.

# du -sh /var/vcap/store/postgres-9.4/pg_log/startup.log
1.7G	/var/vcap/store/postgres-9.4/pg_log/startup.log

By the way, shouldn't these startup logs be located inside the /var/vcap/data/sys/log/postgres directory?

2. On the shield/0 VM, the postgres job produces 27+ GB of logs each day. This fills my production disk very quickly.

For example, I recreated the shield/0 VM yesterday and here is what I have:

# du -sh /var/vcap/data/sys/log/postgres/postgresql.log
27G	/var/vcap/data/sys/log/postgres/postgresql.log

I didn't find a way to control the verbosity of these logs in the postgres-9.4 job spec. Is there any other way?
Would it be possible for the BOSH release to implement some rolling policies on these log files? Or is it up to the deployer to implement some BOSH add-on that does the trick?

3. On the shield/0 VM, the /var/vcap/jobs/shield-agent/bin/ctl startup script has an issue producing this error:

# tail  /var/vcap/data/sys/log/monit/shield-agent.log
/var/vcap/jobs/shield-agent/bin/ctl: line 22: syntax error near unexpected token `}'

This is due to both if_p("shield.agent.autoprovision") and if_p("shield.agent.daemon_public_key") evaluating to false and not rendering their respective blocks. Even when these are set in the deployment properties, which is strange. Anyway, this issue could easily be circumvented with a mere call to /bin/true within the curly braces so that the block is never empty from a Bash syntax point of view.

Hi,
since we are using bosh, we need everything to be deployed through the bosh deployment manifest and not have any manual operation on the API. Currently it is only possible to add a single store (or schedule or retention policy). We would like to extend that.

For example to have something like that in the deployment manifests

shield:
  schedules:
       daily: daily 2am
       monthly: 2st sunday at 0am
       weekly: sundays 9am

which will create

shield  list  schedules
Using https://10.244.2.2:443 (default) as SHIELD backend

Name     Summary  Frequency / Interval (UTC)
====     =======  ==========================
daily             daily 2am
monthly           2st sunday at 0am
weekly            sundays 9am

Is it something you would like to integrate ?

Hi there,
the current bosh release does not include a manifest for testing SHIELD with mariadb.
Testing fast with mariadb in a bosh-lite would great help to find fast potential issues.

Thanks

Hello,

I am using the shield-boshrelease to backup the Cloud Foundry blobstore.

The release are using bzip2 to compress backups so it may take a long time (for huge amount of data).

My question is : what about to have a new feature that enable to change the compress algorithm (user can choose between tar,bzip2,..), so for example having it as a parameter in the bosh-release manifest.

Best regards,

I cannot find where these properties are being referenced.
It look like the logic on the post-start script has changed to expect the shield.agent.autoprovision property to be the server endpoint.

Hello,

I am trying to use the s3 plugin behind a http proxy.

Only a socks5 proxy is allowed by the s3 configuration plugin.

Have you any idea how to run the s3 plugin behind a http proxy?

Best regards,

Latest master builds of shieldd sport a new /v1/meta/pubkey endpoint that returns the SSH public key, suitable for inclusion into an agent's authorized_keys file.

It would be great if we had a new property to enable this, like shield.agent.autoprovision which could be set to the base URL of the shieldd to get the key from. Absense (nullification / empty-stringitude) of this key would then signify "don't autoprovision"

every purge task has the following output:

TASK FAILED!!  shield worker 4 unable to connect to localhost:5444 (ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain)

I assume its just a missconfiguration. Posting it here for feedback.

We should add errands, or auto-provisioning tasks that are capable of analyzing the bosh manifest + current deployment to instantiate jobs/targets/retention policies/stores/schedules with little end-user effort.

The binary path for the database binaries are hard coded into the code. It would cleaner and easier to change if they were made into a constant and that constant is referenced.

If the username of the postgres user for the shield database is changed, the shield daemon will fail to start because it won't be able to access any of the existing relations with the new users. We need to add GRANT statements for the relations so that the new user can take over.

Hey

This is a feature-request.

It would be great if the shield-boshdeployment would support bosh links which would allow to install nginx and shield in a dedicated vm and stuff like shield.daemon.database.host would not have to be set statically.

(I hope to find some time to implement this on my own.)

cheers

We are thinking about deploying the shield-agent within the director vm using bosh-init in order to provide automated bosh director deployments to other people. It should work, right?

So we tried to deploy a bosh director instance with the shield-agent release included ... but currently bosh-init is not able to render networking properties like my_ip defined in https://github.com/starkandwayne/shield-boshrelease/blob/master/jobs/shield-agent/templates/bin/post-start.erb#L25 and it fails.

Is it worth defining an alternative property to assign the IP address: if the new property is defined then take the value, otherwise fall back on the current implementation. What do you think about that?

Hey

This is a feature request.

Can you add xtrabackup as package/job (probably best as new job)? There was a plugin added to shield and it would be awesome if this could be used with this boshrelease out of the box.

(I’d do this on my own but I currently don’t understand where the code needs to be. According to the bosh documentation I’d expect something like a blob directory, where the xtrabackup code/application lives but there is none)

cheers

We have shield deployed to multiple environments and configure the agent job at site level as follows:

properties:
  shield:
    job:
      name: (( grab name))
      <etc>

Now that only plural version of jobs key is supported we cannot dynamically determin the job name when merging the templates via spruce / genesis.

Hi guys,

We've seen that these parameters seem to be unique among multiple shield agents but can be easily overriden. Let me clarify :)

I have 2 shield agents on 2 separate deployments that communicate to a shield daemon. We'll call them node_1 and node_2. Shield agent is configured this way :

node_1

properties:
  shield:
    agent:
      autoprovision: https://xxxx/
    provisioning_key: xxxxx
    targets:
      node1-data-dir:
        plugin: fs
        config:
          base_dir: /var/vcap/data
    schedules:
      daily: daily 3am
    retention-policies:
      weekly: 7d
    jobs:
      backup_data_dir:
        schedule:  daily
        retention: weekly
        target:    node1-data-dir
        store:     shield-local-storage

node_2

properties:
  shield:
    agent:
      autoprovision: https://xxxx/
    provisioning_key: xxxxx
    targets:
      node2-data-dir:
        plugin: fs
        config:
          base_dir: /var/vcap/data
    schedules:
      daily: daily 4am
    retention-policies:
      weekly: 7d
    jobs:
      backup_data_dir:
        schedule:  daily
        retention: weekly
        target:    node2-data-dir
        store:     shield-local-storage

When I deploy both nodes, it results getting only 1 job and 1 schedule with deployment success. I understand that each keys is meant to be unique, it's a proper behaviour.
Problem is that one shield agent can override another one's configuration, it can result in canceled backups and data loss.

What do you guys think ?

best

/CC @nvekemans-ext-orange

Hello! I've been using shield on bosh.
I finished about simple test that backup & restore of postgresql-docker.
So, I've tested backup & restore of cloud foundry.

Below is my plan
first, CCDB
second, UAA
...
Applcation of blobstore
...

Well.. I'm blocked by restore of CCDB. I did only backup,
But, It is not definitely. bcuz restore is not working.

my scenario is very simple.

1. I have a space on cf.
2. backup it.
3. delete the space.
4. restore it.
5. check the space on CCDB.
   but, it is not working about restore.

task log is very long. So, I attached txt file.
It is return exiting 0.
cf restore log (it is not working).txt

Below is my settings.

1. Job on shield for Cloud Foundry
   

2. bosh vms
   

3. cf yml file (cloud foundry version is v238)

• github doesn't support .yml, so i changed .txt

I set setting that is same success of postgresql docker.

I wait your reply :D

Build a pipeline for building, vetting and testing this BOSH release.

When using the shield ui I see the following error in /var/vcap/sys/log/shield/shieldd.log:

2018-01-04 11:41:58.643334246 +000 UTC /var/vcap/packages/shield/bin/shieldd: ERROR: GET /v2/tenants/0ba8c0b6-3976-44eb-b8f5-10a36a82c04e/health errored: no such column: j.fixed_key

Each click in the ui generates the above log line. I'm using v8.3.0. And have just one agent and an empty database.

Hey

Apparently it is currently not possible to run postgres in another vm/instance then nginx+shield-daemon.

Having a bosh2.0 style manifest which puts everything onto one VM:

---
name: shield
director_uuid: <uuid>

releases:
- name: shield
  version: latest

stemcells:
- alias: trusty
  name: bosh-vsphere-esxi-ubuntu-trusty-go_agent
  version: latest


instance_groups:
- azs:
  - az1
  name: shield
  instances: 1
  vm_type: 2cpu_4gbram_20gbdisk
  stemcell: trusty

  networks:
  - name: static
    static_ips: 10.35.96.66
  jobs:
  - name: nginx
    release: shield
  - name: shield-daemon
    release: shield
  - name: shield-agent
    release: shield
  - name: postgres
    release: shield
  properties:
    shield:
      daemon:
        domain: 10.35.96.66
        ssh_private_key: |+
          <private-key-stuff>
        database:
          type: postgres
          host: 10.35.96.66
          port: 5524
          username: shieldadmin
          password: admin
          db: shielddb
      agent:
        autoprovision: "https://10.35.96.66"
    databases:
      address: 10.35.96.66
      databases:
      - citext: true
        name: shielddb
        tag: shield
      - citext: true
        name: sessionsdb
        tag: sessions
      db_scheme: postgres
      port: 5524
      roles:
      - name: shieldadmin
        password: admin
        tag: admin

update:
  canaries: 1
  max_in_flight: 50
  canary_watch_time: 5000-120000
  update_watch_time: 5000-120000

This is working fine.

When trying to give postgres a dedicated vm:

---
name: shield-multi-vms
director_uuid: <uuid>

releases:
- name: shield
  version: latest

stemcells:
- alias: trusty
  name: bosh-vsphere-esxi-ubuntu-trusty-go_agent
  version: latest


instance_groups:
- azs:
  - az1
  name: shield
  instances: 1
  vm_type: 2cpu_4gbram_20gbdisk
  stemcell: trusty

  networks:
  - name: static
    static_ips: 10.35.96.67
  jobs:
  - name: nginx
    release: shield
  - name: shield-daemon
    release: shield
  - name: shield-agent
    release: shield
  properties:
    shield:
      daemon:
        domain: 10.35.96.67
        ssh_private_key: |+
          <private-key>
        database:
          type: postgres
          host: 10.35.96.68
          port: 5524
          username: shieldadmin
          password: admin
          db: shielddb
      agent:
        autoprovision: "https://10.35.96.67"
- azs:
  - az1
  name: postgres
  instances: 1
  vm_type: 2cpu_4gbram_20gbdisk
  stemcell: trusty

  networks:
  - name: static
    static_ips: 10.35.96.68
  jobs:
  - name: postgres
    release: shield
  properties:
    databases:
      address: 10.35.96.68
      databases:
      - citext: true
        name: shielddb
        tag: shield
      - citext: true
        name: sessionsdb
        tag: sessions
      db_scheme: postgres
      port: 5524
      roles:
      - name: shieldadmin
        password: admin
        tag: admin

update:
  canaries: 1
  max_in_flight: 50
  canary_watch_time: 5000-120000
  update_watch_time: 5000-120000

fails with:

Director task 481
  Started preparing deployment > Preparing deployment. Done (00:00:00)

  Started preparing package compilation > Finding packages to compile. Done (00:00:00)

  Started creating missing vms
  Started creating missing vms > shield/e636d5bc-4e27-4800-8d1c-4a9286207f5a (0)
  Started creating missing vms > postgres/d241e01b-1483-429e-8da8-1c9137a66663 (0)
     Done creating missing vms > shield/e636d5bc-4e27-4800-8d1c-4a9286207f5a (0) (00:00:52)
     Done creating missing vms > postgres/d241e01b-1483-429e-8da8-1c9137a66663 (0) (00:00:54)
     Done creating missing vms (00:00:54)

  Started updating instance shield > shield/e636d5bc-4e27-4800-8d1c-4a9286207f5a (0) (canary). Failed: 'shield/0 (e636d5bc-4e27-4800-8d1c-4a9286207f5a)' is not running after update. Review logs for failed jobs: shield-daemon (00:02:23)

Error 400007: 'shield/0 (e636d5bc-4e27-4800-8d1c-4a9286207f5a)' is not running after update. Review logs for failed jobs: shield-daemon

Task 481 error

The shield-daemon fails because it cannot connect to postgres, because there is not even a postgres running:

/d241e01b-1483-429e-8da8-1c9137a66663:/var/vcap/sys/log$ ps aux |grep postgres
/d241e01b-1483-429e-8da8-1c9137a66663:/var/vcap/sys/log$ netstat -tulpen
(No info could be read for "-p": geteuid()=1001 but you should be root.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       User       Inode       PID/Program name
tcp        0      0 0.0.0.0:32785           0.0.0.0:*               LISTEN      103        10455       -               
tcp        0      0 127.0.0.1:33331         0.0.0.0:*               LISTEN      0          12301       -               
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      0          11300       -               
tcp        0      0 127.0.0.1:2822          0.0.0.0:*               LISTEN      0          12300       -               
tcp        0      0 127.0.0.1:2825          0.0.0.0:*               LISTEN      0          11465       -               
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      0          10405       -               
tcp6       0      0 :::22                   :::*                    LISTEN      0          11302       -               
tcp6       0      0 :::47455                :::*                    LISTEN      103        10459       -               
tcp6       0      0 :::111                  :::*                    LISTEN      0          10408       -               
udp        0      0 0.0.0.0:37550           0.0.0.0:*                           103        10452       -               
udp        0      0 0.0.0.0:735             0.0.0.0:*                           0          10404       -               
udp        0      0 127.0.0.1:756           0.0.0.0:*                           0          10436       -               
udp        0      0 0.0.0.0:111             0.0.0.0:*                           0          10399       -               
udp6       0      0 :::735                  :::*                                0          10407       -               
udp6       0      0 :::111                  :::*                                0          10406       -               
udp6       0      0 :::53597                :::*                                103        10457       -

Can we somehow make it possible to separate the postgres instance from shield-daemon?

cheers

UAA based oauth requires a provider_url parameter in order to work. We should add a property for this so people can use UAA based oauth.

Hello,

I am using the shield-boshrelease to backup the Cloud Foundry blobstore but i got an error "timeout".

The size of the blobstore that i want to backup is 12 GB.

Running backup task (using bzip2 compression)
=============================================
Get http://mystorage.com/mybucket/?location=: dial tcp X.X.X.X:80: i/o timeout
bsdtar: Write error
TASK FAILED!! shield worker 2 failed to execute the command against the remote agent XX.XX.XX.XX:5444 (Process exited with: 4. Reason was: ())
Unable to exec '/var/vcap/packages/bsdtar/bin/bsdtar': exit status 1
EXITING 4
TASK FAILED!! No restore key detected in worker 2. Cowardly refusing to create an archive record

I am using the "fs" plugin as target and "scality" as store.

I tested the same configurations with an other folder that contains less data and it worked well.

Is it about file size ?

Best regards,

Getting following error while rendering templates:

Deploying:
  Building state for instance 'bosh/0':
    Rendering job templates for instance 'bosh/0':
      Rendering templates for job 'shield-agent/8ad3e3f29dcd1b8d7a8810f45fd4c929b7928d27':
        Rendering template src: config/agent.conf, dst: config/agent.conf:
          Rendering template src: /home/vcap/.bosh/installations/df1091ff-fe5b-4955-7e1a-cb30c4890b7a/tmp/bosh-release-job341768806/templates/config/agent.conf, dst: /home/vcap/.bosh/installations/df1091ff-fe5b-4955-7e1a-cb30c4890b7a/tmp/
rendered-jobs252948898/config/agent.conf:
            Running ruby to render templates:
              Running command: 'ruby /home/vcap/.bosh/installations/df1091ff-fe5b-4955-7e1a-cb30c4890b7a/tmp/erb-renderer047888025/erb-render.rb /home/vcap/.bosh/installations/df1091ff-fe5b-4955-7e1a-cb30c4890b7a/tmp/erb-renderer047888025
/erb-context.json /home/vcap/.bosh/installations/df1091ff-fe5b-4955-7e1a-cb30c4890b7a/tmp/bosh-release-job341768806/templates/config/agent.conf /home/vcap/.bosh/installations/df1091ff-fe5b-4955-7e1a-cb30c4890b7a/tmp/rendered-jobs252948898
/config/agent.conf', stdout: '', stderr: '/home/vcap/.bosh/installations/df1091ff-fe5b-4955-7e1a-cb30c4890b7a/tmp/erb-renderer047888025/erb-render.rb:189:in `rescue in render': Error filling in template '/home/vcap/.bosh/installations/df1
091ff-fe5b-4955-7e1a-cb30c4890b7a/tmp/bosh-release-job341768806/templates/config/agent.conf' for shield-agent/0 (line 3: #<TypeError: no implicit conversion of nil into String>) (RuntimeError)
        from /home/vcap/.bosh/installations/df1091ff-fe5b-4955-7e1a-cb30c4890b7a/tmp/erb-renderer047888025/erb-render.rb:175:in `render'
        from /home/vcap/.bosh/installations/df1091ff-fe5b-4955-7e1a-cb30c4890b7a/tmp/erb-renderer047888025/erb-render.rb:200:in `<main>'
':
                exit status 1

Exit code 1

Which trace back to .gsub('(name)', spec.name), apparently spec.name is nill because bosh create-env does not set it here. It does set spec.job.name.

In our case we are specifying an agent name without any placeholders so defaulting spec.name to '' as is done for spec.az would be sufficient.

Would a PR for this change be merged?

This "convenience" causes more problems than it solves.

Instead, we should provide a static key for people wanting to experiment with BOSH-lite, and require a value for the public/private RSA keypair (in SSH format) on real deployments.

Then we can get rid of the packages altogether.

When using a floating IP in Openstack, HTTP does not redirect to HTTPS it hangs, attempts to redirect to the internal IP, and then fails.

e.g. http://172.x.x.114 -> http://10.4.1.32 not https://172.x.x.114

On the shield/0 VM, the postgres job produces 27+ GB of logs each day. This fills my production disk very quickly.

For example, I recreated the shield/0 VM yesterday and here is what I have:

# du -sh /var/vcap/data/sys/log/postgres/postgresql.log
27G	/var/vcap/data/sys/log/postgres/postgresql.log

I didn't find a way to control the verbosity of these logs in the postgres-9.4 job spec. Is there any other way?
Would it be possible for the BOSH release to implement some rolling policies on these log files? Or is it up to the deployer to implement some BOSH add-on that does the trick?

(This issue was originally part of issue #48.)

We deployed a bosh with bosh-init and included the shield agent with target and job sections configured. When registering with the shield daemon the 'Remote IP:Port' was configured as :5444. It looks like this line where the ip is being determined isn't interpreted with bosh-init the same way as with regular bosh.

Hello,

I found some bugs under the import errand .

1. monit file:

The current monit file content is wrong , it will try to run the wrong job (store) :

check process store
with pidfile /var/vcap/sys/run/shield/store.pid
start program "/var/vcap/jobs/store/bin/nginx start"
stop program "/var/vcap/jobs/store/bin/nginx stop"
group vcap

==>The monit file should be empty for errand jobs, otherwise bosh will try to restart the job after executing the run.sh script.

2. run.sh script:

The current run.sh script contain an error + need more things to able to establish connection with the shield core

a) Line 11 :
the script is using a wrong name to call the shield cli
/var/vcap/packages/shield/bin/buckler
we have to use
/var/vcap/packages/shield/bin/shield

b) Environment Variables:
the run.sh script use the shield cli to import backups to the shield core , we need to provide some environment variables:

SHIELD_CORE_TOKEN
SHIELD_CORE
SHIELD_API

C)Login to the shield core

After setting the environment variables, we need to login to the shield core .

We may need to use :
/var/vcap/packages/shield/bin/shield api $SHIELD_API shield -k
/var/vcap/packages/shield/bin/shield login

Best regards,

Hi,
in the shield-daemon configuration there are two variables that can be configured

• number of workers (default to 5) link
• general Timeout for a task (default to 24hours) link

The bosh release does not support them, is that on purpose?
Would it be okay for a pull request to add them?

Thanks

Once its cut, I'll look at upgrading https://github.com/genesis-community/shield-genesis-kit

Hello,

The #58 PR aim to ensure that the postgres plugin finds the bin_dir by default.

Let's take a look to the following code :
https://github.com/starkandwayne/shield-boshrelease/blob/master/jobs/postgres/templates/bin/postgres_start.sh

PACKAGE_DIR=/var/vcap/packages/postgres-9.4
PACKAGE_DIR_OLD=/var/vcap/packages/postgres

Line 26 ==>

# Ensure that default postgres bin_dir of the postgres plugin is able to locate the bin_dir
if [ ! -d $PACKAGE_DIR ]; then
  ln -s $PACKAGE_DIR $PACKAGE_DIR_OLD
fi

I guess that the test must be :

if [ ! -d $PACKAGE_DIR_OLD]; then
  ln -s $PACKAGE_DIR $PACKAGE_DIR_OLD
fi

Best regards