Coder Social home page Coder Social logo

epost-dev / opennebula-puppet-module Goto Github PK

View Code? Open in Web Editor NEW
29.0 10.0 40.0 903 KB

The one module allows to install and manage your OpenNebula cloud

License: Apache License 2.0

Ruby 47.82% Python 0.12% Shell 0.97% Puppet 20.20% HTML 30.66% Dockerfile 0.24%

opennebula-puppet-module's Introduction

opennebula-puppet-module

The one (short for OpenNebula) module allows to install and manage your OpenNebula cloud.

Build Status

Table of Contents

  1. Requirements
  2. Using the Module
  3. Usage of opennebula puppet resource types
  4. Support
  5. License

Requirements

Supported Platforms

Centos

We support Puppet 3.1.1 on CentOS 6.7 with OpenNebula 4.12.1. You need to add the EPEL repository because the module needs some packages from there.

Note, there is a required package on the oned controller node that is not managed by this module, the gem nokogiri (or the yum package rubygem-nokori). This can be installed elsewhere in that node's puppet catalog like this:

package {'rubygem-nokogiri':
  ensure => installed,
} ->
class {'::one':}

Puppet Module Dependencies

The ONe-Module needs the following other modules:

How to install:

puppet module install puppetlabs-stdlib
puppet module install puppetlabs-apt
puppet module install puppetlabs-inifile

Using the Module

Example usage for opennebula puppet module

  1. Running as OpenNebula Master with Apache and mod_passenger and Sunstone using kvm and 802.1q VLAN networking:

     class { one:
         oned               => true,
         sunstone           => true,
         sunstone_passenger => true,
     }
    

Attn: needs separate apache config for sunstone.

  1. running opennebula node

     class { one: }
    

Usage of opennebula puppet resource types

Create a ONE Vnet

onevnet { '<name>':
    ensure          => present | absent,
    # name of the bridge to use
    bridge          => 'basebr0',
    #  name of the physical interface on which the bridge wiull run
    phydev          => 'br0',
    dnsservers      => ['8.8.8.8', '4.4.4.4'],
    gateway         => '10.0.2.1',
    # add vlanid 
    vlanid          => '1550',
    netmask         => '255.255.0.0',
    network_address => '10.0.2.0',
}

Create onevnet addressrange

onevnet_addressrange { '<name>':
    ensure        => present | absent,
    onevnet_name  => '<name>',            # this has to be an existing onevnet - will be autorequired if declared
    ar_id         => '<INT>',             # read only value
    protocol      => ip4 | ip6 | ip4_6 | ether,
    ip_size       => '10',
    mac           => '02:00:0a:00:00:96', # optional
    # attributes for ip4 and ip4_6:
    ip_start      => '10.0.2.20'
    # attributes for ip6:
    globalprefix  => '2001:a::',          # optional
    ulaprefix     => 'fd01:a:b::',        # optional
}

Attention: onevnet_addressrange uses the title to uniqly identify address ranges among all Virtual Networks. The title will be set as common attribute with the name PUPPET_NAME. This means: address ranges which are not set by Puppet will not be visible using puppet resource onevnet_addressrange command.

Create a ONE Datastore

onedatastore { '<name>':
    ensure      => present | absent,
    type        => 'IMAGE_DS' | 'SYSTEM_DS' | 'FILE_DS',
    ds_mad      => 'fs' | 'vmware' | 'iscsi' | 'lvm' | 'vmfs' | 'ceph',
    tm_mad      => 'shared' | 'ssh' | 'qcow2' | 'iscsi' | 'lvm' | 'vmfs' | 'ceph' | 'dummy',
    driver      => 'raw | qcow2',
    ceph_host   => 'cephhost', # (optional: ceph only)
    ceph_user   => 'cephuser', # (optional: ceph only)
    ceph_secret => 'ceph-secret-here', # (optional: ceph only)
    pool_name   => 'cephpoolname', # (optional: ceph only)
    bridge_list => 'host1 host2 host3', # (optional: ceph only)
    disk_type   => 'file' | 'block' | 'rdb',
    base_path   => '/some/lib/path/datastore', #Optional
    cluster     => 'somename', # Optional
    cluster_id  => '1234', # Optional
}

Create a ONE Host

onehost { '<name>':
    ensure  => present | absent,
    im_mad  => 'kvm' | 'xen' | 'vmware' | 'ec2' | 'ganglia' | 'dummy' | 'custom',
    vm_mad  => 'kvm' | 'xen' | 'vmware' | 'ec2' | 'dummy' | 'custom' | 'qemu',
    vn_mad  => 'dummy' | 'firewall' | 'vlan' | 'ebtables' | 'ovswitch' | 'vmware' | 'custom',
}

Create a ONE Cluster

onecluster { '<name>':
    ensure     => present | absent,
    hosts      => [ 'host1', 'host2',...],
    vnets      => [ 'vnet1', 'vnet2', ...],
    datastores => [ 'ds1', 'ds2', ...],
}

Create a ONE Image

oneimage { '<name>':
    ensure      => present | absent,
    datastore   => 'default',
    description => 'Image description',
    disk_type   => 'os' | 'cdrom' | 'datablock' | 'kernel' | 'ramdisk' | 'context',
    persistent  => 'true' | 'false',
    dev_prefix  => 'hd' | 'sd' | 'xvd' | 'vd',
    target      => 'hda' | 'hdb' | 'sda' | 'sdb',
    path        => '/tmp/image_file',
    driver      => 'raw' | 'qcow2' | 'tap:aio' | 'file:',
    # non file based images
    source      => '',
    size        => '11200' # 11.2 GB
    fstype      => 'ext3',
}

Create a ONE Template

onetemplate { '<name>':
    ensure                    => present | absent,
    memory                    => '1024',
    cpu                       => '0.2',
    vcpu                      => '4',
    features                  => { 'acpi' => 'yes|no', 'pae' => 'true|false' },
    os                        => { 'kernel' => '/boot/vmkernel', 'initrd' => '/boot/vminitrd', 'arch' => 'x86_64', 'root' => 'hda1', 'bootloader' => '/sbin/lilo', 'boot' => 'hd|fd|cdrom|network' }
    pci_bridge                => '4',
    disks                     => [ 'disk1', 'disk2', ...],
    nics                      => [ 'nic1', 'vnet2', .. ],
    nic_model                 => 'virtio',
    graphics                  => { 'type' => 'vnc|sdl', 'listen' => '0.0.0.0', 'password' => 'myvncpass', 'keymap' => 'de' },
    context                   => { 'VAR1'  => 'value1', 'var2' => 'value2', ...},
    context_ssh_pubkey        => '$USER[SSH_PUBLIC_KEY]',
    context_network           => 'yes' | 'no',
    context_onegate           => 'yes' | 'no',
    context_files             => [ 'init.sh', 'mycontextaddon.sh'],
    context_variable          => # unused,
    context_placemant_host    => # unused,
    context_placemet_cluster  => # unused,
    context_policy            => # unused,
}

Create a ONE VM

onevm { '<name>':
    ensure   => present | absent,
    template => 'template_name',
}

Create a ONE Security Groups (ONe <= 4.12):

onesecgroup {'securitygroup1':
   description => 'Optional description',
   rules       => [ { protocol      => 'TCP|UDP|ICMP|IPSEC|ALL',
                      rule_type     => 'INBOUND|OUTBOUND',
                      ip            => '192.168.0.0',
                      size          => '255',
                      range         => '22,53,80:90,110,1024:65535',
                      icmp_type     => 'optional, only applies for icmp',
                    },
                    { protocol  => 'ALL',
                      rule_type => 'OUTBOUND',
                    },
                    ...
                  ]
}

Support

For questions or bugs create an issue on Github.

How to contribute: CONTRIBUTING.md

License

Copyright © 2013 - 2016 Deutsche Post E-Post Development GmbH

Distributed under the Apache License, Version 2.0.

opennebula-puppet-module's People

Contributors

241comp avatar agx avatar arnetogo avatar brennwert avatar choonming avatar chrisn777 avatar csage avatar escalate avatar flexible avatar gdhgdhgdh avatar jehrhardt avatar jtriley avatar mattthias avatar mcanevet avatar michael-os avatar mmoll avatar nmesstorff avatar rwaffen avatar sebastiansucker avatar tuxmea avatar xylakant avatar yanns avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

opennebula-puppet-module's Issues

Backup script produces errors on nfs mounts

When automated like this (names and addresses changed for data privacy's sake):

[admin@server1 ~]$ crontab -l
...
# Puppet Name: one_db_backup
0 * * * * /var/lib/one/bin/one_db_backup.sh

and writing to an nfs share like this:

[root@server1 ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/rootdg-LogVol00
                      3.9G  1.3G  2.4G  36% /
tmpfs                 1.9G     0  1.9G   0% /dev/shm
/dev/mapper/rootdg-LogVol08
                      976M  1.3M  924M   1% /admin
/dev/mapper/rootdg-LogVol07
                      120M  1.6M  113M   2% /audit
/dev/vda1             283M  148M  120M  56% /boot
/dev/mapper/rootdg-LogVol04
                      969M  1.7M  917M   1% /home
/dev/mapper/rootdg-LogVol03
                      1.9G   25M  1.8G   2% /opt
/dev/mapper/rootdg-LogVol05
                      2.0G   30M  1.8G   2% /tmp
/dev/mapper/rootdg-LogVol02
                      2.0G  500M  1.4G  27% /var
/dev/mapper/rootdg-LogVol06
                      2.0G  3.0M  1.9G   1% /var/cores
/dev/mapper/rootdg-LogVol01
                      2.0G  285M  1.6G  16% /var/log
192.168.20.21:/server1_inf_backup
                      100G  474M  100G   1% /var/lib/one/nfs_backup
192.168.20.21:/server1_inf_datastore_0
                      200G   94G  107G  47% /var/lib/one/datastores/0
192.168.20.21:/server1_inf_datastore_1
                      500G  415G   86G  83% /var/lib/one/datastores/1
192.168.20.21:/server1_inf_datastore_2
                     1014M  4.3M 1010M   1% /var/lib/one/datastores/2

the backup script produces errors like this:

From [email protected]  Fri Dec 22 12:00:03 2017
Return-Path: <[email protected]>
X-Original-To: admin
Delivered-To: [email protected]
Received: by server1.inf.ourdomain.de (Postfix, from userid 9869)
	id 03EC71625; Fri, 22 Dec 2017 12:00:03 +0100 (CET)
From: [email protected] (Cron Daemon)
To: [email protected]
Subject: Cron <admin@server11> /var/lib/one/bin/one_db_backup.sh
Content-Type: text/plain; charset=UTF-8
Auto-Submitted: auto-generated
X-Cron-Env: <LANG=en_US.UTF-8>
X-Cron-Env: <LC_ALL=en_US.utf-8>
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <HOME=/var/lib/one>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=admin>
X-Cron-Env: <USER=admin>
Message-Id: <[email protected]>
Date: Fri, 22 Dec 2017 12:00:03 +0100 (CET)

find: `/var/lib/one/nfs_backup/.nfs': Permission denied

The reason for this is that the script does not limit its file selection to the backup directory but descends into any subdirectories below. The .nfs subdirectory belongs to root, however, and has restrictive permissions (as it should):

[root@serverd1 nfs_backup]# ls -la .nfs
total 36
drwxrwx--- 4 root     root        41 Dec 18 09:20 .
drwxr-xr-x 3 admin admin 28672 Jan  3 09:00 ..
drwx------ 2 root     root         6 Nov  1 10:50 rmt
-rw-r--r-- 1 root     root        97 Dec 18 09:24 rmtab
drwxrwx--- 4 root     root        40 Oct 26 20:00 .sm.ha
[root@server1 nfs_backup]#

A simple fix would be to restrict the find operation to the backup dir itself.

OpenNebula 4.14 support

We'd like to support OpenNebula 4.14, so are working on templating the config files. There are a number of differences between 4.14 and 4.12, so we propose the following:

  • what is currently the master branch becomes a 4.12 branch
  • the master then becomes 4.14
  • the master branch is 4.14 until 4.15 is released, and then we make a 4.14 branch

Let us know your thoughts regarding that. For now we've got a work in progress of updating the config template files on a fasrc/opennebula_4_14 branch (comparison view master...fasrc:opennebula_4_14)

Another option is to version the template directory - ie

└── templates
    ├── 4.12
    ├── 4.14

And then in paths to template files in puppet code:

file { '/etc/one/oned.conf':
  ensure  => file,
  owner   => 'root',
  mode    => '0640',
  content => template("one/${one_version}/oned.conf.erb"),
}

This approach has the benefit of a single branch to support all versions, but might make the code more complicated if we have to split anything out by version other than templates.

Thoughts?

Make hiera optional

ATM there are many explicit hiera calls.
I want to allow users to decide by themselves by making use of puppet 3 automatic data bindings.
This means a larger impact for existing users as we have to change the API (init.pp).
I will prepare a pull request for this one after I finished the types and providers.
But: this change definitely needs a major version bump.

new kvm driver config handling continuously appends and breaks syntax

The recent change to kvm driver config isn't working properly. When both the kvm emulator and nic settings are specified it doesn't idempotently make the updates and instead constantly appends to the file every time puppet runs. Even worse it's appending the wrong output (ie [] on a line by itself) which causes a syntax error:

[]
NIC = [ filter = "clean-traffic", model="virtio" ]

[]
EMULATOR = /usr/libexec/qemu-kvm

[]
NIC = [ filter = "clean-traffic", model="virtio" ]

[]
EMULATOR = /usr/libexec/qemu-kvm

[]
NIC = [ filter = "clean-traffic", model="virtio" ]

[]
EMULATOR = /usr/libexec/qemu-kvm

[]
NIC = [ filter = "clean-traffic", model="virtio" ]

[]
EMULATOR = /usr/libexec/qemu-kvm

Error in logs:

Thu Sep 10 11:23:43 2015 [Z0][VMM][I]:  Loading driver: kvm (KVM)
Thu Sep 10 11:23:43 2015 [Z0][VMM][E]: Error loading driver configuration file /etc/one/vmm_exec/vmm_exec_kvm.conf : syntax error, unexpected CBRACKET, expecting $end at line 60, columns 2797:2799

onedatastore resource type ignores basepath parameter; breaks idempotence

The onedatastore resource type ignores the basepath parameter when creating the datastore. The datastore will be created with a default path. When running puppet a second time puppet tries to change the basepath which fails as the provider doesn't support that action.

How to reproduce:

  • Install centos6 machine
  • install puppet 3.7.2
  • clone the opennebula-puppet-module and the puppetlabs-stdlib repo to a directory called "modules"
[root@centos6 ~]# mkdir modules
[root@centos6 ~]# git clone https://github.com/epost-dev/opennebula-puppet-module.git  modules/one
....
[root@centos6 ~]# git clone https://github.com/puppetlabs/puppetlabs-stdlib.git modules/stdlib
....
  • create a simple site.pp
node 'default' {

  class { one: 
    oned => true,
  }

  onedatastore { 'beispiel':
    ensure   => present,
    type     => 'IMAGE_DS',
    dm       => 'fs',
    tm       => 'ssh',
    basepath => '/srv/beispiel',
  }
}
  • run puppet apply for the first time (should work)
[root@centos6 ~]# puppet apply --modulepath=modules site.pp
...
  • check if the datastore exists; check the path in the xml output. As we can see there is no basepath "srv/beispiel".
[root@centos6 ~]# onedatastore list 
  ID NAME                SIZE AVAIL CLUSTER      IMAGES TYPE DS      TM      STAT
   0 system                0M -     -                 0 sys  -       shared  on  
   1 default            29.4G 89%   -                 0 img  fs      shared  on  
   2 files              29.4G 89%   -                 0 fil  fs      ssh     on  
 100 beispiel           29.4G 89%   -                 0 img  fs      ssh     on  
[root@centos6 ~]# onedatastore list  -x | grep BASE
    <BASE_PATH><![CDATA[/var/lib/one//datastores/0]]></BASE_PATH>
      <BASE_PATH><![CDATA[/var/lib/one//datastores/]]></BASE_PATH>
    <BASE_PATH><![CDATA[/var/lib/one//datastores/1]]></BASE_PATH>
      <BASE_PATH><![CDATA[/var/lib/one//datastores/]]></BASE_PATH>
    <BASE_PATH><![CDATA[/var/lib/one//datastores/2]]></BASE_PATH>
      <BASE_PATH><![CDATA[/var/lib/one//datastores/]]></BASE_PATH>
    <BASE_PATH><![CDATA[/var/lib/one//datastores/100]]></BASE_PATH>
      <BASE_PATH><![CDATA[/var/lib/one//datastores/]]></BASE_PATH>
[root@centos6 ~]# 
  • run puppet apply again (will surely break)
[root@centos6 ~]# puppet apply --modulepath=modules/ site.pp 
Notice: Compiled catalog for centos6 in environment production in 2.73 seconds
Error: Can not modify basepath. You need to delete and recreate the datastore
Error: /Stage[main]/Main/Node[default]/Onedatastore[beispiel]/basepath: change from /var/lib/one//datastores/ to /srv/beispiel failed: Can not modify basepath. You need to delete and recreate the datastore
Notice: Finished catalog run in 5.95 seconds
[root@centos6 ~]# 

ONE provider onetemplate postwork required

onetemplate can not yet modify:

  • memory
  • cpu
  • vcpu
  • disks
  • nics
  • graphics_listen
  • graphics_password
  • graphics_keymap
  • context hash
  • context_ssh
  • context_ssh_pubkey
  • context_network

onetemplate provider can not yet manage

  • host placement
  • cluster placement
  • placement policy

make sudo handling optional

Handling the sudo rules in this module may conflict with whatever people using this module use to handle sudo on their end. We should make sudo rule handling optional, though the default should be to keep current behavior.

This sort of references #205

Strange parsing for oneimage

Given Puppet 3.7.4, CentOS 7 and hiera config:

oneimages:
  test-image:
    datastore: cephstore
    type: datablock
    persistent: true
    dev_prefix: vd
    driver: raw
    disk_type: RBD
    size: 4096
    fstype: raw

and this fragment in a Puppet manifest:

  create_resources(oneimage, hiera_hash(oneimages))

I get the following segment of output:

Debug: Prefetching cli resources for oneimage
Debug: Executing '/usr/bin/oneimage list -x'
Error: Could not set 'present' on ensure: undefined method `to_s_upcase' for :DATABLOCK:Symbol
Error: Could not set 'present' on ensure: undefined method `to_s_upcase' for :DATABLOCK:Symbol
Wrapped exception:
undefined method `to_s_upcase' for :DATABLOCK:Symbol
Error: /Stage[main]/Roles::Onevm/Oneimage[test-image]/ensure: change from absent to present failed: Could not set 'present' on ensure: undefined method `to_s_upcase' for :DATABLOCK:Symbol

All of my current disks are DATABLOCK <TYPE>2</TYPE>, and I did try to add a mapping 2 => DATABLOCK at line 100 of lib/puppet/provider/oneimage/cli.rb but it made no difference - I don't understand enough Ruby to see the logic of what's being attempted :/

Cheers,
Gavin.

Module QuoVadis?

Hey Folks,

I have been using this module since a long time and I would like to use it in the future. The main problem for me is that this module is nailed to puppet 3.1 and also it is not up to date to the Puppetlabs Styleguide anymore.

I thought about the following steps to modify this module:

  • Update it to the current Styleguide for Puppet 3 and ensure ist is running with Puppet 3.latest version
  • Update supported OS Plattforms
  • create a new major release
  • rewrite the code to make it compliant with features introduced by Puppet 4

Please let me know what you think and what you guys at epost have in mind about this module in the future

Use of the HTTP Protocol

Greetings,

I am a security researcher, who is looking for security smells in Puppet scripts. I found instances where the HTTP protocol is used instead of HTTPS (HTTP with TLS). I think this is a smell, and I was wondering why HTTP is used? Is it because of lack of tool support? I am trying to find out if developers are forced to adopt bad practices due to lack of tool support when it comes to the HTTPS protocol.

Any feedback is appreciated.

Source: https://github.com/epost-dev/opennebula-puppet-module/blob/master/manifests/params.pp (line#142)

Error when trying to create an ETHER network

Hi, using one 4.10 on CentOS 7 (a simple PR coming your way to support yum repos / changed packages), I'm trying to define a simple network:

  onevnet { 'tenlocal':
    ensure       => present,
    bridge       => 'br0',
    addressrange => {
      'macstart'  => '02:00:16:90:01:01',
      'size' => '200',
      'type' => 'ETHER',
    }
  }

However I'm getting this error from Puppet 3.7.4:

Error: Invalid parameter network_start(:network_start)
Error: /Stage[main]/Roles::Default/Onevnet[tenlocal]/ensure: change from absent to present failed: Invalid parameter network_start(:network_start)

I don't understand this error since I didn't define network_start - I have an external DHCP server and am using the dummy virtual net driver to just bridge VMs with the rest of my network. This is working well on a manually-configured installation, and I would like to Puppetise the setup.

Cheers,
Gavin.

Documentation does not mention puppetlabs-stdlib

The documentation should mention that puppetlabs-stdlib is needed. If the module is not present puppet fails with

  Error: Unknown function validate_string at /root/modules/one/manifests/params.pp:163 on node centos6

prep release for 1.x branch

before moving forward with BC-incompatible changes we should create a last release of the 1.x version and then bump the major version. There have been several relevant changes since the 1.3.0 release, among the, support for opennebula 4.14. see v1.3.0...1dcefbf

ONE Types and Providers and autorequire

ONE Types do not autorequire other one types.
We need discussion whether we want autorequire.
e.g.
Cluster needs Datastore, VNet and Hosts.
Images need their datastore
Templates need images
VM needs templates.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.