Coder Social home page Coder Social logo

sapcc / cyclone Goto Github PK

View Code? Open in Web Editor NEW
15.0 37.0 6.0 6.58 MB

Clone OpenStack entities easily

License: Apache License 2.0

Makefile 0.51% Go 99.49%
openstack openstack-swift openstack-nova openstack-cinder openstack-instance clone cloud vm virtual-machine availability-zone

cyclone's Introduction

cyclone

Clone OpenStack entities easily.

Why?

In modern clusters compute instances are considered as a Cattle, but there are exceptions, when a compute instance is a Pet and needs care, especially when you need to migrate or clone it to a new OpenStack region or availability zone.

Here comes cyclone (Cloud Clone or cclone) to help you with this task. It takes care about all volumes attached to a VM and clones them with all required intermediate type conversions.

Help

By default Glance image data will be streamed through cyclone and the traffic will be consumed on the execution side. To enable the Glance V2 web-download method, set the --image-web-download flag. This method allows Glance to download an image using a remote URL. It is not recommended to use web-download method for images bigger than 1-10GiB, since Glance service will try to download the image to its intermediate local storage and may cause insufficient disk space error.

A remote URL can be generated using a Swift Temporary URL.

A volume migration is performed by converting a volume to an image and further image migration between regions, then converting an image back to a volume.

A volume migration within the same region is performed using a Volume Transfer method.

By default the tool uses the same credentials from environment variables for the source and destination projects, but for the destination you can define different region, domain and project name. It is also possible to override destination credentials via OpenStack environment variables with the TO_ prefix or via CLI parameters.

~> Note: Be aware about the quota, especially the source project quota, when cloning a volume. It requires up to 2x source volume size Cinder (Block Storage) quota. If a --clone-via-snapshot flag is specified, the quota requirement increases up to 3x source volume size.

~> Note: Cloning a volume within the same region, but different availability zones requires an extra Swift storage quota. If you don't have an ability to use Swift in this case, you can specify a --clone-via-snapshot flag.

~> Note: It is strongly recommended to shut down the VM before you start a migration of a VM or its volumes.

~> Note: By default cyclone writes all OpenStack request/response logs into a cyclone directory, located in System Temporary Directory. Define -d or --debug flag if you want to see these logs in console output.

Clone OpenStack entities easily

Usage:
  cyclone [command]

Available Commands:
  backup
  completion     Generate the autocompletion script for the specified shell
  help           Help about any command
  image          Clone an image
  secret         Clone a secret
  security-group Clone a security group
  server         Clone a server
  share          Clone a share
  version        Print version information
  volume         Clone a volume

Flags:
  -d, --debug                                     print out request and response objects
  -h, --help                                      help for cyclone
      --image-web-download                        use Glance web-download image import method
  -n, --no                                        assume "no" to all questions
      --timeout-backup string                     timeout to wait for a backup status (default "24h")
      --timeout-image string                      timeout to wait for an image status (default "24h")
      --timeout-secret string                     timeout to wait for a secret status (default "24h")
      --timeout-security-group string             timeout to wait for a security group status (default "24h")
      --timeout-server string                     timeout to wait for a server status (default "24h")
      --timeout-share string                      timeout to wait for a share status (default "24h")
      --timeout-share-replica string              timeout to wait for a share replica status (default "24h")
      --timeout-share-snapshot string             timeout to wait for a share snapshot status (default "24h")
      --timeout-snapshot string                   timeout to wait for a snapshot status (default "24h")
      --timeout-volume string                     timeout to wait for a volume status (default "24h")
      --to-application-credential-id string       destination application credential ID
      --to-application-credential-name string     destination application credential name
      --to-application-credential-secret string   destination application credential secret
      --to-auth-url string                        destination auth URL (if not provided, detected automatically from the source auth URL and destination region)
      --to-domain string                          destination domain
      --to-password string                        destination username password
      --to-project string                         destination project
      --to-region string                          destination region
      --to-username string                        destination username
  -y, --yes                                       assume "yes" to all questions

Use "cyclone [command] --help" for more information about a command.

Examples

Clone an image between regions

$ source openrc-of-the-source-project
$ cyclone image 77c125f1-2c7b-473e-a56b-28a9a0bc4787 --to-region eu-de-2 --to-project destination-project-name --to-image-name image-from-source-project-name

~> Note: Please ensure that your OpenStack user has sufficient permissions (e.g. image_admin and swiftoperator user roles) before initiating the above command.

Clone an image between regions using download/upload method

$ source openrc-of-the-source-project
$ cyclone image 77c125f1-2c7b-473e-a56b-28a9a0bc4787 --to-region eu-de-2 --to-project destination-project-name --to-image-name image-from-source-project-name

Clone a bootable volume between regions

$ source openrc-of-the-source-project
$ cyclone volume c4c18329-b124-4a23-8546-cf1ca502ef95 --to-region eu-de-2 --to-project destination-project-name --to-volume-name volume-from-source-project-name

Clone a volume within the same project, but different availability zones

$ source openrc-of-the-source-project
$ cyclone volume 97d682ae-840f-461f-b956-98af30533a22 --to-az eu-de-2a

Clone a server with all attached volumes to a specific availability zone

$ source openrc-of-the-source-project
$ cyclone server 6eb76733-95b7-4867-9f83-a6ab19804e2f --to-az eu-de-2a --to-key-name my-nova-keypair

Clone a server with a local storage to a server with bootable Cinder storage

--bootable-volume 16 will create a 16 GiB bootable volume from the source VM snapshot and create a new VM using this volume.

$ source openrc-of-the-source-project
$ cyclone server 6eb76733-95b7-4867-9f83-a6ab19804e2f --bootable-volume 16 --to-key-name my-nova-keypair

Clone a server with a local disk or a bootable volume only

--bootable-disk-only flag allows to clone a VM with only a local disk or a bootable volume, ignoring all secondary attached volumes.

$ source openrc-of-the-source-project
$ cyclone server 6eb76733-95b7-4867-9f83-a6ab19804e2f --bootable-disk-only --to-key-name my-nova-keypair

Clone a server with a bootable volume to a server with a local disk

--local-disk allows to clone a VM with a Cinder bootable volume to a VM with a local disk.

$ source openrc-of-the-source-project
$ cyclone server 6eb76733-95b7-4867-9f83-a6ab19804e2f --local-disk --to-key-name my-nova-keypair

Clone only server artifacts

The --skip-server-creation flag clones only images or volumes, which are used or attached to the source server. The destination server won't be created. The example below will convert the server's local bootable disk to a bootable block storage, which can be attached to some server lately.

$ source openrc-of-the-source-project
$ cyclone server 6eb76733-95b7-4867-9f83-a6ab19804e2f --bootable-volume 64 --skip-server-creation

Upload a local image file into a backup

Properties must be defined, when a backup supposed to be restored to a bootable volume.

$ source openrc-of-the-source-project
$ cyclone backup upload my-file.vmdk --to-container-name swift-backup-container --volume-size=160 --threads=16 \
  -p hw_vif_model=VirtualVmxnet3 \
  -p vmware_ostype=sles12_64Guest \
  -p hypervisor_type=vmware \
  -p min_ram=1008 \
  -p vmware_disktype=streamOptimized \
  -p disk_format=vmdk \
  -p hw_video_ram=16 \
  -p vmware_adaptertype=paraVirtual \
  -p container_format=bare \
  -p min_disk=10 \
  -p architecture=x86_64 \
  -p hw_disk_bus=scsi

Upload a remote Glance image into a backup

$ source openrc-of-the-source-project
$ cyclone backup upload my-glance-image --to-container-name swift-backup-container --volume-size=160 --threads=16

Transfer a big volume from one region to another

~> Note: The cyclone backup upload command produces high traffic and CPU/RAM usage. It's recommended to run it inside a VM, located in the target region.

In this case you need to convert a volume to an image first:

$ source openrc-of-the-source-project
$ cyclone volume to-image my-cinder-volume --to-image-name my-glance-image

then transfer it within multiple parallel connections to a target backup resource with a further volume restore action.

$ source openrc-of-the-source-project
$ cyclone backup upload my-glance-image --to-container-name swift-backup-container --to-region my-region-1 \
  --volume-size=160 --threads=16 --restore-volume

It's strongly recommended to run the cyclone backup upload command inside a VM, located in the source or the target region.

Create a new volume from an existing backup

$ source openrc-of-the-source-project
$ cyclone backup restore my-backup

Clone an existing backup to another region

~> Note: The cyclone backup clone command produces high traffic. It's recommended to run it inside a VM, located in the source or the target region.

$ source openrc-of-the-source-project
$ cyclone backup clone my-backup --to-region my-region-1 --threads=16

Manila shares support

Manila share type must support replicas, i.e.

$ openstack share type show default -c optional_extra_specs -f json | jq '.optional_extra_specs.replication_type'
"dr"

Clone a Manila share to a new share in a new availability zone

$ source openrc-of-the-source-project
$ cyclone share my-share --to-share-name my-new-share --to-az my-region-1b

Move an existing Manila share to a new availability zone

$ source openrc-of-the-source-project
$ cyclone share move my-share --to-az my-region-1b

Build

$ make
# or within the docker container
$ make docker

cyclone's People

Contributors

dependabot[bot] avatar kayrus avatar notandy avatar rajivmucheli avatar renovate[bot] avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

cyclone's Issues

support env variable OS_PW_CMD

It would be nice if cyclone could also support the OS_PW_CMD to get the password instead of relying on OS_PASSWORD to be set

--to-az option missing in readme/help

Currently, the --to-az option is missing in cyclone help as well as in the README but mentioned in the examples. Is this intended?

Or is that a volume specific flag?

make failes

when running make I get this error:

gofmt -s -w cmd pkg
go vet -mod=vendor ./cmd/... ./pkg/...
build io/fs: cannot load io/fs: open /home/ccloud/cyclone/vendor/io/fs: no such file or directory
make: *** [Makefile:23: vet] Error 1

is there an issue in the Makefile or what am I doing wrong?

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

Detected dependencies

gomod
go.mod
  • go 1.22
  • go 1.22.4
  • github.com/google/uuid v1.6.0
  • github.com/gophercloud/gophercloud v1.12.0
  • github.com/gophercloud/utils v0.0.0-20231010081019-80377eca5d56@80377eca5d56
  • github.com/klauspost/compress v1.17.8
  • github.com/machinebox/progress v0.2.0
  • github.com/majewsky/schwift v1.3.0
  • github.com/sapcc/go-bits v0.0.0-20240530080859-f4579fb3a074@f4579fb3a074
  • github.com/sapcc/swift-http-import v0.0.0-20240605095849-0db239238e71@0db239238e71
  • github.com/spf13/cobra v1.8.0
  • github.com/spf13/viper v1.19.0
  • github.com/xhit/go-str2duration/v2 v2.1.0
  • golang.org/x/term v0.21.0

  • Check this box to trigger a request for Renovate to run again on this repository

Allow to use insecure connection

We have self-signed certs in internal cloud. Now its impossible to use cyclone. Would be nice if it has flag like openstack-cli "--insecure"

Allow use of IDs instead of names

It took me quite a while to understand that I couldn't use the image or project/domain ID instead the name. With ID I also got an authentication error even when my password was correct:

2024/02/22 11:50:26 [dst] OpenStack Request URL: POST https://identity-3.eu-de-1.cloud.sap/v3/auth/tokens
2024/02/22 11:50:26 [dst] OpenStack Request Headers:
2024/02/22 11:50:26 [dst] Accept: application/json
2024/02/22 11:50:26 [dst] Content-Type: application/json
2024/02/22 11:50:26 [dst] User-Agent: cyclone/0.3.0 gophercloud/v1.4.0
2024/02/22 11:50:26 [dst] OpenStack Request Body: {
2024/02/22 11:50:26 [dst]   "auth": {
2024/02/22 11:50:26 [dst]     "identity": {
2024/02/22 11:50:26 [dst]       "methods": [
2024/02/22 11:50:26 [dst]         "password"
2024/02/22 11:50:26 [dst]       ],
2024/02/22 11:50:26 [dst]       "password": {
2024/02/22 11:50:26 [dst]         "user": {
2024/02/22 11:50:26 [dst]           "domain": {
2024/02/22 11:50:26 [dst]             "name": "monsoon3"
2024/02/22 11:50:26 [dst]           },
2024/02/22 11:50:26 [dst]           "name": "IXXXXXX",
2024/02/22 11:50:26 [dst]           "password": "***"
2024/02/22 11:50:26 [dst]         }
2024/02/22 11:50:26 [dst]       }
2024/02/22 11:50:26 [dst]     },
2024/02/22 11:50:26 [dst]     "scope": {
2024/02/22 11:50:26 [dst]       "project": {
2024/02/22 11:50:26 [dst]         "domain": {
2024/02/22 11:50:26 [dst]           "name": "monsoon3"
2024/02/22 11:50:26 [dst]         },
2024/02/22 11:50:26 [dst]         "name": "bbd5e2c3a2944f128b3bba616eb18eb6"
2024/02/22 11:50:26 [dst]       }
2024/02/22 11:50:26 [dst]     }
2024/02/22 11:50:26 [dst]   }
2024/02/22 11:50:26 [dst] }
2024/02/22 11:50:26 [dst] OpenStack Response Code: 401
2024/02/22 11:50:26 [dst] OpenStack Response Headers:
2024/02/22 11:50:26 [dst] Content-Length: 109
2024/02/22 11:50:26 [dst] Content-Type: application/json
2024/02/22 11:50:26 [dst] Date: Thu, 22 Feb 2024 10:50:26 GMT
2024/02/22 11:50:26 [dst] Strict-Transport-Security: max-age=15724800; includeSubDomains
2024/02/22 11:50:26 [dst] Vary: X-Auth-Token
2024/02/22 11:50:26 [dst] Www-Authenticate: Keystone uri="https://identity-3.eu-de-1.cloud.sap/v3"
2024/02/22 11:50:26 [dst] X-Openstack-Request-Id: req-142f086a-fda3-4958-8ecd-5bc68803b9a6
2024/02/22 11:50:26 [dst] OpenStack Response Body: {
2024/02/22 11:50:26 [dst]   "error": {
2024/02/22 11:50:26 [dst]     "code": 401,
2024/02/22 11:50:26 [dst]     "message": "The request you have made requires authentication.",
2024/02/22 11:50:26 [dst]     "title": "Unauthorized"
2024/02/22 11:50:26 [dst]   }
2024/02/22 11:50:26 [dst] }
Error: failed to create a destination OpenStack client: Authentication failed
2024/02/22 11:50:26 Error: failed to create a destination OpenStack client: Authentication failed

I needed to check the keystone log to find:

2024-02-22 10:49:07.242457+00:00 - keystone-api - 2024-02-22 10:49:07.242457 16 WARNING keystone.auth.core [req-2fd194cd-6271-4853-9bbf-21105215c36f greq-368cf11c-90e1-b732-6326-1088faa13f16 usr - prj - dom - usr-dom - prj-dom -] Could not find domain: ec213443e8834473b579f7bea9e8c194.: keystone.exception.DomainNotFound: Could not find domain: ec213443e8834473b579f7bea9e8c194.

and the same for the project Could not find project: bbd5e2c3a2944f128b3bba616eb18eb6.: keystone.exception.ProjectNotFound: Could not find project: bbd5e2c3a2944f128b3bba616eb18eb6.

But as the IDs are correct and OpenStack itself should be able to handle IDs I assume that cyclone is not sending them correctly. If IDs are not possible to use, I'd appreciate a notice in the help that only names are allowed.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.