Hi again,

live migration between two OpenNebula KVM hosts is failing.
Addon-On Version 0.92
linstor-controller version 0.7.3
OpenNebula 5.6.1
OS: Ubuntu 18.04
Datastore: Autoplace, 2 Copies, clonetype: copy

Error from OpenNebula Log:
Wed Jan 30 08:09:43 2019 [Z0][VM][I]: New LCM state is MIGRATE
Wed Jan 30 08:09:50 2019 [Z0][VMM][I]: Successfully execute transfer manager driver operation: tm_premigrate.
Wed Jan 30 08:09:52 2019 [Z0][VMM][I]: ExitCode: 0
Wed Jan 30 08:09:52 2019 [Z0][VMM][I]: Successfully execute network driver operation: pre.
Wed Jan 30 08:09:55 2019 [Z0][VMM][I]: Command execution fail: cat << EOT | /var/tmp/one/vmm/kvm/migrate 'one-255' 'one2' 'one1' 255 one1
Wed Jan 30 08:09:55 2019 [Z0][VMM][E]: migrate: Command "virsh --connect qemu:///system migrate --live one-255 qemu+ssh://one2/system" failed: error: Cannot access storage file '/var/lib/one//datastores/104/255/disk.1': No such file or directory
Wed Jan 30 08:09:55 2019 [Z0][VMM][E]: Could not migrate one-255 to one2
Wed Jan 30 08:09:55 2019 [Z0][VMM][I]: ExitCode: 1

At the same time the linstor-controller shows the following error:
LINSTOR ==> error-reports show 5C49A05B-00000-000022
ERROR REPORT 5C49A05B-00000-000022

============================================================

Application: LINBIT® LINSTOR
Module: Controller
Version: 0.7.5
Build ID: d74305b420fdc878182afa162378a317e6a4a3b9
Build time: 2018-12-21T09:05:14+00:00
Error time: 2019-01-30 08:09:47
Node: sun1
Peer: 0:0:0:0:0:0:0:1:55948

============================================================

Reported error:

Category: Exception
Class name: InvalidNameException
Class canonical name: com.linbit.InvalidNameException
Generated at: Method 'nameCheck', Source file 'Checks.java', Line #71

Error message: Invalid name: Name length 0 is less than minimum length 2

Error context:
The specified resource name '' is invalid.

Call backtrace:

Method                                   Native Class:Line number
nameCheck                                N      com.linbit.Checks:71
<init>                                   N      com.linbit.linstor.ResourceName:33
asRscName                                N      com.linbit.linstor.LinstorParsingUtils:127
createRscDfn                             N      com.linbit.linstor.core.apicallhandler.controller.CtrlRscDfnApiCallHandler:383
createResourceDefinition                 N      com.linbit.linstor.core.apicallhandler.controller.CtrlRscDfnApiCallHandler:136
createResourceDefinition                 N      com.linbit.linstor.core.apicallhandler.controller.CtrlApiCallHandler:248
execute                                  N      com.linbit.linstor.api.protobuf.controller.CreateResourceDefinition:54
executeNonReactive                       N      com.linbit.linstor.proto.CommonMessageProcessor:520
lambda$execute$13                        N      com.linbit.linstor.proto.CommonMessageProcessor:496
doInScope                                N      com.linbit.linstor.core.apicallhandler.ScopeRunner:141
lambda$fluxInScope$0                     N      com.linbit.linstor.core.apicallhandler.ScopeRunner:71
call                                     N      reactor.core.publisher.MonoCallable:92
trySubscribeScalarMap                    N      reactor.core.publisher.FluxFlatMap:126
subscribe                                N      reactor.core.publisher.MonoFlatMapMany:46
subscribe                                N      reactor.core.publisher.Flux:6877
onNext                                   N      reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain:184
request                                  N      reactor.core.publisher.Operators$ScalarSubscription:1640
onSubscribe                              N      reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain:131
subscribe                                N      reactor.core.publisher.MonoCurrentContext:35
subscribe                                N      reactor.core.publisher.MonoFlatMapMany:49
subscribe                                N      reactor.core.publisher.FluxOnAssembly:252
subscribe                                N      reactor.core.publisher.FluxOnAssembly:252
subscribe                                N      reactor.core.publisher.FluxOnErrorResume:47
subscribe                                N      reactor.core.publisher.FluxOnErrorResume:47
subscribe                                N      reactor.core.publisher.FluxOnErrorResume:47
subscribe                                N      reactor.core.publisher.FluxOnErrorResume:47
subscribe                                N      reactor.core.publisher.FluxContextStart:49
subscribe                                N      reactor.core.publisher.Flux:6877
onNext                                   N      reactor.core.publisher.FluxFlatMap$FlatMapMain:372
slowPath                                 N      reactor.core.publisher.FluxArray$ArraySubscription:126
request                                  N      reactor.core.publisher.FluxArray$ArraySubscription:99
onSubscribe                              N      reactor.core.publisher.FluxFlatMap$FlatMapMain:332
subscribe                                N      reactor.core.publisher.FluxMerge:70
subscribe                                N      reactor.core.publisher.FluxOnErrorResume:47
subscribe                                N      reactor.core.publisher.Flux:6877
onComplete                               N      reactor.core.publisher.FluxConcatArray$ConcatArraySubscriber:208
subscribe                                N      reactor.core.publisher.FluxConcatArray:81
subscribe                                N      reactor.core.publisher.FluxPeek:83
subscribe                                N      reactor.core.publisher.FluxPeek:83
subscribe                                N      reactor.core.publisher.FluxPeek:83
subscribe                                N      reactor.core.publisher.FluxOnErrorResume:47
subscribe                                N      reactor.core.publisher.FluxDefer:55
subscribe                                N      reactor.core.publisher.Flux:6877
onNext                                   N      reactor.core.publisher.FluxFlatMap$FlatMapMain:372
drainAsync                               N      reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber:391
drain                                    N      reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber:633
onNext                                   N      reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber:238
drainFused                               N      reactor.core.publisher.UnicastProcessor:234
drain                                    N      reactor.core.publisher.UnicastProcessor:267
onNext                                   N      reactor.core.publisher.UnicastProcessor:343
next                                     N      reactor.core.publisher.FluxCreate$IgnoreSink:573
next                                     N      reactor.core.publisher.FluxCreate$SerializedSink:151
processInOrder                           N      com.linbit.linstor.netcom.TcpConnectorPeer:361
doProcessMessage                         N      com.linbit.linstor.proto.CommonMessageProcessor:215
lambda$processMessage$2                  N      com.linbit.linstor.proto.CommonMessageProcessor:161
onNext                                   N      reactor.core.publisher.FluxPeek$PeekSubscriber:177
runAsync                                 N      reactor.core.publisher.FluxPublishOn$PublishOnSubscriber:396
run                                      N      reactor.core.publisher.FluxPublishOn$PublishOnSubscriber:480
call                                     N      reactor.core.scheduler.WorkerTask:84
call                                     N      reactor.core.scheduler.WorkerTask:37
run                                      N      java.util.concurrent.FutureTask:264
run                                      N      java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask:304
runWorker                                N      java.util.concurrent.ThreadPoolExecutor:1135
run                                      N      java.util.concurrent.ThreadPoolExecutor$Worker:635
run                                      N      java.lang.Thread:844

END OF ERROR REPORT.

The live migration has worked before using an older version of the add-on.

Best Regards
Uli

Hi,

I created a VM from the Sunstone Web Interface in my test setup (2 OpenNebula KVM Nodes with Linstor Satellite & 1 OpenNebula Frontend with Linstor Controller) and powered it off. After that I created a disk snapshot (from the Storage Tab of the VM).

The Linstor Controller shows following configuration

LINSTOR ==> resource list
ResourceName ┊ Node ┊ Port ┊ Usage ┊ State ┊
OpenNebula-Image-1 ┊ srv200 ┊ 7001 ┊ Unused ┊ UpToDate ┊
OpenNebula-Image-1 ┊ srv201 ┊ 7001 ┊ Unused ┊ UpToDate ┊
OpenNebula-Image-1-vm1-disk0 ┊ srv200 ┊ 7002 ┊ Unused ┊ UpToDate ┊
OpenNebula-Image-1-vm1-disk0 ┊ srv201 ┊ 7002 ┊ Unused ┊ UpToDate ┊

LINSTOR ==> snapshot list
ResourceName ┊ SnapshotName ┊ NodeNames ┊ Volumes ┊ State ┊
OpenNebula-Image-1-vm1-disk0 ┊ OpenNebula-Snap-0 ┊ srv200, srv201 ┊ 0: 200 MiB ┊ Successful ┊

Looks as expected.
But when I try to delete the snapshot from Sunstone the oned.log shows following error message and the snapshot still exists:
Tue Jan 22 14:35:22 2019 [Z0][TM][D]: Message received: LOG I 1 Command execution failed (exit code: 1): /var/lib/one/remotes/tm/linstor/snap_delete srv201:/var/lib/one//datastores/100/1/disk.0 0 1 102

Tue Jan 22 14:35:22 2019 [Z0][TM][D]: Message received: LOG I 1 Traceback (most recent call last):

Tue Jan 22 14:35:22 2019 [Z0][TM][D]: Message received: LOG I 1 File "/var/lib/one/remotes/tm/linstor/snap_delete", line 55, in

Tue Jan 22 14:35:22 2019 [Z0][TM][D]: Message received: LOG I 1 main()

Tue Jan 22 14:35:22 2019 [Z0][TM][D]: Message received: LOG I 1 File "/var/lib/one/remotes/tm/linstor/snap_delete", line 48, in main

Tue Jan 22 14:35:22 2019 [Z0][TM][D]: Message received: LOG I 1 res.snapshot_delete("{}-{}".format(consts.SNAP_PREFIX, snap_ID))

Tue Jan 22 14:35:22 2019 [Z0][TM][D]: Message received: LOG I 1 File "/usr/lib/python2.7/dist-packages/linstor/resource.py", line 444, in snapshot_delete

Tue Jan 22 14:35:22 2019 [Z0][TM][D]: Message received: LOG I 1 raise linstor.LinstorError('Could not delete snapshot {}: {}'.format(name, rs[0].message))

Tue Jan 22 14:35:22 2019 [Z0][TM][D]: Message received: LOG I 1 linstor.errors.LinstorError: Error: Could not delete snapshot OpenNebula-Snap-0: Snapshot 'OpenNebula-Snap-0' of resource 'OpenNebula-Image-1' not found.

Tue Jan 22 14:35:22 2019 [Z0][TM][D]: Message received: TRANSFER FAILURE 1 -

It seems to me that the referenced resource ('OpenNebula-Image-1') of the snapshot is the wrong one. The snapshot belongs to OpenNebula-Image-1-vm1-disk0 in my opinion.
I get a similar error message when trying to revert the snapshot.

Best Regards
Uli

Hi again,

FILE_DS functionality is not working. I am using the latest release (0.9.3) of the addon. The documentation states states it should work... "Linstor driver can also be used as a system/context datastore," but the example is only for a SYSTEM_DS.

Steps to reproduce:

1. Create linstor image datastore and system datastore according documentation. VM instantiation (tty linux from ONE Marketplace) works fine.
2. Create linstor file_ds datastore from the following template:
   NAME="hci-file_ds_linstor_a2"
   DS_MAD="linstor"
   TYPE="FILE_DS"
   TM_MAD="linstor"
   LINSTOR_AUTO_PLACE="2"
   LINSTOR_STORAGE_POOL="filauto2copy"
   BRIDGE_LIST= "srv362 srv363 srv364"
3. Upload a testfile to this file_ds datastore works fine
4. Adding the test file to the VM Template (Context -> Files) and instantiate produces following error in oned.log

Sun Feb 3 13:15:43 2019 [Z0][VMM][D]: Message received: LOG I 2 Command execution failed (exit code: 1): /var/lib/one/remotes/tm/linstor/context /var/lib/one/vms/2/context.sh OpenNebula-Image-0:'test' srv364:/var/lib/one//datastores/104/2/disk.2 2 104

Sun Feb 3 13:15:43 2019 [Z0][VMM][D]: Message received: LOG I 2 Traceback (most recent call last):

Sun Feb 3 13:15:43 2019 [Z0][VMM][D]: Message received: LOG I 2 File "/var/lib/one/remotes/tm/linstor/context", line 137, in

Sun Feb 3 13:15:43 2019 [Z0][VMM][D]: Message received: LOG I 2 main()

Sun Feb 3 13:15:43 2019 [Z0][VMM][D]: Message received: LOG I 2 File "/var/lib/one/remotes/tm/linstor/context", line 71, in main

Sun Feb 3 13:15:43 2019 [Z0][VMM][D]: Message received: LOG I 2 shutil.copy2(file, os.path.join(iso_dir, os.path.basename(file)))

Sun Feb 3 13:15:43 2019 [Z0][VMM][D]: Message received: LOG I 2 File "/usr/lib/python2.7/shutil.py", line 144, in copy2

Sun Feb 3 13:15:43 2019 [Z0][VMM][D]: Message received: LOG I 2 copyfile(src, dst)

Sun Feb 3 13:15:43 2019 [Z0][VMM][D]: Message received: LOG I 2 File "/usr/lib/python2.7/shutil.py", line 96, in copyfile

Sun Feb 3 13:15:43 2019 [Z0][VMM][D]: Message received: LOG I 2 with open(src, 'rb') as fsrc:

Sun Feb 3 13:15:43 2019 [Z0][VMM][D]: Message received: LOG I 2 IOError: [Errno 2] No such file or directory: 'OpenNebula-Image-0:test'

Sun Feb 3 13:15:43 2019 [Z0][VMM][D]: Message received: LOG I 2 Failed to execute transfer manager driver operation: tm_context.

Sun Feb 3 13:15:43 2019 [Z0][VMM][D]: Message received: DEPLOY FAILURE 2 -

Sun Feb 3 13:15:54 2019 [Z0][InM][D]: Monitoring datastore default (1)

Best Regards
Uli