If the User will not generate a network or a storage pool and will try to access it he/she will get a misleading AttributeError.

Traceback (most recent call last):
  File "execute_sample.py", line 66, in <module>
    main()
  File "execute_sample.py", line 20, in main
    protocol(environment.context, arguments.sample, arguments.command)
  File "execute_sample.py", line 26, in protocol
    wait_for_ip_address(context, TIMEOUT)
  File "execute_sample.py", line 44, in wait_for_ip_address
    if context.ip4_address is not None:
  File "/usr/local/lib/python3.4/dist-packages/see/context/context.py", line 193, in ip4_address
    if self._ip4_address is None and self.network is not None:
  File "/usr/local/lib/python3.4/dist-packages/see/context/context.py", line 161, in network
    if self._resources.network.isActive():
AttributeError: 'NoneType' object has no attribute 'isActive'

As well, the Tutorial does not mention the network in the context.json file.

When configuring SEE with automatic IP address assignment, if too many instances are started at the same time, one or more might not be able to get a valid IP address and will give up.

SEE inspect the addresses in use when generating a new one. It will pick the next available and try to reserve it. If a concurrent instance will get that IP address at the same time, SEE will detect that and repeat the procedure. This is repeated an arbitrarily amount of times.

If several instances are started at the same time, one or more might exceed the amount of attempts before getting a valid one.

A possible mitigation for the issue would be randomising the IP address selection reducing the risk of concurrent collisions.

Hi,

I think you introduced a regression in the releases after 1.2.6.

Using 1.2.6 it works fine.

My program doesn't work anymore (with 1.2.12, latest release)

Thanks.

I ran in to troubles with version 5.1.0 of qemu with disk.py. Converting from qemu internal snapshot to qcow2 fails with:

RuntimeError: qemu-img convert -f qcow2 -o backing_file=/var/lib/libvirt/images/win7.qcow2 -O qcow2 -s ip_address /home/kedle/see/instances/98ea3022-83de-4bd2-9fb9-90a806124903/98ea3022-83de-4bd2-9fb9-90a806124903 /home/kedle/see/disklogs/ip_address.qcow2
qemu-img: unrecognized option '-s'

Turns out the -s option for convert has been removed in version 3.0:

The qemu-img "convert -s snapshot_id_or_name" argument has been removed. Use "convert -l snapshot_param" instead.

I got disk snapshotting and analysis working by changing the command formulation in disk.py from this:

process = launch_process(QEMU_IMG, "convert", "-f", "qcow2", "-o",
                         "backing_file=%s" % volume_backing_path(volume),
                         "-O", "qcow2", "-s", name,
                         volume_path(volume), path)

to this:

process = launch_process(QEMU_IMG, "convert", "-U", "-f", "qcow2", "-o",
                         "backing_file=%s" % volume_backing_path(volume),
                         "-O", "qcow2", "-l", name,
                         volume_path(volume), path)

I had to include -U (or --force-share) to be able to read the snapshot

--force-share (-U)
If specified, qemu-img will open the image in shared mode, allowing other QEMU processes to open it in write mode. For example, this can be used to get the image information (with ‘info’ subcommand) when the image is used by a running guest. Note that this could produce inconsistent results because of concurrent metadata changes, etc. This option is only allowed when opening images in read-only mode.

Result:

qemu-img convert -U -f qcow2 -o backing_file=/var/lib/libvirt/images/win7.qcow2 -O qcow2 -l post_shutdown /home/kedle/see/instances/d1c9e7ea-0387-4ff8-870c-36510e9e0a3f/d1c9e7ea-0387-4ff8-870c-36510e9e0a3f /home/kedle/see/disklogs/post_shutdown.qcow2

Don't know if this would work with older QEMU. My version information:
Python 3.8.5
QEMU 5.1.0
libvirt 6.5.0

Issue Type

[x] Bug (Typo)

Steps to Replicate and Expected Behaviour

• Examine see/context/resources/test/lxc_test.py, see/context/resources/test/qemu_test.py, see/context/resources/test/vbox_test.py search for returining however expect to see returning.
• Examine see/context/resources/test/vbox_test.py search for allocater however expect to see allocator.
• Examine see/image_providers/test/s3_test.py search for downloaede however expect to see downloaded.
• Examine doc/tutorial_setup.rst search for correcly however expect to see correctly.
• Examine plugins/memory.py search for asyncronous however expect to see asynchronous.

Notes

Semi-automated issue generated by
https://github.com/timgates42/meticulous/blob/master/docs/NOTE.md

To avoid wasting CI processing resources a branch with the fix has been
prepared but a pull request has not yet been created. A pull request fixing
the issue can be prepared from the link below, feel free to create it or
request @timgates42 create the PR.

https://github.com/timgates42/see/pull/new/bugfix_typos

Thanks.

Add a plugin example for filesystem analysis.

Hi,

I was following the tutorial and at this particular section where the code to invoke VNC was abstracted to a separate plugin/hook.
Despite following the steps, the execution does not seem to invoke the VNC plugin.

My current setup is described below:

sandbox.py
Same as what is presented at: http://pythonhosted.org/python-see/tutorial_plugins.html

vnc.py
Same as what is presented at: http://pythonhosted.org/python-see/tutorial_plugins.html

context.json

{
    "hypervisor": "qemu:///system",
    "domain":
    {
        "configuration": "<path to config>.xml"
    },
    "disk":
    {
        "image":
        {
          "uri": "<path to diskimage>.qcow2",
          "provider": "see.image_providers.DummyProvider"
        },
        "clone":
        {
            "storage_pool_path": "<path to pool>/instances",
            "copy_on_write": true
        }
    }
}

hooks.json

{
    "hooks":
    [
        {
            "name": "vnc2.VNCHook",
            "configuration": {
                "start_vnc": "post_poweron"
            }
        }
    ]
}

All the files are in the same directory and at the same level.
Upon executing
python sandbox.py context.json hello --hooks hooks.json --command "dummy-command"

The sandbox instance starts but the vnc plugin and it's virt-viewer command doesn't seem to be triggered by the post_poweron event. In addition, there is a console message saying
No handlers could be found for logger "see.hooks.HookManager"

Am I missing something? I'm running this on a linux environment and my VM is a linux machine as well. virt-viewer was able to be started previously at this step

Thank you for the help!

Currently, only way to provide disk images to SEE is via the implicit use of a libvirt FS based storage pool. There are use cases in which the image could be provided via other means (example: raw FS or using a image manager such as Glance.

To support that, we need to add a plugin based image manager. Most of the Hooks logic can be re-used (maybe moved into a generic module?).

A possible configuration interface:

"disk":
{
    "image": 
    {
        "uri": "/original/image/location",
        "provider": "python.fully.qualified.Name",
        "provider_configuration": {}
    },
    "clone":
    {
        "storage_pool_path": "/destination/storage/pool"
    }
}

Example using current mechanism.

"disk":
{
    "image": 
    {
        "uri": "/var/lib/libvirt/images/disk_image.qcow2",
        "provider": "context.resources.image_managers.LibvirtStoragePool",
        "provider_configuration": 
        {
            "storage_pool_path": ""/var/lib/libvirt/"
        }
    },
    "clone":
    {
        "storage_pool_path": "/home/user/see_instances",
        "copy_on_write": true
    }
}

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/sandbox/task_manager.py", line 38, in process_task
    self.execute_task(task, results_path)
  File "/usr/lib/python3/dist-packages/sandbox/task_manager.py", line 58, in execute_task
    run_sandbox(parameters)
  File "/usr/lib/python3/dist-packages/sandbox/sandbox.py", line 38, in run_sandbox
    identifier=parameters.identifier) as environment:
  File "/usr/local/lib/python3.4/dist-packages/see/environment.py", line 59, in __enter__
    self.allocate()
  File "/usr/local/lib/python3.4/dist-packages/see/environment.py", line 80, in allocate
    self._allocate()
  File "/usr/local/lib/python3.4/dist-packages/see/environment.py", line 88, in _allocate
    self._context)
  File "/usr/local/lib/python3.4/dist-packages/see/hooks.py", line 33, in hooks_factory
    manager.load_hooks(context)
  File "/usr/local/lib/python3.4/dist-packages/see/hooks.py", line 55, in load_hooks
    self._load_hook(hook['name'], config, context)
KeyError: 'name'

It would probably be better to report the problematic Hook and continue the execution.

Xen support should be quite trivial to add.

The Context would be very close to QEMU one.

The user might provide his/her own libvirt network and attach the domain to it via its XML configuration.

In such cases, the Context object will have the network resource still set to None. This is due to the fact that the Context Factory is not aware of XML specific configurations.

This lead to bugs in the network based Plugins as they assume the presence of a libvirt network but they cannot reach it.

Possible solutions are:

• If no network is specified in the Context's JSON configuration, parse the XML and try to understand if the domain is attached to any network. This is a bit controversial as we try to be as much "libvirt config agnostic" as possible.
• Once the domain is created, try to lookup to which network is attached. This solution is simpler but requires libvirt APIs support. Investigation required.

Provide a comprehensive tutorial within the documentation.

The tutorial should cover the following topics:

• Setting up a Sandbox.
• Creating a test environment and controlling it's lifecycle.
• Writing a plugin.
• (optional) a complete example.

Hi,

This is a feature request.

I would like to have a debug mode in SEE, allowing me to log what signals have been triggered, and what associated handlers have being called.

like the following:

logging.debug("signal: %s -> handler: %s", signal, handler)

Hi !

i think i found an issue in see.
In the case where the domain XML provided is not valid for libvirt, see doesn't deallocate all resources properly.
An exception is raised and the network is not destroyed as it should be :

╰─$ virsh -c qemu:///system
Welcome to virsh, the virtualization interactive terminal.

Type:  'help' for help with commands
       'quit' to quit

virsh # net-list
 Name                 State      Autostart     Persistent
----------------------------------------------------------
 46d8ce09-8e60-4000-82e6-862aeba72783 active     no            no
 default              active     yes           yes

virsh # 

Write a plugin able to execute code within the guest.

The plugin should also allow to inject data (ex. files) within the guest.

Hi,

I noticed that when i install python-see from pip, the plugins are also installed:

In [1]: import plugins

In [2]: plugins.__file__
Out[2]: '/usr/local/lib/python3.6/dist-packages/python_see-1.2.6-py3.6.egg/plugins/__init__.py'

In [3]: 

Having a vague named such as plugins installed as a system wide module is a bit dangerous don't you think ?

Thanks.