erigones / esdc-ce Goto Github PK
View Code? Open in Web Editor NEWDanube Cloud :: Community Edition
Home Page: https://danubecloud.org
License: Apache License 2.0
Danube Cloud :: Community Edition
Home Page: https://danubecloud.org
License: Apache License 2.0
Super admit is modify user, and add SSH key for user raises exception:
2016-12-12 16:40:28,270 ERROR [api.views]: Invalid field name(s) given in select_related: 'owner'. Choices are: dc_bound, default_dc, userprofile, auth_token
Traceback (most recent call last):
File "/home/ricco/Erigones/esdc-ce/api/views.py", line 484, in dispatch
response = handler(request, *args, **kwargs)
File "/home/ricco/Erigones/esdc-ce/api/decorators.py", line 91, in handler
return func(*args, **kwargs)
File "/home/ricco/Erigones/esdc-ce/api/decorators.py", line 212, in wrap
return fun(request, *args, **kwargs)
File "/home/ricco/Erigones/esdc-ce/api/accounts/user/sshkey/views.py", line 112, in sshkey_manage
return UserSshkeyView(request, username, title, data).response()
File "/home/ricco/Erigones/esdc-ce/api/accounts/user/sshkey/api_views.py", line 22, in __init__
user = get_user(request, username, exists_ok=True, noexists_fail=True)
File "/home/ricco/Erigones/esdc-ce/api/accounts/user/utils.py", line 29, in get_user
return get_virt_object(request, User, get_attrs={'username': username}, where=where, **kwargs)
File "/home/ricco/Erigones/esdc-ce/api/utils/db.py", line 140, in get_virt_object
obj = get_object(request, model, get_attrs, sr=sr, pr=pr, extra=extra, where=where, **kwargs)
File "/home/ricco/Erigones/esdc-ce/api/utils/db.py", line 39, in get_object
obj = qs.filter(where).get(**attrs)
File "/home/ricco/Erigones/esdc-ce/envs/lib/python2.7/site-packages/django/db/models/query.py", line 328, in get
num = len(clone)
File "/home/ricco/Erigones/esdc-ce/envs/lib/python2.7/site-packages/django/db/models/query.py", line 144, in __len__
self._fetch_all()
File "/home/ricco/Erigones/esdc-ce/envs/lib/python2.7/site-packages/django/db/models/query.py", line 965, in _fetch_all
self._result_cache = list(self.iterator())
File "/home/ricco/Erigones/esdc-ce/envs/lib/python2.7/site-packages/django/db/models/query.py", line 238, in iterator
results = compiler.execute_sql()
File "/home/ricco/Erigones/esdc-ce/envs/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 829, in execute_sql
sql, params = self.as_sql()
File "/home/ricco/Erigones/esdc-ce/envs/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 378, in as_sql
extra_select, order_by, group_by = self.pre_sql_setup()
File "/home/ricco/Erigones/esdc-ce/envs/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 48, in pre_sql_setup
self.setup_query()
File "/home/ricco/Erigones/esdc-ce/envs/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 39, in setup_query
self.select, self.klass_info, self.annotation_col_map = self.get_select()
File "/home/ricco/Erigones/esdc-ce/envs/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 206, in get_select
related_klass_infos = self.get_related_selections(select)
File "/home/ricco/Erigones/esdc-ce/envs/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 754, in get_related_selections
', '.join(_get_field_choices()) or '(none)',
FieldError: Invalid field name(s) given in select_related: 'owner'. Choices are: dc_bound, default_dc, userprofile, auth_token
2016-12-12 16:40:28,275 ERROR [django.request]: Internal Server Error: /dc/user/[email protected]/profile/ssh_key/add/form/
Traceback (most recent call last):
File "/home/ricco/Erigones/esdc-ce/envs/lib/python2.7/site-packages/django/core/handlers/base.py", line 132, in get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/home/ricco/Erigones/esdc-ce/envs/lib/python2.7/site-packages/django/contrib/auth/decorators.py", line 22, in _wrapped_view
return view_func(request, *args, **kwargs)
File "/home/ricco/Erigones/esdc-ce/gui/decorators.py", line 24, in wrap
return fun(request, *args, **kwargs)
File "/home/ricco/Erigones/esdc-ce/gui/decorators.py", line 81, in wrap
return fun(request, *args, **kwargs)
File "/home/ricco/Erigones/esdc-ce/gui/decorators.py", line 69, in wrap
return fun(request, *args, **kwargs)
File "/home/ricco/Erigones/esdc-ce/envs/lib/python2.7/site-packages/django/views/decorators/http.py", line 45, in inner
return func(request, *args, **kwargs)
File "/home/ricco/Erigones/esdc-ce/envs/lib/python2.7/site-packages/django/utils/decorators.py", line 145, in inner
return func(*args, **kwargs)
File "/home/ricco/Erigones/esdc-ce/gui/dc/user/views.py", line 230, in dc_user_profile_sshkey_modal_form
status = sform.save(action='create', args=(user.username, sform.cleaned_data['name']))
File "/home/ricco/Erigones/esdc-ce/gui/forms.py", line 298, in save
res = self.api_call(action, self._obj, self._request, args=args, data=data)
File "/home/ricco/Erigones/esdc-ce/gui/forms.py", line 275, in api_call
return call_api_view(request, method, cls._api_call.__func__, *args, data=dict(data), log_response=True)
File "/home/ricco/Erigones/esdc-ce/api/utils/views.py", line 21, in call_api_view
res = fun(request, *args, **kwargs)
File "/home/ricco/Erigones/esdc-ce/envs/lib/python2.7/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
return view_func(*args, **kwargs)
File "/home/ricco/Erigones/esdc-ce/envs/lib/python2.7/site-packages/django/views/generic/base.py", line 71, in view
return self.dispatch(request, *args, **kwargs)
File "/home/ricco/Erigones/esdc-ce/api/views.py", line 487, in dispatch
response = self.handle_exception(exc)
File "/home/ricco/Erigones/esdc-ce/api/views.py", line 454, in handle_exception
raise exc
FieldError: Invalid field name(s) given in select_related: 'owner'. Choices are: dc_bound, default_dc, userprofile, auth_token
Implement advanced compute node monitoring and graphs in the Danube Cloud API and GUI.
The main idea behind this task is to further integrate the monitoring system into Danube Cloud.
Create a new API call: mon_node_history
similar to mon_vm_history
.
Monitoring items: api/mon/node/graphs.py
CPU utilization
['system.cpu.util[,user,]', 'system.cpu.util[,system,]']
1-minute load average
['system.cpu.load[,avg1]']
CPU jumps
['system.cpu.switches', 'system.cpu.intr']
Memory usage
vm.memory.size[used]
Swap usage
system.swap.size[,used]
- erigones/esdc-factory#212 graphs per each network interface:
NIC ???? bandwidth
['net.if.in[????]', 'net.if.out[????]']
NIC ???? packets
['net.if.in[????,packets]', 'net.if.out[????,packets]']
2 graphs per each storage pool (zpool = ????):
Storage ???? throughput
['zpool.iostat[????,nread]', 'zpool.iostat[????,nwritten]']
Storage ???? I/O
['zpool.iostat[????,reads]', 'zpool.iostat[????,writes]']
NEW: Display free/used space on the dataset: zfs.list[????, used]
or zfs.list[????, avail]
- either a normal graph or maybe a pie chart showing distribution of the used disk space between: zfs.usedds[????]
, zfs.usedrefreserv[????]
and zfs.usedsnap[????]
For example, one can use "G" for gigabytes, "M" for megabytes in disk size input.
Add views for listing actual and historical alerts per VM, Node and DC (similar to task log list).
When adding nodem onitoring the call_mon_histoy_task() function was created. This new function is used by both Node and Vm to retrieve history. However, the dc_bound parameter, which for the node should be set to False.
ssl_key_file
and ssl_cert_file
variable into core.settings, name them e.g. UPDATE_KEY_FILE
, UPDATE_CERT_FILE
The data (notes) can be stored in existing json fields in DB.
Currently changing datacenter is not intuitive. Link to modal window to switch DC is confusing for user.
We need a framework for performing updates and changes of OS packages and services running in our appliances (e.g. changes in PostgreSQL configuration, OS updates, etc.). Let's use Ansible for that. Ansible playbooks should live in this repository.
Critical typo found in POST imagestore_image_manage:
Traceback (most recent call last):
File "/Users/dano/erigones/esdc-ce/api/views.py", line 484, in dispatch
response = handler(request, *args, **kwargs)
File "/Users/dano/erigones/esdc-ce/api/decorators.py", line 91, in handler
return func(*args, **kwargs)
File "/Users/dano/erigones/esdc-ce/api/decorators.py", line 212, in wrap
return fun(request, *args, **kwargs)
File "/Users/dano/erigones/esdc-ce/api/imagestore/image/views.py", line 63, in imagestore_image_manage
return ImageStoreImageView(request, name, uuid, data).response()
File "/Users/dano/erigones/esdc-ce/api/api_views.py", line 98, in response
return fun(*args, **kwargs)
File "/Users/dano/erigones/esdc-ce/api/imagestore/image/api_views.py", line 54, in post
data['manifest_url'] = self.repo.get_image_manifes_url(img['uuid'])
File "/Users/dano/erigones/esdc-ce/vms/models/base.py", line 59, in __getattr__
return self._data[key]
KeyError: 'get_image_manifes_url'
Append VM
to Linux
, BSD
, Windows
and SunOS
ostypes to indicate that these are KVM virtual machines.
Documentation (api docs and user guide) needs to be updated afterwards.
The Ctrl-Alt-Del shortcut is frequently used to access the login window on a Windows machine.
2017-01-01 13:45:31,354 ERROR [api.views]: placeholder_validator() argument after ** must be a mapping, not NoneType
Traceback (most recent call last):
File "/opt/erigones/api/views.py", line 484, in dispatch
response = handler(request, *args, **kwargs)
File "/opt/erigones/api/decorators.py", line 91, in handler
return func(*args, **kwargs)
File "/opt/erigones/api/decorators.py", line 212, in wrap
return fun(request, *args, **kwargs)
File "/opt/erigones/api/dc/base/views.py", line 170, in dc_settings
return DcSettingsView(request, dc, data).response()
File "/opt/erigones/api/api_views.py", line 98, in response
return fun(*args, **kwargs)
File "/opt/erigones/api/dc/base/dc_settings.py", line 38, in put
if not ser.is_valid():
File "/opt/erigones/api/serializers.py", line 647, in is_valid
return not self.errors
File "/opt/erigones/api/serializers.py", line 639, in errors
ret = self.from_native(data, files)
File "/opt/erigones/api/serializers.py", line 464, in from_native
attrs = self.perform_validation(attrs)
File "/opt/erigones/api/serializers.py", line 395, in perform_validation
attrs = validate_method(attrs, source)
File "/opt/erigones/api/dc/base/serializers.py", line 564, in validate_MON_ZABBIX_TEMPLATES_VM_NIC
return validate_array_placeholders(attrs, source, VM_KWARGS.copy().update({'net': 1, 'nic_id': 2}))
File "/opt/erigones/api/dc/base/serializers.py", line 40, in validate_array_placeholders
placeholder_validator(i, **valid_placeholders)
TypeError: placeholder_validator() argument after ** must be a mapping, not NoneType
The API call will be used for implementing templates suggestions dropdown in GUI.
There are few problems that fix for this issue needs to address:
API calls related to image imports (either POST image_manage or POST imagestore_image_manage) ignore our image tags (deploy, resize and erigones attributes in the tags metadata).
Dynamically synchronize host groups in Zabbix according to monitoring hostgroups setting and DC association in esdc.
Add option to skip reservation of vCPUs and RAM for a replicated VM. These resources should be reserved on a compute node only if the VM is promoted to master there.
Both, gunicorn-sio and gunicorn-api use gevent workers and do not reflect new code after SIGHUP.
The problem could be solved by using gunicorn's SIGUSR2 upgrade mechanism, which was used before we've migrated to systemd. With systemd it is not possible to use SIGUSR2 and SIGTERM on the old gunicorn master.
It may be possible to solve the problem just by restarting all gunicorns via systemd.
str.format()
is a bit unsecure. So lets improve validation on DC settings, which support placeholders (this is the only place where we use format()
). This should be done in api.dc.base.serializers.placeholder_validator()
. Please move this function into api.validators
.
Target branch: master (version: 2.3.3)
Otherwise gunicorn/erigonesd can potentially consume all memory.
Hi,
there's a problem with json handling. It affects IP address changing after VM is destroyed and re-deployed. Maybe it also affects other VM settings.
Steps to reproduce:
The VM will be deployed with the old IP address.
Jan
Characters allowed in email address field when registering are not the same as characters checked by url regex.
Definitive solution TBD in comments.
Traceback (most recent call last):
File "/opt/erigones/envs/lib/python2.7/site-packages/django/core/handlers/base.py", line 112, in get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/opt/erigones/envs/lib/python2.7/site-packages/django/contrib/auth/decorators.py", line 22, in _wrapped_view
return view_func(request, *args, **kwargs)
File "/opt/erigones/gui/decorators.py", line 24, in wrap
return fun(request, *args, **kwargs)
File "/opt/erigones/gui/decorators.py", line 53, in wrap
return fun(request, *args, **kwargs)
File "/opt/erigones/gui/dc/user/views.py", line 67, in dc_user_list
return render(request, 'gui/dc/user_list.html', context)
File "/opt/erigones/envs/lib/python2.7/site-packages/django/shortcuts/__init__.py", line 53, in render
return HttpResponse(loader.render_to_string(*args, **kwargs),
File "/opt/erigones/envs/lib/python2.7/site-packages/django/template/loader.py", line 169, in render_to_string
return t.render(context_instance)
File "/opt/erigones/envs/lib/python2.7/site-packages/django/template/base.py", line 140, in render
return self._render(context)
File "/opt/erigones/envs/lib/python2.7/site-packages/django/template/base.py", line 134, in _render
return self.nodelist.render(context)
File "/opt/erigones/envs/lib/python2.7/site-packages/django/template/base.py", line 840, in render
bit = self.render_node(node, context)
File "/opt/erigones/envs/lib/python2.7/site-packages/django/template/base.py", line 854, in render_node
return node.render(context)
File "/opt/erigones/envs/lib/python2.7/site-packages/django/template/loader_tags.py", line 123, in render
return compiled_parent._render(context)
File "/opt/erigones/envs/lib/python2.7/site-packages/django/template/base.py", line 134, in _render
return self.nodelist.render(context)
File "/opt/erigones/envs/lib/python2.7/site-packages/django/template/base.py", line 840, in render
bit = self.render_node(node, context)
File "/opt/erigones/envs/lib/python2.7/site-packages/django/template/base.py", line 854, in render_node
return node.render(context)
File "/opt/erigones/envs/lib/python2.7/site-packages/django/template/loader_tags.py", line 62, in render
result = block.nodelist.render(context)
File "/opt/erigones/envs/lib/python2.7/site-packages/django/template/base.py", line 840, in render
bit = self.render_node(node, context)
File "/opt/erigones/envs/lib/python2.7/site-packages/django/template/base.py", line 854, in render_node
return node.render(context)
File "/opt/erigones/envs/lib/python2.7/site-packages/django/template/defaulttags.py", line 203, in render
nodelist.append(node.render(context))
File "/opt/erigones/envs/lib/python2.7/site-packages/django/template/defaulttags.py", line 305, in render
return nodelist.render(context)
File "/opt/erigones/envs/lib/python2.7/site-packages/django/template/base.py", line 840, in render
bit = self.render_node(node, context)
File "/opt/erigones/envs/lib/python2.7/site-packages/django/template/base.py", line 854, in render_node
return node.render(context)
File "/opt/erigones/envs/lib/python2.7/site-packages/django/template/defaulttags.py", line 447, in render
six.reraise(*exc_info)
File "/opt/erigones/envs/lib/python2.7/site-packages/django/template/defaulttags.py", line 433, in render
url = reverse(view_name, args=args, kwargs=kwargs, current_app=context.current_app)
File "/opt/erigones/envs/lib/python2.7/site-packages/django/core/urlresolvers.py", line 536, in reverse
return iri_to_uri(resolver._reverse_with_prefix(view, prefix, *args, **kwargs))
File "/opt/erigones/envs/lib/python2.7/site-packages/django/core/urlresolvers.py", line 456, in _reverse_with_prefix
(lookup_view_s, args, kwargs, len(patterns), patterns))
NoReverseMatch: Reverse for 'dc_user_profile' with arguments '()' and keyword arguments '{u'username': u'[email protected]'}' not found. 1 pattern(s) tried: ['dc/user/(?P[A-Za-z0-9\@\._-]+)/profile/$']
Observed when creating SuperAdmin and leaving DC bound checked.
When a mgmt callback fails during execution because of an operational error (DBError, Redis/RabbitMQ connection problem) we should retry the task.
Hi,
Where: Datacenter -> Images -> Import disk Image from repository.
What: it would be nice if there was longer default time period for filtering repository images. Last month seems to be too strict and it doesn't show many relevant images.
Thx.
Jan
When restoring backup, user may choose the VM, into which the data will be restored. It would be usefull if there was an option to restore to new server.
Specifically: first item in the list of VMs can be <<new VM>>
and new VM will be created according to definition of original VM.
Currently, the user has to create and deploy new VM manually before the restore.
Jan
/vm/(hostname)
) and GET vm_define (/vm/(hostname)/define
) output (read-only)Dynamically create users and user groups in Zabbix according to Groups in esdc.
The que.utils.delete_task()
is broken (celery inspection of a task candidate for deletion). We should fix this and make the DELETE /task/(task_id)
available for every PENDING task in the system.
The testing should also include use cases involving task cleanup (revoked tasks).
Target branch: v2.4
After #16 was implemented there is still one API call parameter that depends on VM's hostname - target_hostname
in PUT /vm/(hostname_or_uuid)/backup/(bkpname)
. Please update this parameter so it also accepts VM's uuid as value.
Please make the new target_hostname_or_uuid
required and remove the default value.
NOTE: Renaming the parameter will also affect the GUI (python or JS).
$ es create /network/(name) -vlan_id 0 -netmask 255.255.255.0 -nic_tag admin -network 192.168.33.0 -dc_bound false
...
"status": "FAILURE"
"result": {
"dhcp_passthrough": [
"This field is required."
]
},
...
Solution: Add a default value (false
) to dhcp_passthrough in the serializer (this is already documented in the API docs).
Lets show the user actual snapshot names and backup paths in the API. The idea of this change is to enable users to easily identify certain snapshots/backups on the FS, which might be helpful in some (disaster) situations.
Suggested read-only attributes (let's discuss this first):
id
- Snapshot.zfs_name
file_path
- Backup.file_path
(This is a really simple change - 2 lines of code)
When postgresql database is unreachable (or login fails), the erigonesd-beat crashes. After crash, the service stays down. Instead, it should go into infinite connect loop waiting for the database because this can be only temporary problem.
Continuation of issue #16
The field should be a read-only field similar to the hostname
attribute.
Affected files identified so far:
Add help_text to VM tags explaining that a new tag can be created by pressing Enter.
Configure Zabbix actions via esdc API/GUI.
Consider removal of empty hostgroups after this task is implemented .
The sphinx API documentation in doc/api/source
currently generates broken HTML links. That is because I've moved most of the rst files into a subfolder (api
).
It looks like that all broken links are generated through rst_epilog
in conf.py
, so the change should happen there.
You can use wget to find all 404s (e.g. wget -mkEpnv http://127.0.0.1:8000/static/api/doc/index.html
)
Hi,
in GUI serverlist (/servers), if some node is unavailable, the following message appears on the top: "Some servers are not visible or accessible, because some compute nodes are not in online state. Servers are kept untouched. Full server control will be possible as soon as the compute nodes return to online state".
But when the node becomes reachable again and status of all nodes is green, the above message does not go away until the page is manually reloaded.
Also when any node becomes unavailable, the message does not appear until page is reloaded.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.