roboconf / roboconf-platform Goto Github PK
View Code? Open in Web Editor NEWThe core modules and the platform
License: Apache License 2.0
The core modules and the platform
License: Apache License 2.0
Today, imports are mandatory.
If an instance does not resolve an import, it cannot start.
However, there are cases where the imports are optional.
It means that if there is no exporting instance, the current one can start anyway.
And if there are exporting instances, the current one wants to be notified about them.
This is the case for MongoDB when used to replicate data.
A MongoDB instance needs to know about the other ones. But it can start alone if necessary.
Add a PaaS plugin for Heroku provider:
Heroku (http://www.heroku.com/) is a PaaS cloud platform provider for services developed in Ruby, Node.js, Clojure, Java, Play, Python, and Scala (https://devcenter.heroku.com/categories/language-support). Application are deployed using Git (push). Applications are isolated by Linux containers (LXC). Heroku provides addons (https://addons.heroku.com/) for NoSQL databases systems, analytics, messaging, caching ....
The plugin must take into account the deployment of the addons required by the deployed service.
example of bash commands for a service deployment
cd nodered-mqttpanel
heroku create
git push heroku master
heroku addons:add openredis
heroku open
Currently, if a component does not have a directory, it should result in a WARNING message at validation-time. However, runtime exceptions may occur later. They should be ignored when they do not exist, instead of throwing an exception..
Instead of polling for one VM, we should maintain a list of VM under creation and regularly update them.
Whe listing instances, the REST API should reutrn them sorted correctly.
For the moment, there are returned by levels (instances from level 0 first, from level 1 then, then from level 2...).
They should be sorted by ancestor.
level 0, level 0 + level 11, level 0 + level 12, level 1, level 1 + level 11, etc.
This result in display bugs in web clients.
Add WAMP as messaging protocol and service
WAMP (Web Application Messaging Protocol) is an open WebSocket subprotocol that provides two application messaging patterns in one unified protocol: Remote Procedure Calls + Publish-Subscribe
http://wamp.ws/spec/
https://github.com/tavendo/WAMP/blob/master/spec/basic.md
https://github.com/tavendo/WAMP/blob/master/spec/advanced.md
For the moment, agents can use a Bash and a Puppet plug-in.
The Bash plug-in transmits environment variables to the bash scripts it executes.
The Puppet plug-in generates Puppet manifests before executing them.
The way variables are transmitted and the coherence of their usage is very different for both plug-ins. This makes their use complicated.
A templating system should be setup so that plug-ins can generate proper files before executing them. Velocity is a solution among others.
SEVERE: The RuntimeException could not be mapped to a response, re-throwing to the HTTP container
java.lang.IllegalArgumentException: Target directory apache-tomcat-webapp does not exist.
at net.roboconf.core.internal.utils.Utils.extractZipArchive(Utils.java:211)
at net.roboconf.dm.server.ManagementWs.loadApplication(ManagementWs.java:67)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:540)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:715)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:224)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:169)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:927)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:987)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:579)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:309)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
Apr 8, 2014 2:45:01 PM com.sun.jersey.spi.container.ContainerResponse mapMappableContainerException
SEVERE: The RuntimeException could not be mapped to a response, re-throwing to the HTTP container
java.lang.IllegalArgumentException: Target directory apache-tomcat-webapp does not exist.
at net.roboconf.core.internal.utils.Utils.extractZipArchive(Utils.java:211)
at net.roboconf.dm.server.ManagementWs.loadApplication(ManagementWs.java:67)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:540)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:715)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:224)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:169)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:927)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:987)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:579)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:309)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
After a long brainstorming and various tries, it seems that neither Bash or Puppet constitute a user-friendly solution to deploy Node and node modules.
Considering NodeJS as an independent component in a Roboconf graph does not make sense since it is a library. Node being started is a non-sense.
Instead, it is better to only consider Node modules or Node applications as a whole with NodeJS. Therefore, it would be very convenient for a user to have a Node plug-in.
This plug-in would be in charge of:
http://www.rabbitmq.com/access-control.html
By default, the guest user is prohibited from connecting to the broker remotely; it can only connect over a loopback interface (i.e. localhost
).
Remote login with guest user work without configuration with RabbitMQ 2.8.4.
We can undeploy without stop instance of an application.
Many POST operations use anonymous parameters and instance paths.
Jersey's FormParam may be the solution to use named parameters.
In this case, instance paths and anonymous parameters should have a name.
This is a concurrency problem.
Short-spanned requests are processed concurrently.
Roboconf has seen an important development effort these last months, which includes the addition of unit tests. However, code coverage is a little bit under 50%. And many classes could easily be tested.
An effort should be done on it.
LXC (https://linuxcontainers.org/) enables to isolate processes in a Linux physical or virtual machines.
Add the support to create LXC containers in a IaaS VM in order to build PaaS (such as Heroku) for hosting services written in Ruby, Node.js, Clojure, Java, Python, Scala, Erlang ...
Today, if the DM shutdowns brutally, there is now way to reattach it to existing instances (VMs and applications).
The DM should store the list of applications.
It should also store the IP of the messaging server.
Eventually, it should be able to notify agents it needs to restore the model's state.
A new REST operation will be added to start the restoration process.
Configuration files should contain something that helps parsers to identify which version of the model it corresponds to. This will be very useful if we add new instructions later.
It will help tools and the platform to determine whether they support these configuration files.
This only impacts the parsing. The runtime model does not need to be versioned. It will be upgraded naturally. When an incompatibility appears, former parsing models will be removed from the platform and will be marked as not supported. A MANUAL migration of the configuration files will be necessary then.
If an automatic migration si possible, then the parsing and runtime models are compatible.
This transformation will be performed automatically by the parser.
Add the support to create Docker containers
https://www.docker.io/
Add a new plugin (net.roboconf.plugin.api.PluginInterface) for Apache Ant's build.xml scripts.
Scripts should have targets with names corresponding to the lifecycle (start, stop, ...)
The Ant extensions and the library dependencies may be taken into account (Apache Ivy, SCP, ....) : see http://ant.apache.org/manual/install.html#librarydependencies
The version of the Ant runtime may be taken into account
Instances manage a map associating variable groups (component or facet names) and imports. An import is a structure the instance can use to know which other instances export this or that variable.
This import structure should be updated to store the real instance(s) that is or are used by the instance. As an example, a Tomcat instance may have knowledge about several MySQL instances. However, it will most likely use only one. The model should keep track of this.
If an instance uses several instances for a same set of variables (think to a load balancer), then they should all be listed.
Roboconf's source code was migrated to GitHub.
However, use cases and demos cannot be migrated yet. This is because these samples embed binaries. We need a Maven plug-in so that we can commit them in Git repositories.
The plug-in will be in charge of gathering remote resources (including binaries) and of creating an archive Roboconf can handle.
This plug-in should support two project types:
Three goals are required.
This last goal should propose something similar to the Maven assembly plug-in, but more flexible. As an example, it should be able to download artifacts from a web server, from a Maven repository, from various URL schemes, etc.
isInitialized()
returns true or false.
It should be replaced by...
getServerInfo()
... which should include the DM's version and the IP address of the messaging server.
However, this points out a new issue, related to security. Is it a good idea to expose the location of the messaging server through a REST API? Anyone could get it. And due to our use of RabbitMQ, anyone could send or connect to RabbitMQ.
So...
First, we could add credentials on the messaging server. But this is quite heavy since all the agents should have them too. Or, we could consider the location of the messaging server could be set in a configuration file (or in the parameters of the web application). The location of RabbitMQ would not be public anymore. It could only be set through configuration files or through the Java singleton Manager (for cases where the DM is run programmatically). No need of REST for this.
The second solution is much better from a security point of view.
The VM will be deployed, but not its children.
In fact, the DM will send messages to the agent. But since the agent is not yet started (the VM takes time to be instantiated), RabbitMQ will drop the messages (it is configured to drop messages when there is no consumer).
The DM should store these messages until the VM is created.
When Roboconf creates a VM on OpenStack it seems that the Up notification does not contain the IP address.
By using
from the created VM, we see that there is no information at this location.
The DM does not have to ask the agent if it can delete it.
This depends on the IaaS type. For a real IaaS, we do not need to ask. For an in-memory agent or a device, we can proceed differently.
The logs time and date are not enough precise.
It should be improved.
The DM is using IaaS implementations to create virtual machines.
However, IaaS client libraries have conflicting dependencies which make the Maven build fail. Besides, everything is loaded at once in the class loader.
We need to isolate class loaders in the DM.
Since some Roboconf modules have already been bundlized, we should complete this movement. The Iaas implementation should only export the implementation of the IaaS API and keep other packages (including their dependencies) private.
The web app packaging for the DM should be repalced by a Karaf distribution, with loggers configuration. A solution will have to be found also to deploy Jersey into this distribution.
A nesw keyword should be added in the model.
It would allow to create several instances at once.
There will be no base name.
It will take the name and append it a number.
instanceof Toto {
name: toto;
numInstances: 3;
}
... will create toto-1, toto-2 and toto-3.
If users want a nicer name, they will have to use the REST API and a program to do it.
During the migration to GitHub, instance channels have been disabled.
They should be reintroduced.
Instance channels allow to define a better granularity in exchanges between agents. They act as a filter on the message server.
Like the DM, the agents should move to OSGi.
This is less critical than the DM, but more easy to do. This will solve a lot of problems for plug-ins and the extensibility of the platform.
Like the DM:
Currently, these files will result in a parsing error because the parser does not know whether they are instances or a graph portion. Empty files should be ignored for instances. This could be achieved with a parser option.
Dependeing on the plug-in they use, agents can...
There should be some configuration so that we can indicate to the plug-in how it must react to an import change. This must be a user configuration. So, it should appear somewhere in the graph (definition or resources). Notice that a plug-in implementation may ignore it.
Here a graph example :
Apache {
alias: Apache Load Balancer;
installer: puppet;
imports: RSE.ip, RSE.port;
}
RSE {
alias: RSE with mongo auto-configuration;
installer: puppet;
exports: repopath=/home/rse/rse, giturl=http://ci-openpaas.linagora.com/stash/scm/or/rse.git, revision=master, ip, port=8080;
imports: Mongo.ip, Mongo.port, Mongo.db;
}
In the Apache component we want only the IP and the port of the RSE component. But in puppet script, the $rse variable contains all the exports variables of RSE whereas we want only the IP and the port.
When a runtime model error is found while loading a new application, no line number is given. All the other error types (parsing, conversion, etc) have one.
This should be fixed as it could seriously help a user when he gets such an error.
There is currently no formal validation about the existence and the presence of the right plug-in for a given installer name. This should be performed either on the DM, or on the agents.
.properties files must be encode with ISO 8859-1 (latin1) but UTF-8 is better for international project.
I recommend to use other configuration file such as JSON, YAML or XML.
When we deploy an application with Roboconf on a Iaas, the VM creation takes time. It seems RabbitMQ does not store the message if nobody is listening. As a result, the agent does not receive its model (including the root instance it is associated with) and a NPE is thrown.
Add a IaaS implementation for VMware.
This packaging would be a WAR archive with both the DM and the web administration.
For the moment, the virtual image used on a IaaS must contain the agent.
This is very annoying and painful when we test the agent. Everytime it is modified, a new virtual image must be created.
There should be a feature on the DM to upload the agent.
This may remain a simple method in Manager we could only use for tests. The solution would be to reference an agent's archive file the DM will upload by SSH. Once uploaded, the DM will unzip it and start it. Until the VM is running, the DM would poll regularly by SSH, until it works or a certain number of attempts has been reached.
A main problem that remains is about how the DM will set the "root" permissions to the agent. Indeed, the agent should be added to init.d so that it gets root permissions. Or it should be started in "sudo" mode to remain simple. The init.d would then be a solution for productions.
The main advantage of this solution is to have a simple and final virtual image / AMI. It would contain a JVM and Puppet. And we could easily work on real IaaS with snapshot agents.
Seen on an example.
A VM component with no children...
VM {
alias: VM;
installer: iaas;
}
... and instances with Software installed on VM instances.
The instances definition should have resulted in an error when the validation was loaded. The problem was identified because we could not add new instances at runtime.
Instances hierarchy should be checked against the graph.
For the moment, there is message to configure the firewall on the agent's machine.
However, it is not really used.
For every node of the graph, there should be a script that is used to configure the firewall. This script is optional (things like Puppet may configure it automatically). When an instance starts, an agent plug-in would first check if there is a firewall script and execute it.
This would reinforce the security of a Roboconf deployment.
For the moment, the client for RabbitMQ only uses the message server location but not the port.
If RabbitMQ is using a specific port and that you set the message server location to...
... then the port is ignored. The messaging module should parse this URL and configure the port if necessary.
Instance1 depends on Instance2 (= imports variables from Instance2), and everything is up and running (state = DEPLOYED_STARTED for both instances).
If we stop Instance2, Instance1 should move back to INSTALLING state : it does not, although it properly receives configuration information from Instance2.
This looks like a regression (has been working fine in the past).
if( this.messageProcessor != null
&& this.messageProcessor.isAlive())
this.messageProcessor.interrupt();
Replace isAlive by isRunning.
This is located in the clients (messaging modules).
Deploying an application before having deployed the underlying VM causes MQ corruption.
Add a IaaS implementation for OpenStack.
Add a IaaS implementation for Microsoft Azure.
Currently, quality rules are defined in the parent of the Roboconf repository.
This is wrong because the packaging for the parent is "pom". It means quality rules cannot be reused outside this repository.
Besides, these rules should also apply to other Roboconf projects (web administration, Eclipse tools, etc).
Quality rules (checkstyle, findbugs, jacoco) should be moved to another project in its own repository (roboconf-quality). It will be referenced by all the other Roboconf projects.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.