met4fof / agentmet4fof Goto Github PK

We should check if by now we could drop the constraint on matplotlib==3.1.0 not to loose compatibility with newer Python versions.

@bangxiangyong could you provide a test, which checks for the reason of the constraint so we can test if an upgrade would harm the agents?

We might implement a class for or at least deeply integrate the D-SI into the message passing of the agents. This should be considered closely related to #19 .

Further materials about D-SI will be added here shortly.

At hte moment we have to write restructured text to work on the docs. Since we are so much more used to markdown, it would be nice to introduce recommonmark into our pipeline.

We tried that in #50 and found it very promising. The next extractions might be:

• ml_uncertainty
• demo
• ML
• ML_EXPERIMENTS
• SENSORS
• ZEMA_EMC
• ... whatever could be moved from develop.

After that we can clean up the docs and requirements.txt and win

• significantly increased overview
• probably higher coverage
• a much clearer structure
• more presentable, accessible results, possibly with own DOIs and future citations.

Considerations:

1. Zooming/Panning/etc. on an interactive
2. A clear/standardised architecture
3. Types of plots
4. Supporting the web visualisation

We rather want to use the methods Haris already prepared in his subfolder methods by transferring our methods in zema_feature_extract.py and then reusing them by introducing his code as a submodule in ZEMA_EMC.

A separate tutorial on this will be helpful to illustrate how the data memory is used.

Had a meeting with Majid, and realised that clearing the garbage is a better way than simply setting the memory to an empty dictionary.

reference:
https://github.com/bangxiangyong/agentMET4FOF/blob/develop/examples/SENSORS/buffer_sensor.py

Will need to investigate whether which option is better and feasible:

1. Replacing underlying communication of Pyro4 with OPC-UA
2. Providing an external interface to OPC-UA

When I execute pythonrun_dashboard.py in the console I get the following output periodically:

Unable to connect to existing NameServer...
 * Serving Flask app "dashboard.Dashboard" (lazy loading)
 * Environment: production
   WARNING: Do not use the development server in a production environment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on http://127.0.0.1:8050/ (Press CTRL+C to quit)
127.0.0.1 - - [30/Jul/2019 17:24:32] "GET /_favicon.ico HTTP/1.1" 200 -
127.0.0.1 - - [30/Jul/2019 17:24:33] "GET /_dash-layout HTTP/1.1" 200 -
127.0.0.1 - - [30/Jul/2019 17:24:33] "GET /_dash-dependencies HTTP/1.1" 200 -
127.0.0.1 - - [30/Jul/2019 17:24:33] "POST /_dash-update-component HTTP/1.1" 200 -
127.0.0.1 - - [30/Jul/2019 17:24:33] "POST /_dash-update-component HTTP/1.1" 204 -
127.0.0.1 - - [30/Jul/2019 17:24:33] "POST /_dash-update-component HTTP/1.1" 204 -
127.0.0.1 - - [30/Jul/2019 17:24:33] "POST /_dash-update-component HTTP/1.1" 204 -
[2019-07-30 17:24:33,106] ERROR in app: Exception on /_dash-update-component [POST]
Traceback (most recent call last):
  File "PATH\TO\MY\ENV\lib\site-packages\flask\app.py", line 2292, in wsgi_app
    response = self.full_dispatch_request()
  File "PATH\TO\MY\ENV\lib\site-packages\flask\app.py", line 1815, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "PATH\TO\MY\ENV\lib\site-packages\flask\app.py", line 1718, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "PATH\TO\MY\ENV\lib\site-packages\flask\_compat.py", line 35, in reraise
    raise value
  File "PATH\TO\MY\ENV\lib\site-packages\flask\app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "PATH\TO\MY\ENV\lib\site-packages\flask\app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "PATH\TO\MY\ENV\lib\site-packages\dash\dash.py", line 1147, in dispatch
    response.set_data(self.callback_map[output]['callback'](*args))
  File "PATH\TO\MY\ENV\lib\site-packages\dash\dash.py", line 1033, in add_context
    output_value = func(*args, **kwargs)
  File "PATH\TO\MY\REPOS\Met4FoF_Code\agentMet4FoF\dashboard\Dashboard.py", line 423, in plot_monitor_graphs
    if agentNetwork._get_mode() != "Running" and n_interval > 0:
  File "PATH\TO\MY\REPOS\Met4FoF_Code\agentMet4FoF\agents\AgentMET4FOF.py", line 653, in _get_mode
    return self._get_controller().get_attr('current_state')
  File "PATH\TO\MY\REPOS\Met4FoF_Code\agentMet4FoF\agents\AgentMET4FOF.py", line 789, in _get_controller
    self._controller = self.ns.proxy('AgentController')
AttributeError: 'int' object has no attribute 'proxy'
127.0.0.1 - - [30/Jul/2019 17:24:33] "POST /_dash-update-component HTTP/1.1" 500 -

Do we still need this script? If so we should update it.

This leads to a more detailed description of #22.

This could be incorporating MLFlow, where the pipelines (#17) and parameters can be easily varied to compare their performances and logged systematically.

Incorporate the methods for accounting uncertainty developed by Ehlimana into the agents.

This would allow for the extracted examples to work without any further tweaking of the execution environment.

Been having this in mind of viewing data analytics/ML as a workflow (which current implementation for ZEMA EMC with Haris Lulic is kind-of adopting already), that is a sequence of functions applied one after the other.

sklearn reference

Example:
group_of_agents = make_pipeline(FFT(), PCA(pca_parameters), LDA(lda_parameters))

• The arguments are any number of objects which implements the function fit and transform similar to how sklearn structures its classes
• The line above should instantiate 3 agents connected from left to right (FFT -> PCA -> LDA)
• Further, for this to work, a generic ML agent class which accepts these data analytics methods should be developed

The advantage with the agent network architecture:

1. Pipelines which use similar components/agents will only need to process once if the agent connections are made dynamic. The code below should create 4 agents (FFT -> PCA -> LDA & ANN) instead of 6 agents (sum of number of agents in both pipelines)
   Example:

group_of_agents_1 = make_pipeline(FFT(), PCA(pca_parameters), LDA()) 
group_of_agents_2 = make_pipeline(FFT(), PCA(pca_parameters), ANN()) 

2. Entire data processing pipelines can be visualized via dashboard immediately
3. Compatible immediately with sklearn and more fluid integration with data science
4. Promotes incremental development for mathematical components. Such as when investigating the effect of added noise/bias, we can have two pipelines to be compared:

group_of_agents_1 = make_pipeline(AddNoise(), FFT(), LDA()) 
group_of_agents_2 = make_pipeline(FFT(),  LDA()) 

We should gather with @mgrub, @BeSeeTek, @bangxiangyong and @BjoernLudwigPTB for a web conference with whiteboards in the background to make up our minds about how to structure #23 .

It would be nice, if the function create_monitor_data (used here and defined here) can be extended to allow for user-provided x values.

This would help with plotting infinite streams of time-series.

The new function could look something like this: (old behaviour is obtainable by explicitly setting xname or yname to None)

def create_monitor_data(data, xname="time", yname="data", label="Monitor Data"):
    if xname and yname:
        x = data[xname]
        y = data[yname]
    else:
        x = np.arange(len(data))
        y = data

    trace = go.Scatter(x=x, y=y,mode="lines", name=label)
    return trace

As described in #21

from agentMET4FOF.agents import AgentMET4FOF, AgentNetwork, MonitorAgent
from agentMET4FOF.streams import SineGenerator

import torch.nn as nn
import torch.nn.functional as F


class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 16 * 5 * 5)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x


class TrainerAgent(AgentMET4FOF):
    def init_parameters(self,minus_param=0.5,plus_param=0.5):
        self.net = Net()
        self.first_time = True

    def on_received_message(self, message):
        if self.first_time:
            self.send_output(self.net)
            self.first_time = False

class SineGeneratorAgent(AgentMET4FOF):
    def init_parameters(self):
        self.stream = SineGenerator()

    def agent_loop(self):
        if self.current_state == "Running":
            sine_data = self.stream.next_sample() #dictionary
            self.send_output(sine_data['x'])

def main():
    # start agent network server
    agentNetwork = AgentNetwork()
    # init agents
    gen_agent = agentNetwork.add_agent(agentType=SineGeneratorAgent)
    train_agent = agentNetwork.add_agent(agentType=TrainerAgent)
    predict_agent = agentNetwork.add_agent(agentType=TrainerAgent)
    monitor_agent = agentNetwork.add_agent(agentType=MonitorAgent)
    # connect agents : We can connect multiple agents to any particular agent
    agentNetwork.bind_agents(gen_agent, train_agent)
    agentNetwork.bind_agents(gen_agent, predict_agent)
    # connect
    agentNetwork.bind_agents(gen_agent, monitor_agent)
    agentNetwork.bind_agents(train_agent, predict_agent)
    agentNetwork.bind_agents(predict_agent, monitor_agent)
    # set all agents states to "Running"
    agentNetwork.set_running_state()

    # allow for shutting down the network after execution
    return agentNetwork


if __name__ == '__main__':
    main()

Error message as below:

Exception in thread oneway-call:
Traceback (most recent call last):
File "/usr/lib/python3.7/threading.py", line 917, in _bootstrap_inner
self.run()
File "/home/bangxiang93/PycharmProjects/agentMET4FOF_fresh_test/venv/lib/python3.7/site-packages/Pyro4/core.py", line 1891, in run
super(_OnewayCallThread, self).run()
File "/usr/lib/python3.7/threading.py", line 865, in run
self._target(*self._args, **self._kwargs)
File "/home/bangxiang93/PycharmProjects/agentMET4FOF_fresh_test/venv/lib/python3.7/site-packages/osbrain/agent.py", line 1680, in run
self._loop()
File "/home/bangxiang93/PycharmProjects/agentMET4FOF_fresh_test/venv/lib/python3.7/site-packages/osbrain/agent.py", line 1208, in _loop
if self._iterate():
File "/home/bangxiang93/PycharmProjects/agentMET4FOF_fresh_test/venv/lib/python3.7/site-packages/osbrain/agent.py", line 1245, in _iterate
self._process_events(events)
File "/home/bangxiang93/PycharmProjects/agentMET4FOF_fresh_test/venv/lib/python3.7/site-packages/osbrain/agent.py", line 1282, in _process_events
self._process_single_event(socket)
File "/home/bangxiang93/PycharmProjects/agentMET4FOF_fresh_test/venv/lib/python3.7/site-packages/osbrain/agent.py", line 1296, in _process_single_event
self._process_sub_event(socket, address, data)
File "/home/bangxiang93/PycharmProjects/agentMET4FOF_fresh_test/venv/lib/python3.7/site-packages/osbrain/agent.py", line 1452, in _process_sub_event
handler(self, message)
File "/home/bangxiang93/PycharmProjects/agentMET4FOF_fresh_test/agentMET4FOF-develop/agentMET4FOF/agents.py", line 334, in handle_process_data
proc_msg = self.on_received_message(message)
File "/home/bangxiang93/PycharmProjects/agentMET4FOF_fresh_test/agentMET4FOF-develop/agentMET4FOF_tutorials/tutorial_pytorch.py", line 35, in on_received_message
self.send_output(self.net)
File "/home/bangxiang93/PycharmProjects/agentMET4FOF_fresh_test/agentMET4FOF-develop/agentMET4FOF/agents.py", line 263, in send_output
self.send(self.PubAddr, packed_data, topic='data')
File "/home/bangxiang93/PycharmProjects/agentMET4FOF_fresh_test/venv/lib/python3.7/site-packages/osbrain/agent.py", line 1493, in send
topic=topic)
File "/home/bangxiang93/PycharmProjects/agentMET4FOF_fresh_test/venv/lib/python3.7/site-packages/osbrain/agent.py", line 1501, in _send_address
serializer=address.serializer)
File "/home/bangxiang93/PycharmProjects/agentMET4FOF_fresh_test/venv/lib/python3.7/site-packages/osbrain/agent.py", line 85, in serialize_message
return _serialize_callsserializer
File "/home/bangxiang93/PycharmProjects/agentMET4FOF_fresh_test/venv/lib/python3.7/site-packages/osbrain/agent.py", line 50, in
'pickle': lambda message: pickle.dumps(message, -1),
_pickle.PicklingError: Can't pickle <class 'main.Net'>: it's not the same object as main.Net

If it is possible to easily extract the #32 related changes then we open a self-contained Pull Request to make it easier for Max to review, if his needs are met.

Because of the subprocesses we are spawning, in general debugging does not work as usual. We should think about, if it is possible to make that available. It would strip a lot of print statements from the codes we are seeing being written using the agents and make debugging a lot easier.

This will automatically resolve #32 by the proposed overwriting mechanism.

The documentation of components for ML experiments is not complete, and the structure is quite scattered (agents.py , develop.* , etc).

The clarity can be improved by having more explanations and should probably be self-contained as a separate component?

We want to extract the more specific project related examples into separate repositories which are related to agentMET4FOF by the submodule mechanism.

There is some stuff missing and some things we don't need anymore.

i have buffered data and send to agent, but i want to after each new sending, perevious data romves , i mean remove last buffer and replace with new data.

This partly resolves #36 .

We should provide a PyPI.org package and upload a new release whenever something worth mentioning changes in agentMET4FOF. At the same time we should consider Python Semantic Release for automatic semantic versioning.

After setting up a clean python-venv and then pip install agentMET4FOF, execution of import agentMET4FOF.agents yields:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "%userprofile%\python_venv\agentmet4fof\lib\site-packages\agentMET4FOF\agents.py", line 24, in <module>
    from agentMET4FOF.dashboard.Dashboard import AgentDashboard as AgentDashboard        
  File "%userprofile%\python_venv\agentmet4fof\lib\site-packages\agentMET4FOF\dashboard\Dashboard.py", line 6, in <module>
    from agentMET4FOF.dashboard.Dashboard_ml_exp import get_ml_exp_layout, prepare_ml_exp_callbacks, get_experiments_list
  File "%userprofile%\python_venv\agentmet4fof\lib\site-packages\agentMET4FOF\dashboard\Dashboard_ml_exp.py", line 15, in <module>
    from agentMET4FOF.develop.ML_Experiment import load_experiment
ModuleNotFoundError: No module named 'agentMET4FOF.develop'

Details about the system:
OS: Windows 10 Pro 64bit
Python: 3.7.7 and 3.8.1
agentMET4FOF: 0.1.4

Current Workaround:
Use agentMET4FOF from a clone of the github-repo:

• clone the repo to <directory_of_your_choice>
• set up a python-venv with all requirements for agentMET4FOF
• add <directory_of_your_choice> to the PYTHONPATH (either by setting a system variable or by using import sys;sys.path.append("<directory_of_your_choice>")

ZeMA (@TanjaDorst) should use agentMET4FOF together with their machine learning methods with uncertainties and we should support their efforts (e.g. refactor interfaces if this could encourage others to work with agentMET4FOF or integrate required functions)

In #28 we wrote down some links and now have to find out, how to make use of D-SI.

The following classes need class docstrings:

• TransformerAgent
• _Logger
• AgentPipeline

Benedikt left me with a sensor board, to be connected to the agent framework.

SemVer appears to be very beneficial and time saving, if it is realized with python-semantic-release. It definitely enhances our release process by providing changelogs.

There seems to be a problem with running the agents on Windows with Python 3.8.1 and multiprocess==0.70.9. A colleague gets the following error on execution of tutorial 1:

Starting NameServer...
Broadcast server running on 0.0.0.0:9091
NS running on 127.0.0.1:3333 (127.0.0.1) 
URI = PYRO:[email protected]:3333
INFO [2020-04-15 10:11:08.404372] (AgentController): INITIALIZED
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'multiprocess'
INFO [2020-04-15 10:11:30.828139] (SineGeneratorAgent_1): INITIALIZED
INFO [2020-04-15 10:11:39.270462] (MonitorAgent_1): INITIALIZED
[2020-04-15 10:11:39.358227] (SineGeneratorAgent_1): Connected output module: MonitorAgent_1
SET STATE:   Running
[2020-04-15 10:11:39.868856] (SineGeneratorAgent_1): Pack time: 0.0
[2020-04-15 10:11:39.869853] (SineGeneratorAgent_1): Sending: [0.]
[2020-04-15 10:11:39.869853] (MonitorAgent_1): Received: {'from': 'SineGeneratorAgent_1', 'data': array([0.]), 'senderType': 'SineGeneratorAgent', 'channel': 'default'}
[2020-04-15 10:11:39.869853] (MonitorAgent_1): Tproc: 0.0

On the same machine it is working with Python 3.7.7 and multiprocess==0.70.7. We haven't checked the multiprocess issues yet and he will continue with the working 3.7.7 configuration, so for now there is no further investigation.

The tutorials are already well documented, but we could add more comments in the code. This could help us to clarify the process and make it even easier to get started.

We are just implementing the buffering mechanism in #40 and should straight away find out, if we can meet the 10 kHz desirable PTB (the one mentioned by @BeSeeTek) sampling rate for dealing with hardware sensor's.

To make working with the agents easier, we should be able to clarify on how input data should be structured to best fit the agents design.

Regarding streaming data, we should as well keep in mind to adapt the generators of PyDynamic to this design and view this as very closely connected.

That instantly leads to an extended development of the signals themselves probably, which benefits PyDynamic as well, maybe in extending the previously started class Signal. Maybe we can even base the development on that class.

Originally posted by @BjoernLudwigPTB in #18 (comment)

• We have this partially with the ZEMA EMC examples already
• Need to clean up and bring it into the latest ML Experiment form

A full-text search for the filename of assets/stylesheet.css in the project gave no results. After deleting it, all tests and examples still succeeded. Can that file maybe be deleted?

... which includes adapting the tutorial about Haris' work as well. This is the start of #27.

The send_plot funciton converts matplotlib figures into plotly or image base64 string, there is an option to directly send plotly figures as well, but we should test and implement it further.

@mgrub mentioned, that it took him some time to see the differences but thinks, the structure is useful and should be kept.

Since we want to extract the examples anyway (see #50 for a start) we should remove the individual requirements from the global requirements.txt since there are errors arising on some platforms resulting from different or missing C++ compiler versions.

Max has got some implementation ready using a Python's deque und we got some in the examples. We should merge these into one residing in the master.

At the moment the two links targets on the docs startpage
agentMET4FOF agents
agentMET4FOF streams
are empty. We have to check the according rst-files and repair this. The autodocs were their earlier.