e7dal / bubble3 Goto Github PK

add improve progress bars
https://pypi.python.org/pypi/tqdm
https://pypi.python.org/pypi/progress

new sub command, that will show the current status/last status/current processing

remove dummy client which installs into bubble.clients....
-.travis
-behave feature

https://github.com/e7dal/bubble3/blob/master/features/transform.feature
https://github.com/e7dal/bubble3/blob/master/bubble/util/examples.py#L58

think of a good use case and add feature

create dockerfile
and docker compose for "demos"

We found potential security vulnerabilities in your dependencies.

Some of the dependencies defined in optional_requirements.txt, requirements_plus_dataset.txt, and travis_requirements.txt have known security vulnerabilities and should be updated.

Dependencies defined in requirements_plus_dataset.txt 16
An exploitable vulnerability exists in the Databook loading functionality of Tablib 0.11.4. A yaml loaded Databook ca...
requirements_plus_dataset.txt update suggested:
tablib ~> 0.11.5

https://nvd.nist.gov/vuln/detail/CVE-2017-2810

split bubble3 into:
bubble3(basic package) and bubble3-dev(testing and extras)

create an api that can be run with "bubble listen"

api should mostly reflect the cli
-apitest
-bottle / hug

in this example from: https://github.com/e7dal/bubble3_clients/tree/master/knmi_warnings

published_parsed should become
published_parsed.1
published_parsed.2
published_parsed.3
etc

introspection of pulled data looks like this:
In [10]: d['published_parsed']
Out[10]: time.struct_time(tm_year=2018, tm_mon=9, tm_mday=4, tm_hour=7, tm_min=6, tm_sec=0, tm_wday=1, tm_yday=247, tm_isdst=0)

jq . remember/pulled_DEV.jsonl
{
"title": "Code geel: Vanmiddag in het midden en zuiden van het land onweersbuien",
"title_detail.type": "text/plain",
"title_detail.language": null,
"title_detail.base": "http://projects.knmi.nl/RSSread/rss_KNMIwaarschuwingen.php",
"title_detail.value": "Code geel: Vanmiddag in het midden en zuiden van het land onweersbuien",
"summary": "

Vanaf halverwege de middag is er in het midden en zuiden van het land kans op onweersbuien met daarbij lokaal veel neerslag in korte tijd.

<img src="https://cdn.knmi.nl/knmi/map/current/weather/warning/waarschuwing_land_0_new.gif\" width=300> <img src="https://cdn.knmi.nl/knmi/map/current/weather/warning/waarschuwing_land_1_new.gif\" width=300> <img src="https://cdn.knmi.nl/knmi/map/current/weather/warning/waarschuwing_land_2_new.gif\" width=300>

Vanaf halverwege de middag is er in het midden en zuiden van het land kans op onweersbuien met daarbij lokaal veel neerslag in korte tijd.

<img src="https://cdn.knmi.nl/knmi/map/current/weather/warning/waarschuwing_land_0_new.gif\" width=300> <img src="https://cdn.knmi.nl/knmi/map/current/weather/warning/waarschuwing_land_1_new.gif\" width=300> <img src="https://cdn.knmi.nl/knmi/map/current/weather/warning/waarschuwing_land_2_new.gif\" width=300>

root@1c5bc8d0c787:/opt/bubble_test/ndw# bubble -v 10000 -b 10 pull
BubbleCli:here!
BubbleCli:config
stuff:
{'bubble_lib_dir': '/usr/local/lib/python3.6/site-packages/bubble3',
'logfile': '',
'parent': None,
'verbose': 10000,
'verbose_bar': 10}
Hi there,
i'm a inside a Bubble,
my name is BubbleCli.

a.k.a: API in the middle

Helps you where two of your "complex" services don't play well with each other.
And where you need get some information from A to B,
while transforming or filtering the information, simply put:

A:pull>>>bubble:transform>>>B:push>>>
consider me your "small information" manager in between your big data and ESB(s),ETL(s).

Create a bubble, make the information flow and start bubbling .o.O.o.

Traceback (most recent call last):
File "/usr/local/bin/bubble", line 11, in
sys.exit(cli())
File "/usr/local/lib/python3.6/site-packages/click/core.py", line 722, in call
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1063, in invoke
Command.invoke(self, ctx)
File "/usr/local/lib/python3.6/site-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.6/site-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/bubble3/cli.py", line 350, in cli
verbose_bar=barverbose)
File "/usr/local/lib/python3.6/site-packages/bubble3/cli.py", line 102, in init
verbosity=2)
File "/usr/local/lib/python3.6/site-packages/bubble3/init.py", line 272, in say
self._msg(msg=msg, stuff=stuff, verb='SAY', verbosity=verbosity)
File "/usr/local/lib/python3.6/site-packages/bubble3/init.py", line 255, in _msg
blog.info(**tl_item)
File "/usr/local/lib/python3.6/site-packages/structlog/_base.py", line 176, in _proxy_to_logger
args, kw = self._process_event(method_name, event, event_kw)
File "/usr/local/lib/python3.6/site-packages/structlog/_base.py", line 136, in _process_event
event_dict = proc(self._logger, method_name, event_dict)
File "/usr/local/lib/python3.6/site-packages/structlog/dev.py", line 179, in call
event = event_dict.pop("event")
KeyError: 'event'
exit
stuff:
{'config_yaml': 'config/config.yaml',
'dot_bubble': '.bubble',
'gbc': <bubble3.Bubble object at 0x7f3795709e10>,
'profiling': False,
'stages': ['PROD', 'ACC', 'TEST', 'DEV', 'MOCK'],
'start_arrow': <Arrow [2018-06-04T06:50:18.741817+00:00]>,
'steps': ['pulled',
'uniq_pull',
'uniq_push',
'push',
'pushed',
'store',
'stats'],
'storage_types': ['json', 'jsonl']}

use https://pandas.pydata.org/pandas-docs/stable/io.html for export

make more system rule functions, like in:
https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#structure