Coder Social home page Coder Social logo

brewtils's Introduction

Brewtils

Brewtils is the Python library for interfacing with Beergarden systems. If you are planning on writing beer-garden plugins, this is the correct library for you. In addition to writing plugins, it provides simple ways to query the API and is officially supported by the beer-garden team.

Gitter PyPI Build Status Code Coverage Documentation Status Pyup Updates

Features

Brewtils helps you interact with beer-garden.

  • Easy way to create beer-garden plugins
  • Full support of the entire Beer-Garden API
  • Officially supported by the beer-garden team

Installation

To install brewtils, run this command in your terminal:

$ pip install brewtils

Or add it to your requirements.txt

$ cat brewtils >> requirements.txt
$ pip install -r requirements.txt

Quick Start

You can create your own beer-garden plugins without much problem at all. To start, we'll create the obligatory hello-world plugin. Creating a plugin is as simple as:

from brewtils.decorators import system, parameter, command
from brewtils.plugin import RemotePlugin

@system
class HelloWorld(object):

    @parameter(key="message", description="The message to echo", type="String")
    def say_hello(self, message="World!"):
        print("Hello, %s!" % message)
        return "Hello, %s!" % message

if __name__ == "__main__":
    client = HelloWorld()
    plugin = RemotePlugin(client,
                          name="hello",
                          version="0.0.1",
                          bg_host='127.0.0.1',
                          bg_port=2337)
    plugin.run()

Assuming you have a Beer Garden running on port 2337 on localhost, running this will register and start your plugin! You now have your first plugin running in beer-garden. Let's use another part of the brewtils library to exercise your plugin from python.

The SystemClient is designed to help you interact with registered Systems as if they were native Python objects.

from brewtils.rest.system_client import SystemClient

hello_client = SystemClient('localhost', 2337, 'hello')

request = hello_client.say_hello(message="from system client")

print(request.status) # 'SUCCESS'
print(request.output) # Hello, from system client!

In the background, the SystemClient has executed an HTTP POST with the payload required to get beer-garden to execute your command. The SystemClient is how most people interact with beer-garden when they are in the context of python and want to be making requests.

Of course, the rest of the API is accessible through the brewtils package. The EasyClient provides simple convenient methods to call the API and auto-serialize the responses. Suppose you want to get a list of all the commands on all systems:

from brewtils.rest.easy_client import EasyClient

client = EasyClient('localhost', 2337)

systems = client.find_systems()

for system in systems:
    for command in system.commands:
        print(command.name)

This is just a small taste of what is possible with the EasyClient. Feel free to explore all the methods that are exposed.

For more detailed information and better walkthroughs, checkout the full documentation!

Documentation

brewtils's People

Contributors

catattack05 avatar friendship-is-coding avatar hazmat345 avatar jake-ross avatar jlrpnbbngtn avatar loganasherjones avatar pyup-bot avatar sataubman avatar scott-taubman avatar specificallygravity avatar theburchlog avatar turettn avatar west270 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

brewtils's Issues

Marshmallow Warning

The newest version of Marshmallow does not seem to have any breaking API changes for us, but it does throw a warning:

ChangedInMarshmallow3Warning: The json_module option is renamed to render_module in marshmallow 3. ChangedInMarshmallow3Warning

Support optional parameters of type "Bytes"

There does not currently appear to be a good way to have optional parameter of type "Bytes". Optional parameters require either:

  • A defined default
  • nullable to be set to True

Defaults don't really make sense in the context of a "Bytes" parameter, and setting nullable=True currently results in an error getting thrown before the plugin function is actually reached.

We should fix the handler code so that a "Bytes" parameter that is provided with a value of null is properly handled and sent to the plugin function as None, where it can handle that input accordingly.

extras require is broken

Our extras_require won't allow us to install from git anymore. I get the error message:

error in brewtils setup command: 'extras_require' must be a dictionary whose values are strings or lists of strings containing valid project/version requirement specifiers.

Plugin max_concurrent default value

Currently the default behavior is for plugins to have a max_concurrent of one. I think we did this for backwards-compatibility reasons?

I think we should bump the default to something higher (Python seems to like # of processors * 5) and let any plugin developers that want the max_concurrent=1 behavior to say that explicitly.

Add `_raise` flag to SystemClient

When calling in the context of another plugin, it is often the case that you will have to write the following code:

request = self.system_client.do_something()
if request.status == "ERROR":
    raise Exception(request.output)

We should add a flag to SystemClient so people don't have to write this logic over and over.

self.system_client.do_something(_raise=True, _exc_class=Exception)

RestClient requires username and password for certificate authentication

RestClient currently checks that username and password are both set before attempting to authenticate to a garden. This means that even if certificate authentication is enabled, some value must be provided for the username and password in RestClient even if certificates are what will be used for the authentication. This should be corrected so that if certificates are provided, username and password are no longer required.

Pika version upgrade

Pika (the RabbitMQ library) went 1.0 in March. This involved a bunch of syntax changes for things like keyword argument names.

This means that any modules that use pika separately to talk to their own RabbitMQ instance start hitting version conflicts that can not easily resolved with containers or virtual environments.

As pika continues to progress forwards, this means that brewtils will slide further and further behind the curve.

Default Timeout for RestClient

Requests recommends that production code specify a timeout for basically every request. We need to do this to prevent possible hanging when the plugins attempt to talk to Beergarden.

Brewtils unit tests for file related functionality

Of the areas that lack test coverage in brewtils, functionality related to files seems to be in the worst shape:

coverage run -m py.test && coverage report -m --include brewtils/schemas.py,brewtils/rest/easy_client.p

<...snip...>

Name                           Stmts   Miss  Cover   Missing
------------------------------------------------------------
brewtils/models.py               652     64    90%   439-449, 452, 455, 473-477, 480, 483, 513-534, 537, 540, 860, 1125-1128, 1131, 1134, 1198, 1355-1358, 1361, 1364, 1373, 1377-1387, 1426, 1429, 1454, 1457
brewtils/rest/easy_client.py     289     31    89%   71-72, 810, 822, 837-840, 853-858, 881-885, 892-893, 900, 913, 925-926, 951, 968, 1043-1052

A significant number of the misses are related to File, FileChunk, FileStatus, upload_file, upload_bytes, etc.

Expose common things in __init__

90% of the time people want to use: system, command, paramater, RemotePlugin, and SystemClient. We should expose those at the brewtils module level.

Create an action for auto-assigning issues to the backlog project

We're going to attempt to start using a project board for backlog grooming. To facilitate this, we want new issues to automatically get assigned to the Backlog project and placed in the "New Issues" column. The idea is that we'll have an easily visible list of issues that need to be prioritized, either by deciding on where they rank in the backlog or promoting them to the "Dev Board" project where issues that are ready to work will live.

Smarter JSON exception parsing

For requests that error and have a command_type of JSON, we should do some better parsing of the exception. Namely, if the message reported is a dictionary or valid JSON, then we should just parse that instead of putting everything in the "message" attribute.

Update generated docs

The sphinx generated documentation is largely out of date, which means that the public documentation on readthedocs is also outdated. We should fix this, and also make some improvements along the way.

  • use sphinx to generate updated .rst files
  • Add proper deprecation warnings where appropriate so that they show up in the docs
  • Fix / cleanup return types where possible (to ensure proper documentation linkages)
  • There are some instances of notes that are not marked as such. Fix those so that they get properly formatted.

'Brewmaster' references

We need to refactor everything so that Brewmaster things are just aliases for backwards compatibility.

Update RoleAssignmentSchema domain field to contain scope and identifiers

The RoleAssignmentSchema domain field is currently a string because we hadn't yet determined what form domain is going to take. We've since settled on it being a dict of the form:

{ "scope": "string", "identifiers": {} }

We need to create a schema to represent that form above and then make the domain field a nested field for that schema.

SystemClient skip waiting for request

Right now you can either use blocking=True, which will block until the request completes, or blocking=False, which will return a Future that will complete when the request does.

This is pretty minor, but while doing stress-testing I just want to generate a whole bunch of requests and I don't care about the result at all. Using blocking=False is almost what I want, but it still submits a wait task to the ThreadPoolExecutor in the SystemClient. That means we're still constrained by the max_workers there, and it also means that Python won't exit until all the requests are finished (since Python won't terminate until all those workers are done).

I'm not sure if this is even a real issue, much less important enough to fix. It's kind of unexpected that Python doesn't exit, so we should probably at least document that if nothing else.

3.0 Brewtils Breaking Changes

This should track breaking changes in 3.0.

Comments to this can be used to jot down notes or ideas or things to look into (better than post-it notes ๐Ÿ˜„). However, the Authoritative List of breaking changes should managed by editing this description.

Below there's a list of checkboxes, each describing something. Check the box when the code that addresses that something is merged into the v3 branch AND documentation describing the something has been added to the upgrade guide.

  • All Brewmaster references should be removed
  • Rename _version.py to __version__.py
  • Convert enormous lists of kwargs to **kwargs (Plugin, Clients, etc).
  • SystemClient logger should be _logger
  • Plugin max_concurrent default (#47)
  • In decorators rename _commands to _bg_commands.
  • Remove deprecated names RemotePlugin and get_bg_connection_parameters from brewtils.__all__
  • Remove ValidationError, SPECIFICATION from brewtils.__init__
  • Plugins load configuration from command line and environment by default
  • Removed logger parameter from RestClient and EasyClient
  • Removed parser parameter from Plugin and EasyClient (both still accept kwargs so it would just be ignored).
  • Passing both a system definition and metadata to a plugin is now an error
  • Plugin instance attributes have been replaced by deprecated properties
  • Can now specify an alternate parent when making a request
  • Can now defer setting a Plugin client until plugin.run()
  • EasyClient parser attribute has been removed
  • Plugin _start and _stop methods shouldn't actually return anything
  • Global configuration - [System|Easy|Rest]Client uses it automatically
  • Passing a system definition as well as specifying bg_system or bg_version in the client @system decorator is only an error if the info doesn't match. Also, it's now an error if the info from the decorator doesn't match the config values regardless of whether the full system was specified or the helper kwargs were used.
  • Remove validation logic from models (Request.status)
  • EasyClient methods get_instance_status and update_instance_status are deprecated
  • brewtils.queues modules is deprecated
  • pika < 1 is deprectated
  • get_instance_status returns the actual instance, not the instance status
  • The default_exc for get_logging_config is RestConnectionError. It should probably be FetchError to match other GET requests.
  • get_queues default_exc is undefined, so it defaults to RestError. It should probably be FetchError to match other GET requests.
  • clear_queue and clear_all_queues default_exc is undefined, so it defaults to RestError. It should probably be DeleteError
  • pause_job and resume_job now return the Job (instead of always returning None) to match other Patches
  • get_version should probably return response.json(), not the response object
  • Make EasyClient behavior consistent when a get results in a 404. Some methods (system, request) return None, others raise an exception
  • Decide how the bg prefix will be handled for bg_host, bg_port, and bg_url_prefix. Will need to update get_connection_info.
  • SystemClient timeout parameter should support both negative numbers and None as meaning wait forever.
  • Add can_connect method to RestClient
  • get_config method in RestClient shouldn't pass kwargs directly to the session call
  • Plugin now loads logging configuration from Beergarden by default

This did not get done:

  • Remove old specification alternate names
    • We should really add a feature to yapconf that will show a warning if it loads a config value from a deprecated name.

Always url encode when communicating with beer garden

In any of the spots where http calls to a beer garden api endpoint happen we should encode the url first to ensure that special characters get handle properly. This is mostly relevant to accessing the gardens endpoints, but probably could just be done universally without any negative side-effects.

Handle 4XX resposnes better from brew-view

Right now, when trying to update a request, if we get a 4XX error, we will almost certainly retry forever if max_attempts is set to -1 which is the default. We should just abandon requests that have a 4XX error, since by definition it is "our fault" that something is not right.

Add additional plugin parameter

If I want to be able to pass an additional parameter to a plugin script it would be really nice to be able to hook in to the brewtils configuration loading process.

We should make it easier to merge a plugin's arguments with brewtils' so that the generated help includes everything.

Lark Parser Error

Lark moved their errors again and since we haven't pinned our setup.py dependencies, brewtils is broken.

Support Type Annotations

Now that type annotations are a real thing, it would be nice to have brewtils support them. Whatever functionality we would create needs to be backwards compatible, and ensure that it works correctly with python 2.7 - 3.7.

This alleviates the user from having to specify type in the @parameter definition which is a win.

While we are working on this, we also need to think about how popular libraries like mypy work. Another challenge will be deep inspecting when a type references another class, we will have to discover the parameters for that model as well.

Add EasyClient support for creating/destroying gardens

#342 added support for RestClient POST/DELETE requests for gardens. There should be support for these actions on the EasyClient if they exist on the RestClient, especially since EasyClient will help users bypass a layer of serialization needed for REST requests.

Initial Update

Hi ๐Ÿ‘Š

This is my first visit to this fine repo, but it seems you have been working hard to keep all dependencies updated so far.

Once you have closed this issue, I'll create separate pull requests for every update as soon as I find one.

That's it for now!

Happy merging! ๐Ÿค–

Propogate bg_host to the EasyClient->PikaClient connection parameters

In the following section of code, the connection_parameters for the queue are set for the ssl, though without propagating the other connection settings, the default host for AMQ PikaClient appears to be localhost. The fix is to set connection_info.host = bg_host. I'd be interested in getting permission to fork and submit a pull request. The goal is to have a remote or local Python 2.7 plugin the ability to run through a standard Python release.

connection_info = self._instance.queue_info["connection"]

Multiple parameter decorators at once

It would be nice to have a parameters decorator that takes multiple paramaters at once. This would be helpful for the case were parameters are defined elsewhere.

For example:

@parameter(**params[cmd1][param1])
@parameter(**params[cmd1][param2])
@parameter(**params[cmd1][param3])
def cmd1(self, **kwargs):
    pass

could become:

@parameters(params[cmd1])
def cmd1(self, **kwargs):
    pass

Update easy_client handling of import_jobs api response

With the recent changes to the import/jobs endpoint in beer-garden, the EasyClient import_jobs broke slightly in that it is returning stringified json rather than a proper deserialization of the api response. This should be fixed so that it returns a proper dictionary.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.