Implement aiohttp.web OpenAPI 3 server applications with schema first approach.
Home Page: https://rororo.readthedocs.io
License: BSD 3-Clause "New" or "Revised" License
![badabump-release-bot[bot] avatar](https://avatars.githubusercontent.com/in/86060?v=4 "badabump-release-bot[bot]")
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
As of 2.0.0a2 version rororo does not provide any specific error message on validation error, it fully relies on openapi-core and later on which error middleware used in aiohttp.web application (if any).
Need to change this and return the predefined error response if some of request parameters or body is not valid (and same for response after #13 is done).
I'm trying to do a ajax call and retrieve the data passed to it as POST data. How do you do it,
As of 2.0.0a2 rororo provides an example app to cover petstore-expanded schema from OpenAPI 3 Specification Examples.
Need to provide more examples (with tests) to ensure that most of cases are covered by rororo (via openapi-core).
Good schemas to cover:
hobotnica.yaml, which covers Hobotnica API (ref: https://igordavydenko.com/talks/ee-pycon-2019/)When OpenAPI operation schema do not define the security such as,
get:
operationId: "me"
summary: "Retrieve user profile"
tags: ["auth"]
...operation.security still equals to empty list [], not None, which result the fix from #115 result in incorrect handling of global defined security.
Need to find a way to fix this and ensure that global defined security work as expected for such cases.
Parent Issue: #114
It's common task to do error/timeout handling for aiohttp requests. But as I now right now there is no handy way to do it. Suggests to implement two middleware factories for aiohttp apps,
error_middleware_factorytimeout_middleware_factoryWhere error middleware factory should be placed in middlewares settings after timeout middleware factory, so it handle timeout errors as well.
Need to ensure that optional security schemes is supported and result in passing security check by rororo.
Example of operation with optional security scheme,
security:
- {}
- apiKey: []More details at: OAI/OpenAPI-Specification#14
Original report by @chobeat at #84 (comment),
I think I've found an unrelated bug. In unmarshallers.py:198 it tries to access the keys of a None. From what I understand, it's doing recursion on nested dicts but it tries on the None value and it fails. This has been introduced between 2.0.0b3 and 2.0.0rc2. Before it was working properly.
I think it's happening on all my responses that contain at least a None value somewhere.
The idea to run commitizen check only for commits from playpauseandstop, but as of now workflow config has wrong conditional:
if: "github.actor = 'playpauseandstop'"which result in skipped tasks.
Also commitizen job should be required by dev & test jobs.
ps. If possible it would be really nice to provide other updates to Python & poetry steps within the CI jobs.
Instead of customizing attrs and providing custom env_factory need to use environ-config library instead.
Which means,
rororo.BaseSettings should be wrapped into @environ.config decroratorenv_factory and all related code removed from the repoNeed to deploy todobackend application somewhere into internet (Heroku, Netlify, own DigitalOcean instance) and then list it at Todo-Backend official web site.
Also, in production, there should be an auto-removal of data once in an hour to allow any user play with the demo.
Right now rororo dist files (both .tar.gz & .whl) contains CHANGELOG.rst file, which after installed into site-packages root, which has no sense at all.
Need to fix current behavior and do not include changelog into the dist.
Upload with multipart/form-data fails when schema validation is enabled. Exception "ValueError: Could not find starting boundary b'--somestuff' thrown when trying to access multipart info.
Get sample server code from (https://github.com/romado77/upload_test.git)
Run client with path where validation via rororo is enabled:
curl --location --request POST 'http://localhost:8080/api/v1/'
--form 'someinfo="sdjkhflksjhdflkjhlkjh"'
--form 'file=@"./test.zip"'
curl --location --request POST 'http://localhost:8080/api/v2/'
--form 'someinfo="sdjkhflksjhdflkjhlkjh"'
--form 'file=@"./test.zip"'
Python - 3.8.5
aiohttp - 3.7.3
rororo - 2.2.0
Schemas are Python interface to validate request/response data in API or normal views. Schemas are based on JSON Schemas and use jsonschema library for validation.
Below a small example of how Schemas should work inside views,
def view(request):
schema = Schema(module_with_schemas)
data = schema.validate_request(request.GET)
...
return schema.make_response({...})When response is a tuple, but not a list, next exception raised,
File "/Users/playpauseandstop/Projects/project/.venv/lib/python3.7/site-packages/rororo/schemas/schema.py", line 114, in make_response
self._validate(data, response_schema)
File "/Users/playpauseandstop/Projects/project/.venv/lib/python3.7/site-packages/rororo/schemas/schema.py", line 190, in _validate
return self.validate_func(schema, self._pure_data(data))
File "/Users/playpauseandstop/Projects/project/.venv/lib/python3.7/site-packages/rororo/schemas/schema.py", line 178, in _pure_data
return dict(data)
ValueError: dictionary update sequence element #0 has length 3; 2 is required
This is not fully right as tuple can be validated by jsonschema similarly to list.
Need to make it clear, that rororo is an aiohttp.web OpenAPI 3 schema first library.
Also put schema first notes to README and docs.
As of now get_validated_data(request) can contain lists (mutable data structures), need to ensure that validated data has only tuple, mappingproxy & atomic data structures to ensure their immutability between any get_validated_data(request) calls.
On setting up OpenAPI support for aiohttp.web application via setup_openapi function, need, by default, to enable cors_middleware & error_middleware for the application as well, cause of,
Need to provide examples of using schemas and other rororo gotchas. I think the most easiest way is porting swapi (or at least some its part) to aiohttp.web & rororo.schemas.
Need to add rororo to list of aiohttp third party libraries
Problem: Using rororo with aiohttp, I end up reloading the openapi.yaml for every test and populating the app with setup_openapi. This takes up 30% of the total time of my test suite. 15% is just loading the file in read_schema, while another 12% is spent in create_spec.
Proposal: decouple the spec creation from the addition of fields and routes to the app. After this, create a separate setup_openapi (maybe called setup_test_openapi) that can take a Spec object as an additional parameter and set it instead of creating it. The underlying implementation will be the same, but the first half of the current setup_openapi will be optional
I can try to work on this if you don't see conflicts arising
Need to ensure that any errors, not only validation errors, handled properly at create_schema_and_spec call. And that any errors there result in human friendly error message about problems with OpenAPI schema.
colorlog is a nifty way to have colored logs for debug environments. rororo.logging.default_logging_dict should be able to configure colored formatter for user and enable it by default for debug environments.
Hello there !
I was wondering if there are ways to register operations from an aiohttp.web.View class instead of doing it method by method?
For example I would like to do something like:
from aiohttp import web
from rororo import OperationTableDef
operations = OperationTableDef()
# @operations.register ?
class LogCollectionView(web.View):
# @operations.register("log_collection_get") ?
async def get(self):
pass
# @operations.register("log_collection_post") ?
async def post(self):
passThank you in advance!
Using rororo 2.0.0b3
Right now rororo is able to validate response if view handler has not raised an exception.
Need to investigate whether it is possible to validate error responses. Maybe it is possible after #22 by providing validate response middleware before error middleware, but after CORS middleware.
Right now there are couple of utils modules within source code:
rororo.utilsrororo.openapi.utilsBut it's good to rename them using more specific names as suggested in https://breadcrumbscollector.tech/stop-naming-your-python-modules-utils/
Right now, when OAS servers schema looks like:
servers:
- url: "/api/"
description: "Test environment"
- url: "http://localhost:8080/api/"
description: "Dev environment"
- url: "https://staging.url/api/"
description: "Staging environment"
- url: "https://production.url/api/"
description: "Production environment"setup_openapi requires route_prefix="/api" to be passed or it will register OpenAPI operations to unexpected URLs.
Need to find a way to not ask user to pass route_prefix by default and find out if it needed to be used from servers configuration.
jsonschema-rs in theory should allow rororo dramatically improve request/response validation speed.
As of 0.1.0 version error handling is not there, but it is still good idea to prepare basis for drop-in replacement for request/response validation via faster jsonschema alternative.
There is several predefined security schemes, supported by OpenAPI:
apiKeyhttpoauth2openIdConnectNeed to support all of given schemes and if the operation is require some of security scheme, parse request and pass parsed data into OpenAPIContext as security immutable dict.
Hello,
I cant seem to figure out how to extend Validations with custom validations.
Whenever I raise something, i (default?) error.
Example (for someplicity)
@operations.register
async def register_user(request: web.Request) -> web.Response:
with openapi_context(request) as context:
# something in context fails on more detailed validation
raise ValidationError.from_request_errors([OpenAPIParameterError("Something is wrong"), ], base_loc=["fieldname"])
I end up with this error:
{"detail": [{"loc": ["response", 0], "message": "'detail' is not of type object"}]}
I assume that whenever i use with openapi_context(request) as context: i get into the reponse layer.
Sadly i cant figure out how to combine it with my extra validations
Got next error after installing jsonschema==3.0.0 in project with rororo,
File "/Users/playpauseandstop/Projects/moneytracker/moneytracker/api/schemas/tracker_stories.py", line 10, in <module>
from rororo.schemas.utils import defaults
File "/Users/playpauseandstop/Projects/moneytracker/.venv/lib/python3.7/site-packages/rororo/schemas/__init__.py", line 11, in <module>
from .schema import Schema
File "/Users/playpauseandstop/Projects/moneytracker/.venv/lib/python3.7/site-packages/rororo/schemas/schema.py", line 18, in <module>
from .validators import DefaultValidator
File "/Users/playpauseandstop/Projects/moneytracker/.venv/lib/python3.7/site-packages/rororo/schemas/validators.py", line 18, in <module>
class Validator(Draft4Validator):
File "/Users/playpauseandstop/Projects/moneytracker/.venv/lib/python3.7/site-packages/rororo/schemas/validators.py", line 30, in Validator
}, Draft4Validator.DEFAULT_TYPES) # type: dict
File "/Users/playpauseandstop/Projects/moneytracker/.venv/lib/python3.7/site-packages/jsonschema/validators.py", line 131, in _DEFAULT_TYPES
raise _DontDoThat()
jsonschema.validators._DontDoThat: DEFAULT_TYPES cannot be used on Validators using TypeCheckers
Need to fix the error and ensure that rororo works well with jsonschema>=3.0
As it currently marks any release as pre-release, not only those, which have a, b, or rc in tag name.
As of 2.0.0rc0 version, read_openapi_schema is using,
json.loadsyaml.safe_loadfor loading OpenAPI schema (due to its initial format). But in reality, both of given functions are pretty slow, especially for big schema files. Need to,
CSafeLoader, not SafeLoader)ujson, rapidjson or other JSON library)This will allow increase startup time of web.Application as, for example, there is 10x difference between SafeLoader and CSafeLoader,
In [1]: import yaml
In [2]: from api.constants import PROJECT_PATH
In [3]: schema_path = PROJECT_PATH / "openapi.yaml"
In [4]: %timeit yaml.load(schema_path.read_text(), Loader=yaml.SafeLoader)
164 ms ± 10.1 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
In [5]: %timeit yaml.load(schema_path.read_text(), Loader=yaml.CSafeLoader)
14.6 ms ± 339 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
This also will require to update the docs, by adding Speedups page to write down, how it is possible to speed up rororo.
Need to provide brief documentation for rororo project, the structure of documentation may looks like,
rororo.loggerrororo.schemasrororo.settingsNeed to create releases via Github Actions with data from release tags.
Some openapi-core issues, such as,
OpenAPIMediaTypeError exceptionsresulted in specific implementation details, which should be covered in documentation to provide better understanding why rororo need to fix given (and other) issues with openapi-core or other tools.
If security defined for all operations globally as,
security:
- jwt: []and some operation declares empty security list like,
paths:
"/public":
get:
operationId: "retrieve_public_data"
summary: "Retrieve public data"
security: []
...rororo still expected retrieve_public_data endpoint to receive data from jwt security scheme, which is wrong.
Need to fix and ensure that those operations do not require any secure data to work well.
As of 2.0.0rc1 on converting aiohttp web.Request instance to OpenAPIRequest instance body passed as bytes, not string,
if request.body_exists and request.can_read_body:
body = await request.read()This results in issues of unmarshalling strings, when request body defined as,
requestBody:
content:
text/x-python:
schema:
type: "string"Need to fix it by supplying string as OpenAPIRequest.body, which means using await request.text() instead of await request.read()
openapi-core==0.13.3 contains a lot of updates, which might break rororo, need to,
openapi-core versionrororo caused by updateAs openapi-core using strict-rfc7739 under the hood for validating / unmarshaling date time strings, which likely do not support tz aware date time strings, need to ensure that passing tz aware date time strings will be converted into proper Python object.
As rororo uses email-validator for validating email strings need to allow setup check_deliverability and other possible options to validate_email call.
check_deliverability could be disabled to avoid making extra DNS calls, or when there is no need in caring about deliverability (like in test environments).
There is two necessary things that still missed in current implementation of env_factory helper.
When attribute type is an optional something, rororo doesn't try to convert the value from env to something, but returns the Optional[str] as os.getenv does. Which results in invalid expectations between type annotation & runtime execution.
Example below should work,
import uuid
import attr
from rororo.settings import env_factory
@attr.dataclass
class InviteSettings:
secret_key: Optional[uuid.UUID] = env_factory("INVITE_SECRET_KEY")
settings = InviteSettings()
if settings.secret_key:
assert isinstance(settings.secret_key, uuid.UUID)but with rororo==2.0.0b3 it doesn't, as last settings.secret_key will be a str instance, not uuid.UUID
Very often it is a case to provide an env sequence value as string joined with comma separator or some other delimiter.
It is understandable that support something like,
@attr.dataclass
class InviteSettings:
secret_keys: Optional[List[uuid.UUID]] = env_factory("INVITE_SECRET_KEYS")is a bit complicated and can take a time.
But in same time supporting List[str] via env_factory should be doable and helpful.
While it is not the best for the performance, it is needed to have a way to enable validation of response data against the OpenAPI schema.
openapi-core supports it, so should the rororo.
Ensure to support request body with type: "array".
Currently, in 2.0.0a2, rororo cast request body to dict, but this will not work well for array requests.
There are several desired enhancements to aiohttp.web library that widely used in backend applications which should be added to rororo,
Application to support debug mode and proper logging while serving with gunicornadd_route context managerDue to #114 and related PRs #115, #117 & #120 rororo on setup walks through OpenAPI schema in attempt to fix security defintion for the operations.
However, when some path definition describe common parameters as,
"/auth/me/email/{request_uid}":
parameters:
- $ref: "#/components/parameters/RequestUID"rororo doesn't recognize this case, which results in,
.venv/lib/python3.9/site-packages/rororo/openapi/openapi.py:675: in setup_openapi
spec = fix_spec_operations(spec, cast(DictStrAny, schema))
> mapping[operation_data["operationId"]] = operation_data.get(
"security"
)
E AttributeError: 'list' object has no attribute 'get'
Need to fix the issue and expect parameters to be defined for entire path.
After a long test phase it is a time to publish final 2.0.0 version of rororo.
This version highlights changing rororo into schema first library for creating OpenAPI 3 aiohttp.web server applications.
At a moment rororo==2.0.0rc3 deployed to production environment for 5 projects and is literally ready to have final 2.0.0 version.
To ensure that performance is not touched between rororo releases.
As of now I'm thinking about using pyperf as benchmark runner, and benchmarking,
todobackend example,
setup_openapi performancevalidate_request performance by creating 100+ todos via create_todo operation (without Redis layer)validate_response performance by emulating list_todos operation with 100+ todos (without Redis layer)hobotnica example,
validate_security performance for basic authvalidate_security performance for HTTP authTo complete task need to store benchmark results somewhere (gh-pages branch or as release artifact) and have ability to compare benchmark results between releases.
Need to provide various updates to validation errors, such as:
__add__) validation errorsAs of simplifying raising validation errors, need to introduce validation_error_context (via contextvars library), which allows to raise errors such as:
# In views.py
def create_item(request: web.Request) -> web.Response:
with validation_error_context("body"):
item = validate_create_item(get_validated_data(request))
return web.json_response(asdict(item))
# In validators.py
def validate_create_item(data: MappingStrAny) -> Item:
with validation_error_context("subitem"):
subitem = validate_create_subitem(data["subitem"])
if item["field"] != 42:
raise ValidationError.from_dict(field="Unsupported field value")
return Item.from_dict(data, subitem=subitem)
def validate_create_subitem(data: MappingStrAny) -> SubItem:
if data["enum"] not in [42, 420]:
raise ValidationError.from_dict(enum="Unsupported enum value")
return SubItem.from_dict(data)And if (for example) request body will have invalid field or subitem.enum value validator functions will generate next errors,
For invalid field value:
[
{"loc": ["body", "field"], "message": "Unsupported field value"}
]subitem.enum error:
[
{"loc": ["body", "subitem", "enum"], "message": "Unsupported field value"}
]Currently rororo ValidationError has next format,
interface ValidationError {
detail: Array<{
loc: Array<string | number>
message: string
}>
}while fastapi & django-ninja use a bit different format,
interface ValidationError {
detail: Array<{
loc: Array<string | number>
msg: string
type: string
}>
}With that in mind need to update rororo ValidationError to be compatible with other validation errors.
And remove obsolete workarounds for latest fixes in openapi-core
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.