ndsev / zswag Goto Github PK
View Code? Open in Web Editor NEWAccess/Serve zserio services via REST/OpenAPI 🌍.
License: BSD 3-Clause "New" or "Revised" License
Access/Serve zserio services via REST/OpenAPI 🌍.
License: BSD 3-Clause "New" or "Revised" License
Hi, @josephbirkner , I'd like to report a vulnerability issue in pyzswagcl_1.2.1.
As shown in the above dependency graph (Here shows part of the dependency graph, which depends on vulnerable shared libraries), pyzswagcl_1.2.1 directly or transitively depends on 14 C libraries (.so). However, I noticed that some C libraries are vulnerable, containing the following CVEs:
libcrypto-990425a4.so.1.1
and libssl-337c2b28.so.1.1
from C project openssl(version:1.1.1f) exposed 5 vulnerabilities:
CVE-2021-3711, CVE-2021-3712, CVE-2020-7043, CVE-2020-7042, CVE-2020-7041
libuuid-f64cda11.so.1.3.0
from C project util-linux(version:2.27.1) exposed 3 vulnerabilities:
CVE-2018-7738, CVE-2021-37600, CVE-2016-5011
openssl has fixed the vulnerabilities in versions >=1.1.1l
util-linux has fixed the vulnerabilities in versions >=2.37.2
Python build tools cannot report vulnerable C libraries, which may induce potential security issues to many downstream Python projects.
As a popular python package (pyzswagcl has 6,985 downloads per month), could you please upgrade the above shared libraries to their patch versions?
Thanks for your help~
Best regards,
Joe Gardner
The following types of OpenAPI Authentication shall be supported:
These mechanisms are already supported by the zswag client, by either using the HTTP_SETTINGS yaml file or passing custom headers to the constructor.
The idea is to check the provided OpenAPI spec (via URL in the constructor) and cross-check with the entries in the HTTP_SETTINGS file. If the settings file contains the right cookie, api-key, etc. then directly use this. If not, provide an error message.
This part has been moved to #60:
Also add the possibility to programmatically add entries to the HTTP config, in cases where the token needs to be retrieved in a separate step before actually calling the service.
Building target zswagcl-test
fails on MacOS as it redefines a typedef already defined by the MacOS SDK
[build] .../zswag/libs/zswagcl/test/src/openapi-parameter-helper.cpp:9:7: error: typedef redefinition with different types ('Parameter::Style' vs 'unsigned char')
[build] using Style = Parameter::Style;
[build] ^
[build] /Applications/XCode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.0.sdk/usr/include/MacTypes.h:577:41: note: previous definition is here
[build] typedef unsigned char Style;
[build] ^
Currently, it is not possible to work with the HTTPConfig
class directly in code. We need a C++ API to set values on the class instances, and expose this API to Python.
The following changes are requested:
HttpLibHttpClient
's HttpSettings
instance.HttpSettings
programmatically with custom values.The pyzswagcl dependency is currently only deployed for 3.8. It should also be built for 3.9
At the moment, https://github.com/ndsev/zswag#url-compound-parameter is not supported.
It would be handy to be able to use compound parts of the request object in a path for example.
For this we would need the possibility to get a URL encoded string of the compound object which we then can map to a certain piece of the path. So if it is only about a 100% match of a compound object (e.g. a header structure) that would work great, no need to flatten things further.
The actual problem that I would like to target is to be able to map a request object which consists out of a scalar ID and a more complex header structure to a path of /{id}/{url_encoded_blob}
My understanding of zswag as now is that I can only url-encode the complete request object so that the ID cannot be mapped individually which kind of affects the backend more than necessary in terms of deployment.
The zswag-client should support additional OpenApi features to enable more REST-like interfaces:
Comprehensively formulated, this would enable method specs like this:
paths:
<Path>
[get|put|post|patch|delete]:
operationId: <Service-Function>
parameters:
- in: [query|path]
name: <Name>
x-zserio-request-part: <[*|field-name]>
style: [matrix|form|simple]
explode: [true|false]
schema:
format: [string|byte|hex|binary]
requestBody:
content:
application/x-zserio-object
I have observed that when the user provides junk input data (wrong payload type, or no payload) that an internal server error code (500) is returned to the user (before userspace code is called).
I think it would be useful to instead report an http 400 error to show that the error is a user error rather than a server error.
How to reproduce:
Desired result:
Currently, the zswag client lib is reinstantiating its httplib-client for each request. This may be slow, because it breaks HTTP keepalive optimizations. A new socket and a new TCP/SSL handshake are run for every tile.
Currently, zswag extensively uses git submodules to fetch its dependencies. This is not a good approach, because it leads to source duplication a source tree where a dependency appears multiple times. Instead, we will use Conan 2.0, which allows flexible declaration of dependencies via a conanfile.txt
.
In the case of multiple services sharing a similar method name, it is possible that zswag.gen associates the docstring of methodA
in one service with the doc string of methodA
in another service.
The generic C++ client talks to any zserio service that is running via HTTP/REST, and provides an OpenAPI specification of its interface. When using the C++ OAClient with your zserio schema, make sure that the flag -withTypeInfoCode is passed to the zserio C++ emitter.
It should be ensured that the flag -withReflectionCode is passed? Enable reflection code.
Currently, the zswag
client does not interpret the Server's OpenAPI spec. This could be improved, such that the zswag client verifies that the spec conforms to one of the ways the client can pass parameters. Additionally, the client could be moved to a separate package, such that it can be used without pulling the server dependencies.
Currently, the MIME type application/x-zserio-object
is only visible in OpenAPI.yaml for requests. This leaves some gaps:
application/x-zserio-object
should be generated in OpenAPI.yaml for responses.application/x-zserio-object
should be the server's response MIME type.Additionally, a content type should only be set when the body is non-empty. Currently, if the body is empty, and empty-string content type will be put into the request.
We need CI for the following tasks:
wheel
targetThe current Python bindings do not offer docstrings.
Current State: Currently, the zswag server doesn't check if a present OpenAPI YAML is usable because of version compatibility.
Desired Behavior: zswag check if a present OpenAPI YAML is usable (for example zswag could put the version that has been used to generate the OpenAPI spec directly in the YAML file) - zswag should output a message if the OpenAPI spec is not compatible.
I tried to build "C++ Client" in REAME but an error occurred.
Please tell me how to deal with this error.
$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.4 LTS"
$ python3 --version
Python 3.8.10
$ pip --version
pip 20.0.2 from /usr/lib/python3/dist-packages/pip (python 3.8)
$ cmake --version
cmake version 3.16.3
$ gcc --version
gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
$ java --version
openjdk 17.0.3 2022-04-19
$ ant -version
Apache Ant(TM) version 1.10.7 compiled on October 24 2019
$ cd myapp
$ ls
CMakeLists.txt api.yaml client.cpp services.zs zswag/
$ mkdir build
$ cd build
$ cmake ..
$ cmake --build .
...
...
[ 75%] Building CXX object CMakeFiles/myapp.dir/client.o
/home/m-kitahara/work/zswag/example/10_cpp/myapp/client.cpp:7:45: error: ‘MyService’ in namespace ‘myapp_services::services’ does not name a type
7 | using MyService = myapp_services::services::MyService;
| ^~~~~~~~~
In file included from /home/m-kitahara/work/zswag/example/10_cpp/myapp/client.cpp:3:
/home/m-kitahara/work/zswag/example/10_cpp/myapp/build/myapp-zserio-cpp.zserio-gen/myapp_services/services/MyService.h:24:11: note: ‘myapp_services::services::MyService’ declared here
24 | namespace MyService
| ^~~~~~~~~
/home/m-kitahara/work/zswag/example/10_cpp/myapp/client.cpp: In function ‘int main(int, char**)’:
/home/m-kitahara/work/zswag/example/10_cpp/myapp/client.cpp:18:45: error: ‘specUrl’ was not declared in this scope
18 | auto openApiConfig = fetchOpenAPIConfig(specUrl, *httpClient);
| ^~~~~~~
/home/m-kitahara/work/zswag/example/10_cpp/myapp/client.cpp:26:28: error: ‘MyService’ has not been declared
26 | auto myServiceClient = MyService::Client(openApiClient);
| ^~~~~~~~~
make[2]: *** [CMakeFiles/myapp.dir/build.make:63: CMakeFiles/myapp.dir/client.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:497: CMakeFiles/myapp.dir/all] Error 2
make: *** [Makefile:130: all] Error 2
Currently, we have zswagcl::HttpService
which only works for zsr-based requests. In order to work for Python too, the request builder part must be extracted into a utility OpenApiClient
class, like this:
class OpenApiClient {
void callMethod(
string methodName,
function<vector<buffer>(string)> valueFun, // path->value(s)
function<vector<string>(string)> enumFun); // path->members
}
From Python side:
zserio.ServiceClient
and overloads ServiceClient.callMethod()
PyOpenApiClient
as member to which calls are forwardedTasks:
Currently, request MIME types other than application/x-zserio-object
lead to a parsing error. The behavior of the parser should be changed, such that other request body MIME types are simply ignored.
Additionally, the zswag client should set the HTTP Accept
header, so that remote services know exactly what format to send.
As the OpenAPI generator now supports adding a base yaml with an OpenAPI info section, it would be awesome to get this also in the client code.
// Fetch the OpenAPI configuration using the HTTP client
auto openApiConfig = fetchOpenAPIConfig(openApiUrl, *httpClient);
I would assume that the above call would give me access to it.
It would be helpful if zswag would provide a Docker base image with all requirements necessary to run a zserio-swagger-based service.
As per the documentation, to have base64 encoding, the format must be specified as "byte," "base64" or "base64url".
By default, zswag.gen does not support setting this format on the command line.
It would be nice to have the format element supported on the command line. Since Zserio is a binary format, it will always need base64 encoding.
In order to support integration with other services I would like to host extra endpoints which are not part of the zserio service itself like /health or /metrics.
Options explored:
Some experimenting revealed that the issue could be solved by making a small change to https://github.com/ndsev/zswag/blob/v1.4.0/libs/zswag/app.py#L178 where an existing routing controller should be kept if it was explicitly defined in the openapi spec.
Open questions:
The following formats should be supported
string
: Default string representation of the x-zserio-request-part field (only works for native types).hex
: Hex representation of the x-zserio-request-part field.byte
| base64
: Base64 representation of the x-zserio-request-part field.base64url
: Base64url representation of the x-zserio-request-part field.binary
: Binary representation of the x-zserio-request-part field.Tasks:
We should build pyzswagcl also for Python 3.10 and 3.11, so that applications can use these more modern Python versions in conjunction with zswag.
As per zserio 2.5.0, there is a native zserio C++ reflection interface, so we can transition away from ZSR.
See ndsev/zserio#103
Start docs with a visualization of the artifacts in a typical project using zswag (zserio sources, generated zserio, zswag-gen and the OpenAPI spec, implementations of client/service). Ideally with a super-short overview of how the workflow steps utilize different parts of zswag.
Move the component UML further down - it is informative, but the details are not important for all users and the image looks scary for the uninvolved.
Rename "Server Component" section to "Python Server" and add a stub section "C++ Server" noting that it does not exist.
It would be nice if a new Zswag project could be instantiated from the command-line, akin to cargo new
.
For developers working with languages other than Python, it would be helpful if they could use the OpenAPI YAML generation/validation functionality without a Python function call. This could by achieved by exposign these features in a CLI tool
Support decompression on the client in case the server (and OpenAPI spec) offers a supported Content-Encoding
See https://www.iana.org/assignments/http-parameters/http-parameters.xhtml#content-coding for a list of compressions. I would suggest to start with supporting deflate
and zstd
Currently, only the path
and query
locations are supported. We shall support header
as well.
Currently, the zswag client cannot handle redirects. To handle them, we need to set set_follow_location(true) for the internal httplib client.
Let us deploy the zswagcl and httpcl libraries to conan center. This will simplify using zswag as a dependency, as there is then no need anymore to copy the sub-dependencies into the main project's conanfile.
The following changes must be documented:
OAClient
)x-zserio-request-part
, application/x-zserio-object
Mime Typehttpcl
/http-settings.yamlI tried to build zswag on msys2 but an error occurred.
The details of the error are not output and I am not sure how to deal with it.
Do you have any advice on how to resolve this error?
Also, what development environment (OS, etc.) does the zswag team use?
$ python3 --version
Python 3.9.13
$ pip --version
pip 22.0.4 from C:/msys64/mingw64/lib/python3.9/site-packages/pip (python 3.9)
$ cmake --version
cmake version 3.23.2
$ gcc --version
gcc.exe (Rev2, Built by MSYS2 project) 12.1.0
$ java --version
openjdk 18.0.1.1 2022-04-22
OpenJDK Runtime Environment (build 18.0.1.1+2-6)
OpenJDK 64-Bit Server VM (build 18.0.1.1+2-6, mixed mode, sharing)
$ ant -version
Apache Ant(TM) version 1.10.12 compiled on October 13 2021
Install dependent packages related to keychain.
$ pacman -S mingw-w64-x86_64-libsecret
$ pacman -S glib2
$ git clone --recursive https://github.com/ndsev/zswag.git
$ mkdir build
$ cd build
$ cmake ..
-- Building for: Ninja
-- The C compiler identification is GNU 12.1.0
-- The CXX compiler identification is GNU 12.1.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: C:/msys64/mingw64/bin/cc.exe - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: C:/msys64/mingw64/bin/c++.exe - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found OpenSSL: C:/msys64/mingw64/lib/libcrypto.dll.a (found version "1.1.1o")
-- Found Python3: C:/msys64/mingw64/bin/python3.9.exe (found version "3.9.13") found components: Interpreter Development Development.Module Development.Embed
-- Build spdlog: 1.10.0
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- Build type: Release
-- pybind11 v2.6.3 dev1
CMake Warning (dev) at C:/msys64/mingw64/share/cmake/Modules/CMakeDependentOption.cmake:89 (message):
Policy CMP0127 is not set: cmake_dependent_option() supports full Condition
Syntax. Run "cmake --help-policy CMP0127" for policy details. Use the
cmake_policy command to set the policy and suppress this warning.
Call Stack (most recent call first):
deps/pybind11/CMakeLists.txt:98 (cmake_dependent_option)
This warning is for project developers. Use -Wno-dev to suppress it.
-- Performing Test HAS_FLTO
-- Performing Test HAS_FLTO - Success
=> Running main zserio ant install ...
=> Running zserio cpp extension ant install ...
=> Running zserio doc extension ant install ...
=> Running zserio bundle install ...
CMake Error at deps/zserio-cmake-helper/CMakeLists.txt:62 (message):
Failed to build zserio.jar in
C:/msys64/home/70206703/work/zswag/deps/zserio.
The pyzswagcl
wheel depends on OpenSSL. Therefore, the binaries must be shipped inside of it.
As cpp-httplib does now support OpenSSL 3.0, we can move away from 1.1.
Implement extended logging on all steps executed by zswag, including timings.
In case zswag is waiting on some (service) response, zswag should do periodic logging of what it is doing as well.
The behavior should be parameterized (disabled by default).
Reason: currently zswag api acts as a black box. If a service connected to is responding slow or not at all it is unclear why this is happening.
By adding a more verbose mode such issues can be better investigated.
Currently, only generic HTTP error codes are reported. These should be annotated with some known reasons:
Implement parameters for OAServer
to indicate preferences regarding HTTP method and parameter passing style when the spec is auto-generated.
Currently, we are only compatible with 2.3rc1. We will need to make adjustments for the final 2.3.
Every error message generated by the parser should mention the offending value, the allowed values, and the location in the document where the error occurred.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.