haufe-lexware / wicked.haufe.io Goto Github PK
View Code? Open in Web Editor NEWAn API Management system based on Mashape Kong
Home Page: http://wicked.haufe.io
License: Other
An API Management system based on Mashape Kong
Home Page: http://wicked.haufe.io
License: Other
From which Haufe-Lexware repo comes the content of http://wickedhaufeio.readthedocs.io/en/latest/
I thought it would be this repo, which I quess it is... but the readthedocs is not updated with the changes made 18 days ago. Anyhow: I am trying to run this service on DC/OS but building the wicked-api documentation is not sufficient with my experience. In the readthedocs there is a reference to docker-build.sh which I can't find in any of the Haufe-Lexware repos.
In other documentation I read the following: The STATIC CONFIG is created at deployment time by building a static configuration container directly on the docker host which is to host the API Portal. This is usually done by keeping a special Dockerfile inside the deployment repository, which in turn clones the configuration repository into this container.
So I indeed created a repository, but I have no clue which how to address this repo to the docker image on docker hub. Are do I have to write this intelligence my self... then I would be handy on how to build the wicked-api
Thx for your help and feedback.
Daniel
There are some changes to Kong from 0.8.3 to 0.9.x which make it impossible to just switch out the version (breaking API changes). This has to be taken care of when starting the API (migrating configuration settings) or starting the Kickstarter.
Known issues:
rate-limiting
and response-ratelimiting
configuration parameter continue_on_error
has been renamed to fault_tolerant
, and the async
property has been removedDue to a misconfiguration the portal-mailer did not work on a deployed API portal and no verification mails were sent out to users who had registered.
After the misconfiguration was fixed the verification mails were sent out, but by then the verification link already was expired and when trying to follow it the attached error message appeared.
The unverified user can login to the portal anyway and can register applications - so what's the point of the verification?
Which content is stored in the "Dynamic" volume. We have noticed when we want to start with a complete new "static" configuration that it is best to clean the previously dynamic mounted volume as well. We can of course within our setup choose not to mount the dynamic folder to an external volume, but are thoughts were that this folder might contain certain data which the Portal API might use when the docker was restarted again?
So it would be helpful to know if the dynamic data folder content is of any value within a running Portal API setup.
All components should be able to expose their version information; it's stored in the package.json, and could be exposed in the /ping
end point. The wicked.portal
should surface this information on the "System Health" page (available to Administrators).
Hi Martin,
We have successfully deployed our management portal to a live environment (using correct domain names in the Azure VM where we have deployed the gateway and portal). We also have working API analysis using the Kong http plugin (sending log data to a backend database) and elasticsearch/logstash/kibana to detect and visualize any logging events that occur on an API call. Also, since our logging data is stored in a database, we can implement an alert mechanism for API call errors etc using a cron schedule to listen for error events and push out alert emails/texts.
The final piece to the puzzle is API billing using the rate-limiting plans in kong that you have integrated in the portal. Have you thought of how billing against these plans could be implemented? Is there a further billing application that would need to be integrated within the portal where customers need to input credit card details to buy a specific plan?
Any thoughts you have on this would be greatly appreciated.
Regards,
Aaron
In my test portal I created a user in the admin group with kickstarter and removed the default admin user (id #1) from the admin group to mitigate the risk of a well-known admin user.
That had the unfortunate consequence that the portal-kong-adapter, portal-mailer and portal-chatbot containers did no longer start.
To mitigiate that risk anyway it would be useful to create the admin user without a passwort from the start - iff another admin user was configured.
In order to gain more flexibility and safety during updates of both configuration and images, the following behavior of components depending on the Portal API may be very helpful:
Configuration-Hash
or similar)This would enable a much easier handling of configuration updates: It would be enough to stop/restart the portal API, and all other components would automatically draw the new configuration. Such behavior would be ideal for environments such as Kubernetes.
Hello,
I have followed all the steps in the following file for local deployment of the portal:
https://github.com/Haufe-Lexware/wicked.haufe.io/blob/master/doc/deploying-locally.md
However, when I go to the url: https://portal.local/ I get the following screen saying: "The Portal is not yet fully initialized. Please reload in a couple of seconds." However nothing happens when I reload this screen.
Any ideas how I could rectify this issue?
Many thanks.
Hello,
I am encountering the following error when I run a 'docker-compose up' on my local instance. Any ideas what could be causing this?
$ docker-compose up -d --remove-orphans --force-recreate
Recreating portalapi_portal-api-data-static_1
Creating portalapi_portal-api-data-dynamic_1
Creating portalapi_kongdb-data_1
Creating portalapi_portal-api_1
Creating portalapi_kong-database_1
Creating portalapi_portal-mailer_1
Creating portalapi_kong_1
Creating portalapi_portal_1
Creating portalapi_portal-chatbot_1
Creating portalapi_portal-kong-adapter_1
Creating portalapi_publb_1
ERROR: for publb Cannot start service publb: driver failed programming external connectivity on endpoint portalapi_publb_1 (578a9deb553e684847f11591a5a4f0a7602c356a5f7b6ad587cb40e24a6d96f4): Error starting userland proxy: mkdir /port/tcp:0.0.0.0:1936:tcp:172.19.0.9:1936: file exists
Encountered errors while bringing up the project.
Regards,
Aaron
Thanks for creating and sharing the portal.
I have installed it on a local VM and it seems its working (checked locally thru lynx) but how do I access it from other machines on the nerwork.
I am new to Docker and only installed for this project.
here are the IPs that I have on the VM but my local network is 192.x, so other IPs cannot be used:
br-3a826545134d Link encap:Ethernet HWaddr xx:xx:xx:xx:xx:xx
inet addr:172.x.x.x Bcast:0.0.0.0 Mask:255.255.0.0
--
docker0 Link encap:Ethernet HWaddr xx:xx:xx:xx:xx:xx
inet addr:172.x.x.x Bcast:0.0.0.0 Mask:255.255.0.0
--
eth0 Link encap:Ethernet HWaddr xx:xx:xx:xx:xx:xx
inet addr:192.168.x.x Bcast:192.168.1.255 Mask:255.255.255.0
This is closer to a production setup, since you won't be running management system on localhost.
FYI: I used instructions at http://wicked.haufe.io/gettingstarted.html
When trying to access a page which is protected by a user login (would return a 403), the portal should redirect to the login page, and after successfully logging in, should redirect back to the place the request came from.
Hi Martin,
How is the portal and api-portal application built? Is it a node.js application with API calls to kong database, mailer, chatbot etc? If so, is it possible to view the structure of these APIs in the code?
Aaron
To make killing containers e.g. in Kubernetes, the portal-api should react to SIGTERM (and SIGINT) and shut down itself when it receives this signal.
This is especially needed when running in Kubernetes, as the scheduler first sends a SIGTERM, then, after 30s, a SIGKILL. If a container is to shut down, it should do so immediately (but gracefully) when it receives SIGTERM.
Side note: The other containers (mailer, chatbot, kong adapter) do.
Which is the indicator for the wicked portal to indicate that the portal is running in development mode? We have created our own environment, but each time when we launch it, the portal display the (see title message) on every page bottom.
It would be good to have integration tests for testing the kong adapter functionality; this is especially useful to make sure a new version of Kong does not break wicked.
I tried to create a new environment for the host test.smartpilot-apim.haufe-ep.de, and thus I thought I'd call it by the host name.
That leads to an error page which is quite unpleasant to look at - a simple hint at the name field would have been sufficient.
... because the config hash is calculated from the /usr/src/app dir, and not the /var/portal-api/static dir. Now this is really embarrassing.
Hi Martin,
We encountered a slight error when deleting an API on kickstarter. On restart of the portal when an API is deleted we get the screen: 'The Portal is not yet fully initialized. Please reload in a couple of seconds.'
I debugged using 'docker-compose up' and the error seems to be that the portal is still expecting a subscription to the deleted API (called twitter-api) on a given application (test-app). The error specifically is:
"Application 'test-app' has a subscription to invalid API 'twitter-api' "
Any idea how we can rectify this? Perhaps we need to delete the entire entry for 'twitter-api' in the Kong database.
Regards,
Aaron
Current workflow for running the unit tests involves a lot of manual typing, if you want to do it locally.
It would be beneficial if it was possible to run the unit tests locally with a simple script, instead of having to create temporary docker images just to do that.
Hello,
I have configured the Email section within kickstarter and require that new users receive a verification email before being able to use their account. I have tried numerous different ports (25, 465, 587) and smtp servers but no configuration seems to work. Have you encountered this issue before or do you know how I could debug the issue on the command line?
Regards,
Aaron
For transition scenarios where the wicked core components are updated one by one (which will be the case for Kubernetes and alike), the wicked components should do a standard version check on the portal API when calling it. It should pass an "expected version" e.g. as a header to the upstream API, and if present, this version is checked by the API and in case it does not match the call is rejected.
This will be optional behavior which will be implemented by the standard components, but which will not be enforced by the portal API in case the additional header is not present (this will be a similar behavior as described in #36).
Containers which receive such a faulty version, and/or have a version mismatch, will fail early and fast with a descriptive error message.
As soon as the version of the container has been updated, the operations would return to normal again.
In case there is an error when requesting an access token via the implicit flow, using the Kong Adapter's OAuth2 functionality, it must answer with a redirect back to the client's redirect URI with an error in the fragment as specified in RFC6479, section 4.2.2.1. This is currently not the case.
I am getting following message
The Portal is not yet fully initialized. Please reload in a couple of seconds.
State: Waiting for API...
If a user does not have a Display Name in GitHub, login/signup fails with status 500, "Failed to fetch user profile".
My API is returning links with wrong schema if used from the API portal.
When using the API directly the links are returned with the correct scheme.
The reason is the setting of the request header X-Forwarded-Proto which is always http even when the wicked portal is only serving https.
Interestingly the value of this particular header is displayed as https in the kong http-log, but it arrives as http at my API server.
you can take the one from the developer blog for a start
The kong adapter should provide functionality which makes it easy to implement the Resource Owner Password Grant Flow on top of Kong.
Why do we require kickstarter to add API's? Since the Portal UI should be an interface on top of KONG one should be able to configure all KONG API supported features... not? So it would be better to add can configre API settings via the wicked Portal based on ADMIN rights or other group settings then each time building a new "static" dataset for the API that one wants to support.
If a typo is made in the definition of the backend API url this is somewhat hard to find out before the portal is deployed.
A health check like a HEAD request to the given API url could save a lot of stress - especially for beginners.
This would also be useful in the kick starter, e.g. in the form of a verify API url link
Creating a new environment variable which needs multiple lines, such as certificates or RSA keys, is rather difficult.
new property
(or simliar)Edit the environment file explicitly once and add "hello\nworld" as the value for the new variable; this will make the kickstarter use a text area instead of an input field from then on (as long as you don't remove the value or just enter a single value).
I am unable to get fluentd work with kong? Any help
screenshot in readme of repo still shows gummibear :)
It seems that when the kong-adapter service is restarted, that it tries to set in Kong via REST POST call again previously configured API's. This will result in an error. Maybe it would be better this the service first check if there is already an API configured, if not create it... if so check if certain config settings require and update.
npm info using [email protected]
npm info using [email protected]
npm info lifecycle [email protected]prestart: [email protected]start: [email protected]
npm info lifecycle [email protected]
{ 'config.path': 'Cannot create file. Make sure the path is valid, and has the right permissions' }
/usr/src/app/bin/kong-adapter:62
throw err;
^
Error: kongAction POST on apis/22bd42d4-f14b-49ec-90f1-d144c74110c4/plugins did not return the expected status code (got: 400, expected: 201).
at Request._callback (/usr/src/app/kong/utils.js:124:25)
at Request.self.callback (/usr/src/app/node_modules/request/request.js:187:22)
at emitTwo (events.js:106:13)
at Request.emit (events.js:191:7)
at Request. (/usr/src/app/node_modules/request/request.js:1044:10)
at emitOne (events.js:96:13)
at Request.emit (events.js:188:7)
at IncomingMessage. (/usr/src/app/node_modules/request/request.js:965:12)
at emitNone (events.js:91:20)
at IncomingMessage.emit (events.js:185:7)
When starting wicked in the latest 0.10.0 version, the kong adapter fails, as the portal API returns a 500
status when asking for the auth-servers
.
Workaround: In the static
configuration directory, create an empty auth-servers
directory, add a dummy.txt
file and retry. Alternatively, start the kickstarter in version 0.10.0 once and check in the changes.
Will be fixed in 0.10.1.
Most people tend to omit the scheme from urls nowadays and there are browser that don't retry on https automatically.
In this regard it would be nice if the portal could redirect requests on port 80 to port 443 automatically.
Hello,
I have now successfully deployed the portal both on my local and on a cloud based Linux machine. Now I want to add a new API which can be accessed from the following backend url:
http://tes-secure.cloudapp.net:8081/article-pageview-metrics/get?articleId=2707217
As such, in the Kong Configuration for my new API I have put in the following credentials:
I have also added key authentication to this. However, when I run the following url in postman (including the header key):
https://api.portal.local/article-pageview-metrics/get?articleId=2707217
I get a 404 response 'No listener for endpoint /?articleId=2707217.
Any ideas what could be happening?
Regards,
Aaron
This is a requirement from Haufe:
This boils down to the following requirements:
auth
type is added, oauth2-implicit
redirect_uri
, which is needed for the authorization server/oauth2/register
which takes user data and returns an access tokenUse Cases which can be implemented using this:
The authorization server can, in the SAML case, be registered as a SAML SP, and can federate the authentication to the APIs; this enables you to just register the API Portal once, and then use wicked to enable self service SSO to your SAML identity provider. This is very nifty.
In a newly created configuration, the default value for PORTAL_CHATBOT_URL
in default.json
was incorrectly set to http://portal-api:3004
. It has to be http://portal-chatbot:3004
.
In previous versions, the chatbot actually ignored was what stated in default.json
/globals.json
and just checked the environment variable directly; if it wasn't found, it defaulted to http://portal-chatbot:3004
on Linux, and otherwise to http://${LOCAL_IP}:3004
.
Fix: Update your default.json
to have the correct value for PORTAL_CHATBOT_URL
.
Hello,
I have managed to deploy the API portal on a separate Linux VM but as yet cannot get it running on my local Windows laptop. Docker is installed and running but when I run the command:
docker run -it --rm -v C:\Users\pc\Documents\api_portal:/var/portal-api -p 3333:3333 haufelexware/wicked.portal-kickstarter --new
I don't get any created files in the repo I created: C:\Users\pc\Documents\api_portal. I can view the kickstarter GUI at http://localhost:3333/ but no local files have been created. Any ideas why this command doesn't work for us on Windows but does on Linux?
The env checkbox for the env list in the title was enabled via "kickstarter" but when launching the wicked portal api we always notice : WARNING: Environment variable PORTAL_MAILER_SMTP_HOST is defined but never used..... this parameter was also configured in the end via the "kickstarter" tool but no mails are ever send out. We also notice the below error message although that the port is configured for smtps (which can only be configured via kickstarter and not via an environment variable).
{ Error: Greeting never received
at SMTPConnection._formatError (/usr/src/app/node_modules/smtp-connection/lib/smtp-connection.js:493:15)
at SMTPConnection._onError (/usr/src/app/node_modules/smtp-connection/lib/smtp-connection.js:479:16)
at SMTPConnection. (/usr/src/app/node_modules/smtp-connection/lib/smtp-connection.js:420:18)
at ontimeout (timers.js:365:14)
at tryOnTimeout (timers.js:237:5)
at Timer.listOnTimeout (timers.js:207:5) code: 'ETIMEDOUT', command: 'CONN' }
It's possible to use a fragment delimiter in the redirect URI of an application; this is in turn (deep down) rejected by Kong.
You can notice such errors by monitoring the pending events on the kong adapter; if these never reach 0
, some event could not be processed in the back end.
Workaround: Don't use fragments in the redirect URI.
Fix: Reject the hash character in redirect URIs.
After an import of a configuration using the /deploy/import
API, the wicked components using a machine user fail (with 403
return codes) when trying to access the API.
This is especially bad as the importer waits for the kong adapter to finish its initialization before it returns a success code.
Workaround: Manually restart the components after an import.
It would be convenient to have an easy means of using mutual SSL when calling backend services. Nginx has built-in support for this, but we would need to surface it in the configuration in some way, preferably by injecting it via environment variables, and not by requiring somebody to extend the Kong image.
For starters, only a single certificate will be supported; this means all backend APIs requiring mutual SSL will need to share one private key. This is not ideal, but better than nothing.
Is there a preferred order of launching Wicked Portal docker containers? We have seen that when a new "static" config was created with "kickstarter" that sometimes the other running docker containers should also be restarted when the new "Portal API" is launched.
See also #25:
Deleting an abandonded API is perfectly fine and should be possible. Right now it's not that simple to do that right, as deleting an API requires all subscriptions to be deleted first. To make this simpler, the following functionality would be good:
Thoughts? Is this aiming too low, are there other big problems that need to be taken care of?
In some cases, it's not abundantly clear that some panel boxes can be dropped down to show additional content, e.g. the "Toolbox" on the API Kong Config page.
These boxes should at least have a »
(») in the title, and if possible also show a "hand" symbol when hovering over them. Right now, the only thing which shows it can be clicked is that the title is underlined (at hover).
The kickstarter pre-configures the file-log
plugin for Kong when creating a new API. This does not make sense, as the paths have changed in the Kong image since 0.8.x to 0.9.x.
This should be taken out.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.