How do I set up Fluentd, ElasticSearch and Kibana for use with wicked? about wicked.haufe.io

[DonMartin76]

This question is a little vague perhaps. Can you be a little more specific?

I can just tell that the combination of fluentd and file-log seems a little off; I would expect you to use http-log with a fluentd instance/end point somewhere. That fluentd is just a log forwarder is clear, right? You will need more components to get useful data out, like ElasticSearch and Kibana.

If you don't have expertise in running those kinds of systems, using an out of the box SaaS solution like Loggly or Galileo might be advisable.

/Martin

[TESSaranya]

I meant the fluentd. I am working on a Ubuntu server with docker on an Azure VM, I have my td-agent downloaded, but i am not able to get it work with my kong. When I use a Curl kong command, it says cannot reach http:/kong:8001.

[DonMartin76]

That curl http://kong:8001 does not work from the docker host is expected; kong is a virtual DNS name inside the docker network, and it's not exposed to the outside (i.e. to the docker host). Neither would you ever expose the Kong Admin API Port without securing it to the outside world.

Can you describe what you are trying to accomplish? What is the result you expect or want?

[TESSaranya]

I am trying to log my request and response of my wicked APIs

[TESSaranya]

Elasticsearch and Kibana seems to be the option, but I am not sure how to go about it. Any help?

[DonMartin76]

I haven't done this yet for wicked, as we are internally using our own GreyLog logging instance, so that I haven't had the need to do this yet. It is definitely on my todo list to write some documentation on how to do this, but I can give you some hints what has to be done now.

• There are (base) docker images for all three components, so you can combine them into their own docker-compose.yml file;
• You will probably need to create a suitable configuration for fluentd, but simply forwarding to elasticsearch should work for starters, as the log format of http-log is JSON
• I would use a separate deployment for this, possibly to a different docker host
• Configure your APIs to use the http-log plugin and insert the end point your fluentd instance listens to as the receiving end

A sample (NOT suited for production) can be found in one of our github repositories:
https://github.com/Haufe-Lexware/docker-templates/tree/master/logaggregation

Things to get straight:

• You will not want to expose your logging port to the public (if possible); if you have to, secure it via SSL and possibly Basic Auth
• You should also secure the Kibana application in a suitable way, otherwise it'd be available to the public, which is most probably not desired.

[TESSaranya]

Thanks Martin, I will try that out surely.

[TESAARON]

Hi Martin,

On this same topic, I have been trying to implement the http-log plugin with our published API. The goal is to send log data in json format to a listening http endpoint that can then be consumed by logstash/elastic or sent directly to a back-end database. Unfortunately I was not able to get this working. Below is the plugin configuration I have used:

[
{
"name": "http-log",
"config": {
"http_endpoint": "http://tes-secure.cloudapp.net:8081/log",
"method": "POST",
"timeout": "1000",
"keepalive": "1000"
}
}
]

where within my application I have an endpoint that accepts a POST to the url http://tes-secure.cloudapp.net:8081/log. However, this does not send any log data to the endpoint. I have tested using curl:

curl -X POST -d "type":"testing POST to url","value":"100" http://tes-secure.cloudapp.net:8081/log

and this does indeed send the enclosed json data to the endpoint. Am I missing some basic understanding here of how this should work?

Many thanks,
Aaron

[DonMartin76]

I have done some testing, and for me it seems to work in principle. I can just guess on a couple of other things which may go wrong:

• Does the docker Host physically have access to your logging server?
• E.g., are the ports open for outbound traffic in Azure?

If you deployed to your Docker Host using your own machine, you can test this by doing the following:

$ docker ps
... # read out the container ID or NAME of the Kong instance
$ docker exec -it <ID or NAME> bash
[root@80c91ab31513 /]$ curl -X POST -d "testing=access" http://tes-secure.cloudapp.net:8081/log
Successful Post[root@80c91ab31513 /]$ 

If that does not succeed, you are having a connectivity issue from Kong to your logging "service". If that does succeed, we will have to look for other reasons.

[TESAARON]

Hi Martin,

Thanks for your response. The curl from the kong container was successful. I think you may be correct regarding the outbound rules with Azure. I have been on a support call with Microsoft separately regarding the exposure of say https://portal.thinkevolvesolve.ie and they are currently investigating why the rules we have are not working.

However, if the curl from the Linux VM is working this would suggest outbound rules are working, no?

Many thanks,
Aaron

[DonMartin76]

Yes, the outbound rule seems to be working if the curl works from within the container.

While you're in the Kong container, check /usr/local/kong/logs/error.log, do you see anything suspicious there?

[TESAARON]

This gives back an empty file.

I have tested the http-log plugin on my laptop and it works - it's sends logging data to the endpoint - so this suggests that implementation on the Azure VM is where the error lies. Most likely as you have suggested due to in/outbound rule setting.

I will let you know whenever Microsoft resolve this to confirm the source of the error.

[DonMartin76]

Perfect. Thanks. Any information which leads to others not running into this situation is appreciated.

[DonMartin76]

@TESAARON: You wrote you had managed to set up an ELK stack for your wicked instance. Mind to share how you configured it?

[TESAARON]

Hi Martin,

Yes no problem. Here are the steps I followed to configure ELK (elastic, logstash, kibana) with the API portal.

1. In Kickstarter, when configuring the API, use the Kong http plugin. This will look like the following:
   [
   {
   "name": "http-log",
   "config": {
   "http_endpoint": "http://localhost.net:8081/log",
   "method": "POST",
   "timeout": "1000",
   "keepalive": "1000"
   }
   }
   ]
2. In my API runtime I have a listener at http://localhost.net:8081/log that pushes the posted log to a MySQL database in the backend.
3. Downloaded and unpack the following ELK stack in a directory i.e. 'elk_docker'
   ELK.zip
4. Type the following commands:
   $ cd path/to/elk_docker
   $ docker-compose build
   $ docker-compose up
5. This should build the elk container and start it running.
6. If configured properly this will now listen for changes to the table: logging.tbl_api_log in the backend mysql database.
7. Therefore any calls to the API will now:
   (a) send a log report to the MySQL table (logging.tbl_api_log)
   (b) elastic and logstash will listen for any new entries to this table
   (c) kibana (accessible via localhost:5601) will visualize these logs.
8. For further information on the configured ELK stack refer to:
   https://hub.docker.com/r/sebp/elk/

Regards,
Aaron

[DonMartin76]

Thanks!

[DonMartin76]

This is no longer really interesting to set up specifically for wicked. Wicked's containers all log to stdout, and thus you can ship off all logs for both the wicked and the kong containers to wherever you want.

It does not solve the analytics part, but that's still not a priority for wicked.