philips-software / logproxy Goto Github PK

Starting LogProxy v1.6x the "pure passthrough" functionality was introduced allowing LogEvent.logData.message encoded in base64 to be passed unaltered to Kibana. In versions prior to v1.6x LogProxy would encode clear text into base64 automatically.

Recently Kibana experiencing long downtimes which drives us to inspect log messages from services using CF and makes it a huge hassle to decode each message manually.

It would be nice to have an on/off switch for "pure passthrough" functionality on TF level or even better to have LogProxy automatically recognize the clear text vs base64encoded and handle it accordingly.

Will appreciate any help on the matter! @loafoe

• Add option to disable delivery workers
• This should only be possible if RabbitMQ is selected as queue mechanism

I am moving my issue to official github

Can we get a FAAS which will be easier to use instead of doing all the below setup for iron.io?

1. Configure logproxy
2. Get the code docker pull philipssoftware/logproxy
3. Create a new micro service using the keys and binding to log proxy service
4. Do we need a rabitmq service also?
5. Cf push the micro service which will enable iron.io logs to push to Kibana
6. Need help how iron.io config will use the proxy service?

Ideally want to have a small design document explaining the components and why this exists and if there were any design choices to be made that influenced it.

• Design Document
• Pull Request Template
• Way or Working document (improve the existing documentation)
• Static Code Analysis
• Build pipeline incorporating Integration Tests, etc. prior to merge into mainline.
• Badges

In our Healthdot Kibana logging, we see certain logs being duplicated and others missing.

E.g. The following snippet of logging:

Resulted in the following duplication in Kibana:

It seems that when the duplication occurs, logproxy itself logs a sequence of messages such as:

The error that occurred doesn't seem to be logged, so it is currently unclear what the root cause is.

Ability to run logproxy in following modes:

• web
• worker
• web and worker together.

To deal with building up queues.

Authenticated endpoints for

• Health Monitoring: System Health HTTP Endpoint (includes health of RMQ or dependency services)
• Metrics: Prometheus style (probably have to look at a Datadog Agent/New Relic agent support later)

Support for a Vault/Spring Cloud Config endpoint for a central configuration.

For our product (PICS / Cardiology Reporting) we are updating all base images to have the latest/best image with the latest security fixes.
Can we update the docker base image
FROM golang:1.20.1-alpine to golang:1.21.6-alpine or golang:1.21.6-alpine3.19?

• Add support for service identities with LOG.CREATE scope
• Eliminates the need for outdated shared key / secrets API signing

As a System Operator using logproxy to drain logs
I want to monitor my running logproxy instances using Datadog APM
so that I can monitor, trend and respond to issues occurring in my environment.

Refer to https://docs.datadoghq.com/tracing/setup/go/ for information on how to integrate tracing.

• Provide more succinct error message in case of config issues
• e.g. do a test connect to verify connectivity

When providing DhpLogEvents, any characters that would be deemed invalid by HSDP Logging are replaced. However, when providing a LogEvent resource this is not the case.

E.g. The following LogEvent is passed to HSDP logging with invalid characters in the custom field, resulting in an error:

Note that logproxy can also generate LogEvents with invalid values. The LogEvent processing adds some fields with empty string values if they are not provided in the incoming resource (we noticed this for the originatingUser, serviceName, category and component fields). As empty string values are not allowed, this results in errors from HSDP. Perhaps any added fields should get a default value if no value is provided in the incoming resource (for now we've added some defaults to our own code to avoid this).

This is not intended behaviour, so needs to be removed

Dependabot couldn't parse the go.mod found at /go.mod.

The error Dependabot encountered was:

go: github.com/spf13/[email protected] requires
	github.com/grpc-ecosystem/[email protected] requires
	gopkg.in/[email protected]: invalid version: git fetch -f origin refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /opt/go/gopath/pkg/mod/cache/vcs/748bced43cf7672b862fbc52430e98581510f4f2c34fb30c0064b7102a68ae2c: exit status 128:
	fatal: The remote end hung up unexpectedly

View the update logs.

Hi,
Log proxy is not able to parse logs which are not in LogEvent format and the logs are mapped to random guid values.
However it works fine with log drainer.
Application side logs:
2022-11-11T21:41:20.83+0530 [APP/PROC/WEB/0] OUT Request GET /sample/logtest1 => 200
2022-11-11T21:41:20.83+0530 [APP/PROC/WEB/0] OUT {"resourceType":"LogEvent","id":"a996e783-891b-4c88-bcf0-cf9b131751eb","logTime":"2022-11-11T16:11:20.836Z","category":"TraceLog","eventId":"0","component":"Logging.RequestLoggingMiddleware","transactionId":"","originatingUser":"User","logData":{"message":"Request GET /sample/logtest1 => 200, ThreadName: Thread 40, ThreadId: 40"},"severity":"Information","applicationName":"samplelogapp","applicationVersion":"54c35af5-0894-4e94-bca8-3f86e190832b","serviceName":"client-edi-foundation.hds-foundation-dev.samplelogapp","serverName":"DefaultServer","applicationInstance":"3910d915-98e1-4071-911e-76908ab753f6"}

Sample Log:

Logs on kibana:

• Add Integration Tests (Support a HTTP Mock scenario by default with option to send to a real endpoint)
• Link to pipeline
• Performance Tests

Dependabot couldn't parse the go.mod found at /go.mod.

The error Dependabot encountered was:

go: github.com/spf13/[email protected] requires
	github.com/grpc-ecosystem/[email protected] requires
	gopkg.in/[email protected]: invalid version: git fetch --unshallow -f origin in /opt/go/gopath/pkg/mod/cache/vcs/748bced43cf7672b862fbc52430e98581510f4f2c34fb30c0064b7102a68ae2c: exit status 128:
	fatal: The remote end hung up unexpectedly

View the update logs.

Dependabot can't resolve your Go dependency files.

As a result, Dependabot couldn't update your dependencies.

The error Dependabot encountered was:

github.com/philips-software/logproxy/handlers: cannot find module providing package github.com/philips-software/logproxy/handlers
	github.com/philips-software/logproxy/queue: cannot find module providing package github.com/philips-software/logproxy/queue
	github.com/philips-software/logproxy/shared: cannot find module providing package github.com/philips-software/logproxy/shared

If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.

View the update logs.

Dependabot can't resolve your Go dependency files.

As a result, Dependabot couldn't update your dependencies.

The error Dependabot encountered was:

github.com/philips-software/logproxy/handlers: cannot find module providing package github.com/philips-software/logproxy/handlers
github.com/philips-software/logproxy imports
	github.com/philips-software/logproxy/queue: cannot find module providing package github.com/philips-software/logproxy/queue
github.com/philips-software/logproxy imports
	github.com/philips-software/logproxy/shared: cannot find module providing package github.com/philips-software/logproxy/shared

If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.

View the update logs.