trivago / gollum Goto Github PK

The package currently used for producer.ElasticSearch is using a deprecated protocol and hasn't been updated for at least 2 years. We should maybe switch to https://github.com/olivere/elastic which is not as easy to use but supports the full feature set.
This will probably end up to be a complete rewrite of that producer.

Hi.

I am writing a daemon for sending logs to AWS cloudwatch logs service. My friend showed me your project and I am interested in contributing my code as a plugin. My question is, under which license my contribution needs to be ?

The SimpleProducer does not include an Enqueue function, i.e. it does not implement Modulate functionality. The BufferedProducer / BatchedProducer assume queues which might not be needed by plugins utilising libraries already having queues (e.g. Sarama).
The latter should be able to derive from a DirectProducer type which implements the enqueue function that implements modulation and forwards to a given callback.

This issue is related to PR #117 and a subtask from #82

We need a new formatter which is be able to copy/set meta data to a message.

Evaluate if https://github.com/sirupsen/logrus can be a sufficient replacement for tlog.
We could benefit from several features brought by this package + external developers most likely already know it.

• log scopes must be possible
• logrus must connect to the gollum pipeline
• logs before startup and after shutdown must be handled correctly

Our Datacenter Operations team switched the backbone interface on a server from physical to logical.

This means that the interface gollum used to send data to Kafka went down and came back up later as a different device
This did not trigger a reconnect to the Kafka cluster.

This ticket was created at 10.03.2016 15:47 in out internal bug tracker.
Right now i don't have details which gollum version we were using. I guess the OS of the Server was FreeBSD.

So it might be that the bug was already fix. But i thought it make sense to write it down here and to check this again to confirm that this was fixed already.

running docker run -it --rm trivago/gollum
results in docker: Error response from daemon: oci runtime error: exec: "/go/src/gollum/gollum": stat /go/src/gollum/gollum: no such file or directory.

current
ENTRYPOINT ["/go/src/gollum/gollum"]

should be
ENTRYPOINT ["/go/src/github.com/trivago/gollum/gollum"]

or how about 'go install' after 'go build'?

After an internal discussion we came to the point that the current Gollum 0.5.0 format can be improved further in terms of maintainability and reusability.
As we're pushing a config change with 0.5.0 anyway it does make sense to add this on top of the current changes even though they have not been released yet.

Proposed example:

WriteToKafka:
  Type: Producer
  Stream: "*"
  Modulators:
    - filter.Rate
    - format.Hostname
    - format.Envelope
      PostFix: "\r\n"
  Plugins:
    - producer.Kafka:
      Broker: ["192.168.0.1","192.168.0.3","192.168.0.2"]
      Topic: "*"
    - producer.Elasticsearch:
      Master: "192.168.10.1"
      Index: "*"

To achieve this we have to adjust the base types and separate the actually plugin configuration from the "standard framework" configuration.
Creating multiple plugins out of a single config block can be done by generating multiple config blocks from one block in the config parsing section. As an example: the config above would be separated into two blocks with a producer.Kafka as single item and a producer.Elasticsearch as single item.

Given a Gollum configuration:

"HttpOut01":
    Type: "producer.HTTPRequest"
    Streams: "http_01"
    Address: "http://localhost:8099/test"
    RawData: false

The (string) .port property in the resulting prod.HTTPRequest struct is set to anything following the semicolon in the Address setting, instead of the port number -- in this case, 8099/test.

The documentation is ambiguous as to how the path component of the Address URL is supposed to get parsed and used by Gollum.

In practice, the string value .port seems to get catenated back into the struct's .address property, which is then used by .isHostUp() and .sendReq(). The.port property itself isn't used.

With this usage, it could be better to just validate the address's syntax properly and use the value verbatim.

Also, the URL paths' semantics (i.e. that they're ignored by both the HTTP consumer and producer) should be more explicit.

PR submitted: #75

GoWork/src/github.com/trivago/gollum$
go build
# github.com/trivago/gollum/consumer
consumer/http.go:170: undefined: body
consumer/http.go:170: undefined: err
consumer/http.go:171: undefined: err
consumer/http.go:173: undefined: err in err.Error
consumer/http.go:178: undefined: body

The examples and documentation for the Kinesis producer seem to be slightly wrong. I think it should be 'StreamMapping' and not 'StreamMap'.

Currently there is a lot of confusion in the wording between the "stream" concept and the "stream" plugins. As of this we want to rename all "stream" plugins to "router".

To be a little bit more precise on this issue.
Stream plugins were added on top of the original stream concept. Streams are more or less a "name tag" for messages or a name for the pipeline a message goes through.
The original idea behind stream plugins was to create a "stream configuration" but the concept quickly grew beyond just that. This lead to more and more confusion over time. In conversations it was never really clear if stream/piplines or stream/plugins were meant.
In addition to that we saw that with 0.5.0 the stream plugins could be dedicated to routing and filtering only which brought us to the name "router".

So the idea behind this task is that routers should only do two things:

• message filtering (like a firewall)
• message forwarding (like a router)

As the "firewall" method can be implemented as a routing mechanism, too (forward to null), the term "router" makes even more sense. Also see #81 for another step in that direction.

Docker supports various log drivers to gather and forward logs. Just to name a few:

• CloudWatch
• Fluentd
• Splunk
• ...

This ticket is about starting a discussion about a (potential) gollum log driver.
An alternative idea could be to build upon the syslog or json-file log driver and add a chapter into the gollum documentation how to use those things.

The current architecture documentation (read the docs RST files) are still from the 0.3.x era.
Change them according to the new structure and add FAQ about architecture by plugin developers.

• High level architecture of message passing
• Explaining the "stream width" (ModulatorQueues)
• Purpose of the different plugin types
• Components, purposes and type hierarchy
• Basic implementation examples

Update - Open tasks before v0.5.0 release:

• Finalise release notes
• Describe Metadata handling in Terminology
• Add Guide to update plugins from v0.4 to v0.5
• Run gollum benchmark of v0.4 VS v0.5
• Add docs for metrics
• Add docs for healthchecks
• Describe buffered- and batched-producer components (optional)

This issue is related to PR #117 and a subtask from #82

Goal is to update all modulator for optional meta data usage:
ApplyTo: "payload" # [payload, meta:key]

Steps to reproduce the problem:

1. Use the following gollum config:

- "consumer.Http":
    Address: "localhost:9090"
    WithHeaders: true

- "producer.Console":
    Formatter: "format.Envelope"
    EnvelopePrefix: "[start]"
    EnvelopePostfix: "[end]"

2. Issue the following HTTP request:

wget https://uploads.maximilianhils.com/2017-03/repro
cat repro | nc localhost 9090

3. Observe that output correctly ends with "<<<":

$ gollum -c repro.conf
[start][snip http headers]>>>>>>>>>>>>[snip content]<<<<<<<<<<<[end]

4. Set WithHeaders to false, restart gollum
5. Issue the same HTTP request again. The message body is now truncated:

$ gollum -c repro.conf
[start]>>>>>>>>>>>>[snip content]NoZW1hIiB4bWxuczpjd21wPSJ1cm4[end]

I want to use Sourcegraph for gollum code search, browsing, and usage examples. Can an admin enable Sourcegraph for this repository? Just go to https://sourcegraph.com/github.com/trivago/gollum. (It should only take 30 seconds.)

Thank you!

The version of github.com/aws/aws-sdk-go is pinned in ./glide.yaml to the repo's commit ID because the sources we have under ./vendor/github.com/aws/aws-sdk-go do not match the library's intended version (1.8.2).

We should set the version to something meaningful, do a glide update, and then verify that the plugin(s?) using that library still work.

Commit 4068106

There might be a bug in the socket producer code.

error log snippet

ERROR: socket.go:171: Socket response error: read tcp 172.17.0.2:57002->my-mac-ip:5880: i/o timeout

Observations

• gollum says that error log every time after flushing(producing) packets into the socket.
• But, on the receiving side (which is listening to 5880 port in this case), packets arrive intact. There is no losing packet as far as I observed.

Envirionment

• macOS Sierra , CoreOS 1010.5 (occurs on linux and osx both)
• Docker engine 1.12.1
• gollum v0.4.4

If you need additional info, let me know.
Thanks.

For this, all sockets and file handles need to be passed to a new process that is started by the active Gollum process (no fork!).

The main challenge will be to correctly reassign the handles to the corresponding plugins in the new Gollum instance.

Config changes are not allowed when doing this.

Maybe we can use GOB + unix domain sockets to transfer data between both processes.

Important: This is just an idea and this ticket can be used for a discussion. You have ideas? Feel free to add them here. You have concerns? Add a comment! Or maybe you have an idea how to develop it? Feel free to run a prototype.

For this feature, the architecture change that is coming in v0.5 is required.

Configs should be reloadable by some kind of unix signal or by socket communication (mini protocol).
The modification part should be done by calling Configure() again. Plugins will have to react accordingly.

Server part:

• Password protected (provided via command line parameter, SHA512 encoded)
• Add plugin: id + config
• Remove plugin: id
• Modify plugin config: id + config
• Reset all (like signal part)

Signal part:

• Use any not-yet-used signal like SIGUSR
• Create a diff between current state and config loaded
• Add new plugins
• Reconfigure existing plugins
• Remove all plugins not existing in new config
• Start new producers
• Start new consumers

Important: This is just an idea and this ticket can be used for a discussion. You have ideas? Feel free to add them here. You have concerns? Add a comment! Or maybe you have an idea how to develop it? Feel free to run a prototype.

For this feature, the architecture change that is coming in v0.5 is required.

We introduced a fuse mechanism some time ago.
The idea was to have a circuit break mechanism in place that allows producers to stop consumers from pushing new messages into the system. Producers ought to "burn" a fuse in certain high pressure situations and "replace" it when pressure has been removed from the system.

In practices we saw several problems with that approach

1. Producers have to identify situations where it makes sense to burn a fuse. This turned out to be a tricky question and most plugin authors simply ignore that feature probably because of this
2. Consumer need to react on burned fuses. Fuses cannot be integrated well into the core because consumers need to react differently on a burned fuse.
3. Fuses tend to convert non-blocking consumers into blocking consumers potentially breaking systems further up (i.e. the systems sending data to gollum)

So in short: Developers were either not aware of the feature and/or it was to confusing to use. It introduced unwanted side effects and missed the purpose in many situations.

As of this I would rather see this feature being removed and maybe replaced by another system that is more dynamic like reactive streams.

See Formula Cookbook and Acceptable Formulae.

How this can look like you can get this from the What I Talk About When I Talk About CLI Tool By Golang #gocon presentation.

I wiki/cookbook of common patterns would be really helpful.

Seems to be missing some newlines which makes it harder to read.

The ForEachStream method in the stream registry can trigger a race.
The intention of the following code is to copy the map, but instead it creates a reference.
See https://github.com/trivago/gollum/blob/master/core/streamregistry.go#L168

registry.streamGuard.Lock()
routers := registry.routers
registry.streamGuard.Unlock()

AFAIR this pattern also exist in other locations in the code.
Those should also be fixed.

the implementation of native.Systemdconsumer only calls sd_journal_open once, and the resulting journal object does not pick up new journal files after rotation, so after the systemd log rotates the consumer halts without any errors.

Add a new commandline option -cc to convert a config from old to new format. This should be a best effort conversion (i.e. manual work might be required afterwards)

This ticket was created internally at 01.02.2016 17:02 from @arnecls

A lot of plugins use the batching concept on top of a BufferedProducer.
This should be generalised, especially as a Batch does not require a buffer.

• Have a look at producer.File and similar implementations to extract the pattern into a common base type
• Derive all batch-style producers from this new base type

I stumbled upon https://github.com/goreleaser/goreleaser.
It looks pretty decent for releasing go applications. And it has native Homebrew support.

You can configure it via a small yml file inside your repository.
Here is an example from Hugo: https://github.com/spf13/hugo/blob/master/goreleaser.yml

What is your opinion?

Currently, as of version 0.3.2, you have to define the Loopback consumer if you want to use the DROPPED stream mentioned in the ChannelTimeoutMs parameter of the Kafka producer.

Because there is no case in which you would not want to use the DROPPED stream with a ChannelTimeoutMs set higher than 0, it would make sense to have the loopback-consumer functionality for the DROPPED stream from the get-go.

Example config which doesn't work right but would with the proposed change:

- "consumer.Console":
    Enable: true
    Stream: "textStream"

- "producer.Kafka":
    Enable: true
    Channel: 65535
    ChannelTimeoutMs: 10
    Stream:
      - "textStream"
    Topic:
      "textStream": "webserver_textStream"
    Servers:
      - "192.168.0.5:9092"
      - "192.168.0.6:9092"
      - "192.168.0.7:9092"
    Formatter: "format.Timestamp"
    Timestamp: "20060102150405 "

- "producer.File":
    File: "/appdata/gollum/dropped/textStream.log"
    Stream:
      - "_DROPPED_"
    Formatter: "format.Envelope"
    Rotate: true
    RotateSizeMB: 512
    RotateAt: "00:00"
    Compress: true

Example which does work using the LoopBack consumer:

- "consumer.Console":
    Enable: true
    Stream: "textStream"

- "consumer.LoopBack":
    Enable: true
    Channel: 8192
    Routes:
      "_DROPPED_": "dropped"

- "producer.Kafka":
    Enable: true
    Channel: 65535
    ChannelTimeoutMs: 10
    Stream:
      - "textStream"
    Topic:
      "textStream": "webserver_textStream"
    Servers:
      - "192.168.0.5:9092"
      - "192.168.0.6:9092"
      - "192.168.0.7:9092"
    Formatter: "format.Timestamp"
    Timestamp: "20060102150405 "

- "producer.File":
    File: "/appdata/gollum/dropped/textStream.log"
    Stream:
      - "dropped"
    Formatter: "format.Envelope"
    Rotate: true
    RotateSizeMB: 512
    RotateAt: "00:00"
    Compress: true

Some producers implement metrics based on their input streams.
Production showed that this is something required for all streams to have a better overview and to be able to debug problems.

Metrics should be implemented on router level.

The package source code in the vendor directory is missing the license information. This might be related to FiloSottile/gvt#46 or FiloSottile/gvt#23
This is probably violating most of the licenses used by packages in the vendor directory.

Sarama cluster has support for consumer groups: https://github.com/bsm/sarama-cluster

Note sarama-cluster requires kafka >= 0.9.

It seems that prod.format is never initialized and that causes a panic:

Debug: Loading consumer.Kafka
Debug: Loading stream.Broadcast
Debug: Loading producer.Redis
Debug: Loading stream.Broadcast
Debug: Configuring stream.Broadcast for bidrequests
Debug: Configuring stream.Broadcast for events
Debug: Using fallback stream for *
Debug: Configuring producer.Redis
Debug: Using fallback stream for _DROPPED_
Debug: Configuring consumer.Kafka
Debug: Starting *producer.Redis
Debug: Starting *core.LogConsumer
Debug: Starting *consumer.Kafka
We be nice to them, if they be nice to us. (startup)
Initializing new client
client/metadata fetching metadata for all topics from broker localhost:9092
Connected to broker at localhost:9092 (unregistered)
client/brokers registered new broker #0 at swarm-wily64:9092
Successfully initialized new client
Connected to broker at swarm-wily64:9092 (registered as #0)
consumer/broker/0 added subscription to rtb.events/0
client/metadata fetching metadata for all topics from broker localhost:9092
PANIC:  runtime error: invalid memory address or nil pointer dereference
goroutine 7 [running]:
runtime/debug.Stack(0x0, 0x0, 0x0)
    /usr/local/go/src/runtime/debug/stack.go:24 +0x80
github.com/trivago/gollum/shared.RecoverShutdown()
    /home/vagrant/swarm/go/src/github.com/trivago/gollum/shared/utilities.go:220 +0x389
panic(0xacd200, 0xc8200100c0)
    /usr/local/go/src/runtime/panic.go:443 +0x4e9
github.com/trivago/gollum/producer.(*Redis).getValueFieldAndKey(0xc820076f20, 0xc82023403e, 0xaff, 0xfc2, 0xbadd59fc5937d476, 0xbadd59fc5937d476, 0x7f2a15d854b8, 0xc820090b00, 0xecebad9c0, 0x26a0fdf2, ...)
    /home/vagrant/swarm/go/src/github.com/trivago/gollum/producer/redis.go:148 +0xb6
github.com/trivago/gollum/producer.(*Redis).storeHash(0xc820076f20, 0xc82023403e, 0xaff, 0xfc2, 0xbadd59fc5937d476, 0xbadd59fc5937d476, 0x7f2a15d854b8, 0xc820090b00, 0xecebad9c0, 0x26a0fdf2, ...)
    /home/vagrant/swarm/go/src/github.com/trivago/gollum/producer/redis.go:173 +0x47
github.com/trivago/gollum/producer.(*Redis).(github.com/trivago/gollum/producer.storeHash)-fm(0xc82023403e, 0xaff, 0xfc2, 0xbadd59fc5937d476, 0xbadd59fc5937d476, 0x7f2a15d854b8, 0xc820090b00, 0xecebad9c0, 0x26a0fdf2, 0xfa5ce0, ...)
    /home/vagrant/swarm/go/src/github.com/trivago/gollum/producer/redis.go:113 +0x31
github.com/trivago/gollum/core.(*ProducerBase).messageLoop(0xc820076f20, 0xc8200db6f0)
    /home/vagrant/swarm/go/src/github.com/trivago/gollum/core/producer.go:485 +0xc4
github.com/trivago/gollum/core.(*ProducerBase).MessageControlLoop(0xc820076f20, 0xc8200db6f0)
    /home/vagrant/swarm/go/src/github.com/trivago/gollum/core/producer.go:584 +0x63
github.com/trivago/gollum/producer.(*Redis).Produce(0xc820076f20, 0xc8200db3d0)
    /home/vagrant/swarm/go/src/github.com/trivago/gollum/producer/redis.go:249 +0x38f
main.multiplexer.run.func1()
    /home/vagrant/swarm/go/src/github.com/trivago/gollum/multiplexer.go:383 +0x3f
github.com/trivago/gollum/shared.DontPanic(0xc8200d9de0)
    /home/vagrant/swarm/go/src/github.com/trivago/gollum/shared/utilities.go:233 +0x3a
created by main.multiplexer.run
    /home/vagrant/swarm/go/src/github.com/trivago/gollum/multiplexer.go:384 +0x727

Master betrayed us. Wicked. Tricksy, False. (signal)
Filthy little hobbites. They stole it from us. (shutdown)
Debug: Closing consumer *core.LogConsumer
Debug: Closing consumer *consumer.Kafka
Debug: Waiting for consumers to close
Debug: Recieved stop command
Closing Client
Closed connection to broker localhost:9092
Closed connection to broker swarm-wily64:9092
consumer/broker/0 disconnecting due to error processing FetchRequest: kafka: broker not connected
kafka: error while consuming rtb.events/0: kafka: broker not connected
It's the only way. Go in, or go back. (flushing)
Debug: Closing producer *producer.Redis
Debug: Waiting for producers to close
Debug: Recieved stop command
Debug: All dependencies resolved

Gollum v0.3.2, FreeBSD 10.1 (vm)

I am currently trying to enable logging gollum errors and warnings into a logfile.

My test-config is the following:

- "consumer.Console":
    Enable: true
    Stream: "test_stream"

- "producer.File":
    File: "/appdata/gollum/stream/test_stream.log"
    Stream:
      - "test_stream"
    Formatter: "format.Envelope"
    Rotate: true
    RotateSizeMB: 512
    RotateAt: "00:00"
    Compress: true
# Trying to cause a warning with the following line
    foo: bar

- "producer.File":
    File: "/var/log/gollum.log"
    Stream:
      - "_GOLLUM_"
    Formatter: "format.Envelope"
    Rotate: true
    RotateSizeMB: 512
    RotateAt: "00:00"
    Compress: true

Using loglevel 1 will cause the warning to be printed onto my console and leave me with an empty/non existing logfile.

Warning: pluginconfig.go:58: Unknown configuration key in producer.File: foo

Using loglevel 3 will still cause the warning to be printed onto my console, but log the startup and shutdown messages into my log-file:

Logfile:

We be nice to them, if they be nice to us. (startup)
Master betrayed us. Wicked. Tricksy, False. (signal)
Filthy little hobbites. They stole it from us. (shutdown)

Console:

Warning: pluginconfig.go:58: Unknown configuration key in producer.File: foo

Is there a possibility to write all errors/warnings into a logfile without having to redirect stderr via the console-command?

Currently we have a lot of cases were messages are separated into key and value (e.g. after reading from kafka) or where a key has to be generated from a payload (routing, redis, kafka, etc.).
Along with that comes the need to modify keys separately that lead to different approaches and config + code bloat.
By adding keys to messages we will reduce processing overhead and can built much more streamlined pipelines that are easier to understand and debug.

See/discuss changes in comments

• Add a "key" to the Message struct ([]byte)
• Consumers reading from key/value stores should set the key
• Producers requiring a key should take the key field into account (can be empty)
• Modulators for keys must be possible (ideally by defining an ApplyTo field or sth. similar)
• Implement a "Copy" formatter that allows copying data between key and value fields

Are there any plans to use chat tool like gitter for this repo for collaboration?

As discussed in #100, the internal tool used to create Gollum's documentation should be rewritten / given a thorough cleanup prior to revisiting the documentation itself.

Currently formatters are able to modify the stream of a message.
With the changes imposed by #80 and #82 we could remove that functionality again.

Thoughts so far:

• If a formatter currently does "routing and stripping" we could put the stripped part to the key and then route based on the key in a router
• Possible problem: Preserving the key?
• It makes sense that routers to routing, not that much that formatters do it
• In that regard the different component take over the following purpose
  • consumer: converting messages into a common exchange format
  • router: change origin based on payload / metrics
  • producer: convert message into a specific service format

The intention of this task is to ease development for plugin writers regarding spooling support and adds more consistency to the way messages are modulated and forwarded.
By removing the support for message modification from the routers we can define an "original" to be generated by the consumer and can be sure no additional modification has been made before the message reaches a producer. By dropping only original messages we avoid double-formatting of messages e.g. sent to a spooling producer.

• The message generated from by a consumer AFTER applying modulators is seen as "original"
• A function "CloneOriginal" has to be implemented in the Message struct
• Remove formatter support from routers (see #80)
• When dropping a message a clone of the original is ALWAYS dropped (with streams set correctly)
• Modulating of messages has to move out of the concrete producer implementation unless SimpleProducer is used as a base struct (i.e. as of the current state BufferedProducer calls Modulate)
• Storing / generating originals has to be removed from all producers doing such a thing

The term Drop lead to a lot of confusion.
Most people thing of "drop" as an destructive operation (like in mysql, ip, etc.). As of this drop was mistaken as a discard in the past quite often.
The "Drop" mechanism was originally thought of as kind of an exception handling mechanism: If you cannot deliver a message, drop it, so somebody else might catch it. So basically it implements a "retry by routing" operation.

• Find all occurrences of "Drop*" and replace them by "Retry*"
• Make it clear that this operation is a "retry in case of error" in the documentation

Hi, I intended to extend the HTTP consumer with basic auth. Unfortunately I am struggling in figuring out where to register new config values I added to the consumer. Code can be found here: https://github.com/glaslos/gollum/commit/bfacc98f93e13968debea892426f2c1390857f75
I'd appreciate some feedback.

Please add support for fluent protocol consumer based on https://github.com/fluent/fluentd-forwarder

When using the revision of elastigo specified in the godep manifest, I get the following error:

🌰  GOPATH=$(pwd) GOBIN=$(pwd)/bin go build -o bin/gollum
# github.com/trivago/gollum/producer
src/github.com/trivago/gollum/producer/elasticsearch.go:221: not enough arguments in call to prod.indexer.Index

Updating it to the latest commit allows gollum to build.

Go 1.5.1. System:

CPU: quad-core 64-bit haswell
OS X: 10.11.1-x86_64
Xcode: 7.1.1
CLT: 7.1.0.0.1.1444952191
Clang: 7.0 build 700
X11: 2.7.8 => /opt/X11
System Ruby: 2.0.0-p645
Perl: /usr/bin/perl
Python: /usr/local/bin/python => /usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/bin/python2.7
Ruby: /usr/local/bin/ruby => /usr/local/Cellar/ruby/2.2.3/bin/ruby
Java: 1.8.0_66, 1.8.0_20

When using AWS STS credentials, producer/s3.go does not know how to switch to the required AWS STS role.

Given a Gollum configuration:

"StdIn":
    Type: "consumer.Console"
    Streams: "console"
    Fuse: "file"

"S3Out":
    Type: "producer.S3"
    Region: "eu-central-1"
    CredentialType: "shared"
    CredentialFile: "/home/myusername/.aws/credentials"
    CredentialProfile: "test_profile"
    StreamMapping:
        "console": "my-testbucket/console"
        "*" : "my-testbucket/asterisk"

And an ~/.aws/credentials file:

[test_profile]
aws_access_key_id = AKI.....
aws_secret_access_key = asdfgh1234
source_profile = default
role_arn = arn:aws:iam::123456789:role/foo-bar
region = eu-central-1
output = table

Gollum will simply try to connect using the default profile of that user and ignores the role_arn setting.

There seems to be support for assuming roles in the AWS Go SDK:

https://github.com/aws/aws-sdk-go/blob/master/aws/credentials/stscreds/assume_role_provider.go

So it could be useful to add support for this in Gollum.

Currently we (and probably others, too) use the metrics provided by gollum to do health checking.
A better way would be to allow plugins to return a health state that can be queried by a given http endpoint.

• Add a new commandline flag to gollum to open up an http endpoint like gollum --health=:80
• A general health check should be added to /health (maybe based on error/warning messages per second)
• Health checks should be added to /health/plugin_id
• Plugins can implement a HealthCheck interface GetHealth() (Status, string) where status can be of ok, warning or critical
• We should discuss wether to use different http status codes and if yes, which ones depending on the returned status

After ctrl-c I got a panic

^CMaster betrayed us. Wicked. Tricksy, False. (signal)
Filthy little hobbites. They stole it from us. (shutdown)
Debug: Closing consumer *core.LogConsumer
Debug: Closing consumer *consumer.Kafka
Debug: Waiting for consumers to close
Debug: Recieved stop command
Closing Client
Closed connection to broker localhost:9092
Closed connection to broker swarm-wily64:9092
consumer/broker/0 disconnecting due to error processing FetchRequest: kafka: broker not connected
kafka: error while consuming rtb.events/0: kafka: broker not connected
It's the only way. Go in, or go back. (flushing)
Debug: Closing producer *producer.Redis
Debug: Closing producer *producer.Kafka
Debug: Waiting for producers to close
Debug: Recieved stop command
Debug: All dependencies resolved
Debug: Recieved stop command
Debug: All dependencies resolved
Producer shutting down.
producer/broker/0 shut down
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x4e0afa]

goroutine 524 [running]:
panic(0xacd200, 0xc8200100c0)
    /usr/local/go/src/runtime/panic.go:481 +0x3e6
github.com/trivago/gollum/producer.(*Kafka).pollResults(0xc820077080)
    /home/vagrant/swarm/go/src/github.com/trivago/gollum/producer/kafka.go:274 +0x72a
github.com/trivago/gollum/producer.(*Kafka).(github.com/trivago/gollum/producer.pollResults)-fm()
    /home/vagrant/swarm/go/src/github.com/trivago/gollum/producer/kafka.go:466 +0x20
github.com/trivago/gollum/core.(*ProducerBase).tickerLoop(0xc820077080, 0xb2d05e00, 0xc8202aa280)
    /home/vagrant/swarm/go/src/github.com/trivago/gollum/core/producer.go:467 +0x5d
github.com/trivago/gollum/core.(*ProducerBase).tickerLoop.func1()
    /home/vagrant/swarm/go/src/github.com/trivago/gollum/core/producer.go:476 +0x32
created by time.goFunc
    /usr/local/go/src/time/sleep.go:129 +0x3a

Gollum adds a sequence number to each message.
Originally the idea was to give messages a kind of "unique" id.
This however does not work because of multiple reasons.

• Sequence numbers are bound to a consumer (i.e. there are collisions between consumers)
• Sequence numbers are reseted after a restart

This means that sequence number cannot be used for anything meaningful.
What it is used for currently is for hashing and loadbalancing. This two usecases can easily be implemented in the router / formatter itself.

• Remove sequence number from messages and all functions associated with it
• Where used replace it with a locally stored number