airbnb / plog Goto Github PK

As soon as that feature is ironed out and used in production, we should document them filters properly.

Verify that each UDP listener has its own receiving threads.

It would be useful to expose minimal histograms of GC stats.

See ManagementFactory.getGarbageCollectorMXBeans()

Proposal to be reviewed:

• Use \0 as a line prefix for advanced protocol.
• Advanced protocol supports commands, including stats.
• Advanced protocol allows to specify tags, ask for acknowledgement(?).

Based on #10.

• One extra field (2 bytes) in the header is for the metadata length.
• The metadata is prepended to the payload.
• The metadata consists of one MessagePack object (we could add more afterwards), an array of strings, called "tags".
• Those tags get mapped to byte[][] in Message.
• Tags from the last processed fragment win; well-behaving clients only set tags on the last fragment to avoid reducing the chunk size or increasing packet count.

Netty offers a native transport.
It should soon allow for great perf improvements by using recvmmsg/sendmmsg.
See netty/netty#2719 for early support.

Sadly Plog currently hardcodes the transport to Nio. This should be configurable.

See ManagementFactory.getMemoryMXBean().

If messages reach the end of the pipeline, we should do something about it.

Separate but similar to end of pipeline exceptions counting mechanism?

• Allows multiple buffers, one per app would be a good practice
• Per-port Kafka topics

• Removes the dependency to Kafka
• Removes a bunch of code from plog-server into separate modules
• Unifies the model around Netty 4 channels

Would break the config format.

~~README.mdon https://github.com/ajoberstar/gradle-git should cover it.~~ (doesn't supportgit describe`-like naming)

Tags like v4.0.0 should turn version into "4.0.0", and non-tagged releases should contain an unique suffix.

This will make it easy to spin separate projects off, like plog-upshot.

With min_size=1, max_size=1, size_increments=1,

Exception in thread "main" java.lang.RuntimeException: No sizes! Decrease plog.stress.size_increments

• Integration tests should pass
• We shouldn't play around around with the console as we can trivially have our own in-memory "sink"

Stats are a bit limited right now; per-port stats would be really useful, but keeping global stats would be nice too.

Currently, each UDP and TCP port has to be fully configured.
Offer UDP & TCP default configuration sections that get merged with each port at startup.

We should keep Plog simple and focused. However app-specific filters could provide value: fire-and-forget eventing in apps is great, but when events need some extra work, there's currently no good way to keep that work local (hence distributed horizontally and away from network spies) yet out of the app workers.

Such tasks could include filtering out sensitive information, converting messages between formats, annotating them with information about the host or its state, etc.

We could have per-port filters, with the following design decisions for a first shot:

• In the port config, add the optional:

  filters = [
    { provider: "your.class",
      ...
    }, ...
  ]
  

• your.class would be found in the classpath, have to implement:

  interface FilterProvider { PlogFilter getHandler(Config cfg); }
  

• And PlogFilter in turn:

  interface PlogFilter extends ChannelHandler { /* extras? */ }
  

This model would allow handlers to hold state, turn each message into zero or more messages, let exceptions get logged and counted in our stats (TBV).

Questions left out of this initial implementation, hopefully forever as they would add complexity:

• Filter metrics reporting (poll by adding to the interface, push via the pipeline?)
• ?

The stats format changed with the move to custom handler chains.

Adapt the datadog plugin, make use of the YAML config file to specify what should be reported.

Allow filters to report stats.

Netty allows us to bind multiple times, which is a very useful performance improvement for UDP sockets under Linux as it allows for multiple listeners.

In that case, it would be useful to allow threads = 0 and skip the thread pool altogether in UDPListener, to avoid the dispatch overhead (including extra context switches).

• If a message has kt:foo and kt:bar tags, forward the message to the foo and bar topics.
• Configuration now specifies a default topic, used for messages that do not have such tags.
• If the default topic is not set, a warning is printed on startup and messages are ignored (allows for more clever pipelines)
• The magical value null can be used for the default or tag to avoid sending to Kafka, for example for warmup or to use the same port for multiple sinks and client-side routing.

So no need to firewall off.

... so we can sum them into a single metric.

System properties aren't that great.

Could be useful to companies relying on JMX for monitoring, eg via Jolokia.

The classpath should match testCompile in plog-distro.

This could be very useful to quickly hack around stuff.