Veneur CPU usage based on metric packets processed per second. about veneur

[prudhvi]

cc @aditya-stripe @antifuchs Does this CPU usage look normal/expected to you?

[ChimeraCoder]

That's not expected. Could you share a CPU profile so we can see where that's getting clogged up?

[prudhvi]

Cool thanks, am curious how far are we off ? Should it be doing double this or 10x this?

profile_120sec.pdf
profile_300sec.pdf

We have flush interval of 30 seconds and run veneur as a sidecar to app in LXC containers. We have the following config for veneur. Also FYI we do run a fork of veneur where we do some metricname parsing and renaming before we flush to signalfx in the sink signalfx.go.

  num_readers: 20
  num_span_workers: 10
  num_workers: 96
  read_buffer_size_bytes: 2097152
  metric_max_length: 32768
  span_channel_capacity: 100
  metric_max_length: 32768
  trace_max_length_bytes: 16384
  ssf_listen_addresses:
    - unix:///data/app/tmp/veneur-ssf.sock
  statsd_listen_addresses:
    - unixgram:///data/app/tmp/veneur.sock

I am attaching the CPU profile for 120 seconds and 300 seconds.

[prudhvi]

@aditya-stripe , Sorry before spending more time into investigating bottleneck for CPU usage, am curious to know how much is expected throughput for 1 logical CPU in veneur for metrics consumption via statsd.

I noticed similar metric processed throughput per 1 logical CPU when using UDP too.
60k/sec is what the https://github.com/stripe/veneur#benchmarks says too but wanted to confirm , is that a old number that needs updating?

[prudhvi]

Also attaching the UDP profiles (when sending metrics via UDP instead of unix domain socket)
udp_profile_300sec.pdf
udp_profile_120sec.pdf