@aliusmiles commented on Thu Jun 24 2021
Current Vector Version
Use-cases
Vector agent reading AWS ALB logs from S3 bucket and pushing them to Loki.
Attempted Solutions
vector-agent running as scalable Deployment instead of DaemonSet.
Proposal
Right now Helm chart for vector-agent uses DaemonSet and is rather hardcoded for only one role: collecting container logs from k8s nodes. In my case Promtail is already doing that so I want to use Vector to only get LB logs from s3 bucket into Loki. I've already tested that locally with minimal effort and minimal config, which is just awesome (especially out-of-the-box parse_aws_alb_log transformer) :-) But when it got to deployment I've realized official chart won't suite my case at all: it's possible to add custom config and disable default sources and sinks, but running vector as DaemonSet for that just doesn't make sense.
I imagine this might be quite common case - using vector for collecting logs from sources other than container logs or even running multiple agents - one for container logs as daemon, another for s3 logs as single replica, others for something else as auto-scalable deployment.
I remember seeing some chart with had templates for both DaemonSet and Deployment, controlled by variable like deploymentType: daemonset|deployment. Would be nice to have something like this in vector-agent chart. Should be relatively simple to introduce additional variable and template, although some extra changes might be needed, for example: Deployment for collecting logs from s3 does not really need /var/log mounted from k8s node, but still might need volume for data-dir, if type is set to Deployment then optionally add hpa, etc.
I believe such feature would add much more flexibility to official chart and allow it to be used in more scenarios. If anybody is interested I can try to draft a PR for that.
@spencergilbert commented on Thu Jun 24 2021
Hey! I think your use case would be suited for our vector-aggregator chart. We're currently working on getting it ready to leave beta, but it's definitely usable in it's current state.
Had you seen that chart yet?
@aliusmiles commented on Thu Jun 24 2021
Just checked it out, I should be able to use it for my case, thanks! Just need to disable default sources/sinks/service and add custom config. I have actually saw that chart earlier but haven't looked inside as I was under impression that it can only aggregate data from vector agents.
The only thing that it's a StatefulSet, so no hpa. Doesn't matter for me but if someone needs to get a lot of logs from s3 or other source, I believe deployment with hpa would've been better.
Possible solution: just add yet another chart for 'vector-forwarder' or something? :) basically vector-agent without host volumes but with deployment and hpa? Though I'm not sure how much demanded that would be, maybe leave this issue open for a while to see if there will be any upvotes?
My personal case should be solved with aggregator chart I think, will try tomorrow. Thanks for the suggestion!
@spencergilbert commented on Thu Jun 24 2021
👍 @aliusmiles that's definitely a good point. At risk of supporting too many charts that are similar but a little different it would be nice to support HPAs. We did originally choose StatefulSet for the persistent volumes.
I'm happy to leave this open to see if there's general interest in a more "ephemeral, but scalable" strategy. Or maybe HPAs will support StatefulSets one day 😄
@aliusmiles commented on Fri Jun 25 2021
aggregator chart worked quite nicely for me, thanks!
One minor suggestion: maybe wrap headless-service template in the same condition as service? Since I've disabled default source, I don't have service in my deployment but chart still deployed headless service with no ports. Not a big issue but still.
Also, is there any issue or something to keep track on when vector-aggregator chart will be by fully released?
P.S. Here's my configs in case someone get interested:
Configs
values.yaml
replicas: 1
image:
tag: "0.14.0-alpine"
nameOverride: "vector"
serviceAccount:
create: true
annotations:
eks.amazonaws.com/role-arn: IRSA_ROLE_ARN
tolerations: []
storage:
mode: managedPersistentVolumeClaim
managedPersistentVolumeClaim:
size: 10Gi
vectorApi:
enabled: false
vectorSource:
enabled: false
internalMetricsSource:
enabled: false
prometheusSink:
enabled: false
haproxy:
enabled: false
sources:
s3_alb_logs:
type: "aws_s3"
region: "AWS_REGION"
compression: "gzip"
sqs:
queue_url: "SQS_QUEUE_URL"
transforms:
alb_logs:
type: "remap"
inputs:
- "s3_alb_logs"
source: " . = parse_aws_alb_log!(.message)"
sinks:
loki:
type: "loki"
inputs:
- "alb_logs"
endpoint: "http://loki:3100"
encoding:
codec: "json"
labels:
forwarder: "vector"
event: "{{ event_field }}"
source: "alb_logs"helmfile.yaml
repositories:
- name: timberio
url: git+https://github.com/timberio/vector@distribution/helm?ref=master
releases:
- name: vector
chart: timberio/vector-aggregator
namespace: monitoring
values:
- envs/{{ .Environment.Name }}/values/vector.yaml
@spencergilbert commented on Fri Jun 25 2021
@aliusmiles thanks! I'll take a look at that and open an issue based on what I find.
I don't have a hard date for that yet, but it's one of our largest focuses. We'll definitely do a blog post/highlight/tweet/etc when it happens :)
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent