Seems to be sending junk data and no other messages. about fluentd-kubernetes-sumologic

[stevezau]

Can you paste the log from the fluentd container on STDOUT/STDERR ?

Also, have you pulled the latest version?

Please provide more details on your configuration

[kevin-sumo]

Thanks Steve for taking a look. Here is the config.

apiVersion: extensions/v1beta1 
kind: DaemonSet 
metadata: 
name: fluentd 
labels: 
app: fluentd 
version: v1 
spec: 
template: 
metadata: 
labels: 
name: fluentd 
spec: 
volumes: 
- name: pos-files 
emptyDir: {} 
- name: host-logs 
hostPath: 
path: /var/log/ 
- name: docker-logs 
hostPath: 
path: /var/lib/docker/containers 
containers: 
- image: sumologic/fluentd-kubernetes-sumologic:latest 
resources: 
limits: 
cpu: 50m 
memory: 100Mi 
requests: 
cpu: 30m 
memory: 70Mi 
name: fluentd 
imagePullPolicy: Always 
volumeMounts: 
- name: host-logs 
mountPath: /mnt/log/ 
readOnly: true 
- name: docker-logs 
mountPath: /var/lib/docker/containers 
readOnly: true 
- name: pos-files 
mountPath: /mnt/pos/ 
env: 
- name: COLLECTOR_URL 
valueFrom: 
secretKeyRef: 
name: sumologic 
key: collector-url 
--- 
apiVersion: v1 
data: 
collector-url: {{ sumologic_kubernetes_url|b64encode }} 
kind: Secret 
metadata: 
name: sumologic 
namespace: default 
type: Opaque

[fikander]

@stevezau I'm having the same issue - 1000s of logs as above. I've disabled kube-apiserver log to prevent running out of space in sumo.
It's been deployed yesterday to kubernetes 1.5.2 based cluster, off the master of this repo.

Here's the manifest I used:

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: fluentd
  labels:
    app: fluentd
    version: v1
spec:
  template:
    metadata:
      labels:
        name: fluentd
    spec:
      volumes:
      - name: pos-files
        emptyDir: {}
      - name: host-logs
        hostPath:
          path: /var/log/
      - name: docker-logs
        hostPath:
          path: /var/lib/docker
      containers:
      - image: sumologic/fluentd-kubernetes-sumologic:latest
        name: fluentd
        imagePullPolicy: Always
        volumeMounts:
        - name: host-logs
          mountPath: /mnt/log/
          readOnly: true
        - name: docker-logs
          mountPath: /var/lib/docker/
          readOnly: true
        - name: pos-files
          mountPath: /mnt/pos/
        env:
        - name: COLLECTOR_URL
          valueFrom:
            secretKeyRef:
              name: sumologic
              key: collector-url

This is an example of logs from fluentd:

{"log":"2017-02-15 21:04:55 +0000 [warn]: dump an error event: error_class=KeyError error=\"key not found: \\\"kubernetes\\\"\" tag=\"kube-controller-manager\" time=1487192691 record={\"severity\"=\u003e\"I\", \"pid\"=\u003e\"6\", \"source\"=\u003e\"leaderelection.go:247\", \"message\"=\u003e\"lock is held by ip-172-20-75-85 and has not yet expired\", \"_sumo_metadata\"=\u003e{:log_format=\u003e\"json\", :source=\u003e\"k8s_kube-controller-manager\", :category=\u003e\"kubernetes/kube-controller-manager\"}}\n","stream":"stdout","time":"2017-02-15T21:04:55.371179902Z"}
{"log":"2017-02-15 21:04:55 +0000 [warn]: dump an error event: error_class=KeyError error=\"key not found: \\\"kubernetes\\\"\" tag=\"kube-scheduler\" time=1487192691 record={\"severity\"=\u003e\"I\", \"pid\"=\u003e\"5\", \"source\"=\u003e\"leaderelection.go:247\", \"message\"=\u003e\"lock is held by ip-172-20-32-29 and has not yet expired\", \"_sumo_metadata\"=\u003e{:log_format=\u003e\"json\", :source=\u003e\"k8s_kube-scheduler\", :category=\u003e\"kubernetes/kube-scheduler\"}}\n","stream":"stdout","time":"2017-02-15T21:04:55.372112493Z"}
{"log":"2017-02-15 21:04:57 +0000 [warn]: dump an error event: error_class=KeyError error=\"key not found: \\\"kubernetes\\\"\" tag=\"kube-controller-manager\" time=1487192694 record={\"severity\"=\u003e\"I\", \"pid\"=\u003e\"6\", \"source\"=\u003e\"leaderelection.go:247\", \"message\"=\u003e\"lock is held by ip-172-20-75-85 and has not yet expired\", \"_sumo_metadata\"=\u003e{:log_format=\u003e\"json\", :source=\u003e\"k8s_kube-controller-manager\", :category=\u003e\"kubernetes/kube-controller-manager\"}}\n","stream":"stdout","time":"2017-02-15T21:04:57.37121862Z"}
{"log":"2017-02-15 21:05:00 +0000 [warn]: dump an error event: error_class=KeyError error=\"key not found: \\\"kubernetes\\\"\" tag=\"kube-scheduler\" time=1487192695 record={\"severity\"=\u003e\"I\", \"pid\"=\u003e\"5\", \"source\"=\u003e\"leaderelection.go:247\", \"message\"=\u003e\"lock is held by ip-172-20-32-29 and has not yet expired\", \"_sumo_metadata\"=\u003e{:log_format=\u003e\"json\", :source=\u003e\"k8s_kube-scheduler\", :category=\u003e\"kubernetes/kube-scheduler\"}}\n","stream":"stdout","time":"2017-02-15T21:05:00.371432512Z"}
{"log":"2017-02-15 21:05:01 +0000 [warn]: dump an error event: error_class=KeyError error=\"key not found: \\\"kubernetes\\\"\" tag=\"kube-controller-manager\" time=1487192697 record={\"severity\"=\u003e\"I\", \"pid\"=\u003e\"6\", \"source\"=\u003e\"leaderelection.go:247\", \"message\"=\u003e\"lock is held by ip-172-20-75-85 and has not yet expired\", \"_sumo_metadata\"=\u003e{:log_format=\u003e\"json\", :source=\u003e\"k8s_kube-controller-manager\", :category=\u003e\"kubernetes/kube-controller-manager\"}}\n","stream":"stdout","time":"2017-02-15T21:05:01.371181377Z"}
{"log":"2017-02-15 21:05:02 +0000 [warn]: dump an error event: error_class=KeyError error=\"key not found: \\\"kubernetes\\\"\" tag=\"kube-scheduler\" time=1487192699 record={\"severity\"=\u003e\"I\", \"pid\"=\u003e\"5\", \"source\"=\u003e\"leaderelection.go:247\", \"message\"=\u003e\"lock is held by ip-172-20-32-29 and has not yet expired\", \"_sumo_metadata\"=\u003e{:log_format=\u003e\"json\", :source=\u003e\"k8s_kube-scheduler\", :category=\u003e\"kubernetes/kube-scheduler\"}}\n","stream":"stdout","time":"2017-02-15T21:05:02.371416978Z"}
{"log":"2017-02-15 21:05:04 +0000 [warn]: dump an error event: error_class=KeyError error=\"key not found: \\\"kubernetes\\\"\" tag=\"kube-scheduler\" time=1487192701 record={\"severity\"=\u003e\"I\", \"pid\"=\u003e\"5\", \"source\"=\u003e\"leaderelection.go:247\", \"message\"=\u003e\"lock is held by ip-172-20-32-29 and has not yet expired\", \"_sumo_metadata\"=\u003e{:log_format=\u003e\"json\", :source=\u003e\"k8s_kube-scheduler\", :category=\u003e\"kubernetes/kube-scheduler\"}}\n","stream":"stdout","time":"2017-02-15T21:05:04.372356299Z"}

[stevezau]

Should be fixed in latest release thanks to @fikander