laisky / go-fluentd

rewrite fluentd in golang

Go-Fluentd

Rewrite fluentd-server by Golang, Higher performance with less resource requirement.

• At-Least-Once guarantee(disk WAL)
• log concatenation by head regexp expression
• log parsing by regexp expression(support embedded json)
• log filter by custom plugins(acceptorfilters & tagfilters)
• multiple receivers(support multiple protocols: msgpack, http, syslog, kafka, ...)
• multiple senders(support multiple backend: elasticsearch, fluentd, ...)
• multiple environments deployment(--env: sit, perf, uat, prod)

Already running on our PRODUCION since 2018/9.

When processing 1000mbps logs flood:

• dstat

  

• monitor

  

• profile

  

Documents:

• 中文文档
• Quick Start
• Build with Docker

Description

LogAggregator + Concator + Parser + Producer.

Origin logs emitted from docker look like:

'{"container_id": "xxxxx", "log": "2018-03-06 16:56:22.514 | mscparea | ERROR  | http-nio-8080-exec-1 | com.laisky.cloud.cp.core.service.impl.CPBusiness.reflectAdapterRequest | 84:'
'{"container_id": "xxxxx", "log": "Exception in thread "main" java.lang.IllegalStateException: A book has a null property"}'
'{"container_id": "xxxxx", "log": "\tat com.example.myproject.Author.getBookIds(Author.java:38)"}'
'{"container_id": "xxxxx", "log": "\tat com.example.myproject.Bootstrap.main(Bootstrap.java:14)"}'
'{"container_id": "xxxxx", "log": "Caused by: java.lang.NullPointerException"}'
'{"container_id": "xxxxx", "log": "\tat com.example.myproject.Book.getId(Book.java:22)"}'
'{"container_id": "xxxxx", "log": "\tat com.example.myproject.Author.getBookIds(Author.java:35)"}'
'{"container_id": "xxxxx", "log": "\t... 1 more"}'

After Concator(TagPipeline > concator_f):

&FluentMsg{
    Id: 12345,
    Tag: "spring.sit",
    Message: map[string]interface{}{
        "container_id": "xxxxx",
        "log": "2018-03-06 16:56:22.514 | mscparea | ERROR  | http-nio-8080-exec-1 | com.laisky.cloud.cp.core.service.impl.CPBusiness.reflectAdapterRequest | 84: Exception in thread "main" java.lang.IllegalStateException: A book has a null property\n\tat com.example.myproject.Author.getBookIds(Author.java:38)\n\tat com.example.myproject.Bootstrap.main(Bootstrap.java:14)\nCaused by: java.lang.NullPointerException\n\tat com.example.myproject.Book.getId(Book.java:22)\n\tat com.example.myproject.Author.getBookIds(Author.java:35)\n\t... 1 more",
    },
}

After Parser(TagPipeline > parser_f):

&FluentMsg{
    Id: 12345,
    Tag: "spring.sit",
    Message: map[string]interface{}{
        "container_id": "xxxxx",
        "time": "2018-03-06 16:56:22.514",
        "level": "ERROR",
        "app": "mscparea",
        "thread": "http-nio-8080-exec-1",
        "class": "com.laisky.cloud.cp.core.service.impl.CPBusiness.reflectAdapterRequest",
        "line": 84,
        "message": "Exception in thread "main" java.lang.IllegalStateException: A book has a null property\n\tat com.example.myproject.Author.getBookIds(Author.java:38)\n\tat com.example.myproject.Bootstrap.main(Bootstrap.java:14)\nCaused by: java.lang.NullPointerException\n\tat com.example.myproject.Book.getId(Book.java:22)\n\tat com.example.myproject.Author.getBookIds(Author.java:35)\n\t... 1 more",
    },
}

Then Producer can send logs to anywhere (depends on Senders).

Run

directly run:

go run -race main.go \
  --config=./docs/settings/tiny_settings.yml \
  --env=sit \
  --log-level=debug

run by docker:

docker run -itd --rm --name=go-fluentd -p 24225:24225 -p 8080:8080 \
    -v /etc/configs/go-fluentd:/etc/go-fluentd \
    -v /data/log/fluentd/go-fluentd:/data/log/fluentd/go-fluentd
    ppcelery/go-fluentd:1.13.1 \
    ./go-fluentd \
        --config=/etc/go-fluentd/settings.yml \
        --env=perf \
        --addr=0.0.0.0:8080
        --host=x.x.x.x
        --enable-auto-gc

docker images version

• stable
• release
• dev
• <feature taskid>

Roles

• Acceptor (consists of Recvs)
• AcceptorPipeline (consists of AcceptorFilters)
• Journal
• Dispatcher
• TagPipeline (consists of TagFilters)
  • Concator
  • Parser for each tag
• PostPipeline (consists of PostFilters)
• Producer (consists of Senders)

Acceptor

Contains multiply Recvs (such as KafkRecv & FluentdRecv), can listening tcp port or fetch msg from kafka brokers.

AcceptorPipeline

Contains multiply AcceptorFilters, be used for ignore or retag specific messages. All filters should return very fast to avoid blocking.

Journal

...

Dispatcher

...

TagPipeline

...

PostPipeline

...

Producer

...