Coder Social home page Coder Social logo

sumologic-docker-logging-driver's Introduction

Overview

Docker logging driver plugins extend Docker's logging capabilities. You can use the Sumo logging driver plugin to send Docker container logs to the Sumo cloud-based service. Once your log data is in Sumo, you can use the Sumo web app to search and analyze your log data.

Note: Docker plugins are not yet supported on Windows; see Docker's logging driver plugin documentation.

The Sumo logging plugin driver is supported by Sumo Logic. If you have issues or questions, create an issue on GitHub.

Set up Sumo logging driver plugin

Setting up the Sumo plugin involves setting up an HTTP endpoint on Sumo to receive Docker container log data, and configuring Docker to use the plugin.

Step 1 Configure Sumo to receive Docker logs

In this step you create, on the Sumo service, an HTTP endpoint to receive your Docker logs. This process involves creating an HTTP source on a hosted collector in Sumo. In Sumo, collectors use sources to receive data.

  1. If you don’t already have a Sumo account, you can create one by clicking the Free Trial button on https://www.sumologic.com/.

  2. Create a hosted collector, following the instructions on Configure a Hosted Collector in Sumo help. (If you already have a Sumo hosted collector that you want to use, skip this step.)

  3. Create an HTTP source on the collector you created in the previous step. For instructions, see HTTP Logs and Metrics Source in Sumo help.

  4. When you have configured the HTTP source, Sumo will display the URL of the HTTP endpoint. Make a note of the URL. You will use it when you configure Docker to send data to Sumo.

Step 2 Install Plugin

On each Docker host with containers from which you want to collect container logs, install the plugin by running the following command in a terminal window:

$ docker plugin install sumologic/docker-logging-driver:1.0.6 --alias sumologic --grant-all-permissions

This defaults to amd64; for arm64, please pull the corresponding version:

$ docker plugin install sumologic/docker-logging-driver:1.0.6-aarch64 --alias sumologic --grant-all-permissions

NOTE The --alias is required for using it on AWS ECS

NOTE As of version 1.0.6, the plugin image is only available from the repo sumologic/docker-logging-driver. Prior to 1.0.6, the plugin was also available at store/sumologic/docker-logging-driver. However, with the deprecation of the Docker Publisher Center, the store repo has been deprecated as well.

To verify that the plugin is installed and enabled, run the following command:

$ docker plugin ls

ID                  NAME              DESCRIPTION                       ENABLED
b72ceb1530ff        sumologic         Sumo Logic logging driver         true

Step 3 Configure Docker to use the plugin

The Docker daemon on each Docker host has a default logging driver; each container on the Docker host uses the default driver, unless you configure it to use a different logging driver.

To use the Sumo plugin, you need to configure one or more containers to use the plugin. Use Option A below to use the sumologic plugin on a single container. Use Option B to set up all containers on a host to use the plugin.

Option A Start a container to use Sumo driver

To run a specific container with the logging driver:

  • Use the --log-driver flag to specify the plugin.
  • Use the --log-opt flag to specify the URL for the HTTP source you created in Step 1.

For example:

$ docker run --log-driver=sumologic --log-opt sumo-url=sumo_source_url

where sumo-source-url is the URL that Sumo assigned to the HTTP source you created.

The following command starts the container whose name is your_container to use the Sumo plugin, specifies the URL for the HTTP source, and sets several optional --log-opts options. For more information about these and other options, see log-opt options below.

$ docker run --log-driver=sumologic \
    --log-opt sumo-url=sumo-source-url \
    --log-opt sumo-batch-size=2000000 \
    --log-opt sumo-queue-size=400 \
    --log-opt sumo-sending-interval=2000ms \
    --log-opt sumo-compress=false \
    --log-opt ... \
    your_container

where:

  • sumo_sourceurl is the URL of your HTTP Source.
  • your_container identifies a container.

The container should start sending logs to Sumo Logic.

Option B Configure all containers on Docker host to use Sumo driver

The Docker daemon for a Docker host has a default logging driver, which each container on the host uses unless you configure it to use a different logging driver. This procedure shows you how to update a Docker host’s daemon.json file so that all of the containers on the host use the Sumo plugin, and know the URL for for sending logs to the Sumo service.

For more information about configuring Docker using daemon.json, see Daemon Configuration File in Docker help.

  1. Find the Docker host’s daemon.json file, located by default in /etc/docker on Linux hosts.

  2. To set the Sumo as the default logging driver for a Docker host, set the log-driver key to “sumologic”. For an example, see the daemon.json excerpt below this procedure.

  3. To specify the URL for sending logs to Sumo, use the log-opts key to set sumo-url to the URL of the HTTP source you created in Step 1. For an example, see the daemon.json excerpt below this procedure.

  4. Specify any other desired log options. For supported options, see log-opt options below.

  5. Restart Docker for the changes to take effect.

Example excerpt from daemon.json

{
  "log-driver": "sumologic",
  "log-opts": {
    "sumo-url": "https://<deployment>.sumologic.com  		\
     /receiver/v1/http/<source_token>"
  }
}

Step 4 Search and analyze container log data

Once your container or containers are set up to send logs to Sumo, you can log onto the Sumo web app and start searching and analyzing the data. For help in getting started see Search in Sumo help.

log-opt options

To specify additional logging driver options, you can use the --log-opt NAME=VALUE flag.

Option Required? Default Value Description
sumo-url Yes HTTP Source URL
sumo-source-category No HTTP source category Source category to appear when searching in Sumo Logic by _sourceCategory. Use {{Tag}} as the placeholder for the tag option. If not specified, the source category of the HTTP source will be used.
sumo-source-name No container's name Source name to appear when searching in Sumo Logic by _sourceName. Use {{Tag}}as the placeholder for the tag option. If not specified, it will be the container's name.
sumo-source-host No host name Source host to appear when searching in Sumo Logic by _sourceHost. Use {{Tag}}as the placeholder for the tag option. If not specified, it will be the machine host name.
sumo-compress No true Enable/disable gzip compression. Boolean.
sumo-compress-level No -1 Set the gzip compression level. Valid values are -1 (default), 0 (no compression), 1 (best speed) ... 9 (best compression).
sumo-batch-size No 1000000 The number of bytes of logs the driver should wait for before sending them in bulk. If the number of bytes never reaches sumo-batch-size, the driver will send the logs in smaller batches at predefined intervals; see sumo-sending-interval.
sumo-sending-interval No 2s The maximum time the driver waits for number of logs to reach sumo-batch-size before sending the logs, even if the number of logs is less than the batch size. In the format 72h3m5s, valid time units are "ns", "us" (or "µs"), "ms", "s", "m", and "h".
sumo-proxy-url No Set a proxy URL.
sumo-insecure-skip-verify No false Ignore server certificate validation. Boolean.
sumo-root-ca-path No Set the path to a custom root certificate.
sumo-server-name No Name used to validate the server certificate. By default, uses hostname of the sumo-url.
sumo-queue-size No 100 The maximum number of log batches of size sumo-batch-size we can store in memory in the event of network failure, before we begin dropping batches. Thus in the worst case, the plugin will use sumo-batch-size * sumo-queue-size bytes of memory per container (default 100 MB).
tag No {{.ID}} Specifies a tag for messages, which can be used in the "source category", "source name", and "source host" fields. Certain tokens of the form {{X}} are supported. Default value is {{.ID}}, the first 12 characters of the container ID. For more information and a list of supported tokens, see Log tags for logging driver in Docker help.

Uninstall the plugin

To cleanly disable and remove the plugin, run:

$ docker plugin disable sumologic/docker-logging-driver
$ docker plugin rm sumologic/docker-logging-driver

TLS 1.2 Requirement

Sumo Logic only accepts connections from clients using TLS version 1.2 or greater. To utilize the content of this repo, ensure that it's running in an execution environment that is configured to use TLS 1.2 or greater.

sumologic-docker-logging-driver's People

Contributors

mariepmcgarry avatar mhumesf avatar rnishtala-sumo avatar rvmiller89 avatar samjsong avatar yuting-liu avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

sumologic-docker-logging-driver's Issues

docker-logging-driver plugin does not exist?

Hi, I just noticed that I am unable to pull the docker-logging-driver from several servers, though I was able to earlier today:

$ docker plugin install store/sumologic/docker-logging-driver:1.0.2 --alias sumologic --grant-all-permissions
Error response from daemon: pull access denied for store/sumologic/docker-logging-driver, repository does not exist or may require 'docker login'

Did something just happen?

When will logs get dropped by this plugin?

Hi,
This plugin looks really interesting, I have a couple of questions regarding particular lines of code:

logrus.Error(fmt.Errorf("%s: Log batch queue full, dropping oldest batch", pluginName))

logrus.Warn(fmt.Sprintf("%s: Log is too large to batch, dropping log. log-size: %d bytes",

These lines seem to indicate circumstances in which logs will be lost (log too large to batch and batch full). Could you clarify the circumstances in which this would occur (not sure what exactly 'batch' is or what size it may be)?

Is anyone using this plugin in production?

Cheers!
Bryan

Errors in the docker logs

We are using the sumologic docker logging driver with our docker containers in order to ship logs up to sumo. The logging driver works very well, but we've found a number of strange errors like the following in our syslog on the host:

Jun 19 01:12:05 ip-172-25-130-83 dockerd: time="2020-06-19T01:12:05Z" level=error msg="time="2020-06-19T01:12:05Z" level=info msg="sumologic: Sending logs batch. batch-size: 126 bytes"" plugin=48634abf754d3e0bd219aeb8af6d89b58aace92250d0375b17b45fa39ca16366

It seems the logging driver is printing an info message, and dockerd is picking it up and sending it along to syslog as an error, but we're at a loss as to where we can suppress these logs on the host side. We would prefer not to use a pre-processing rule to do this.

We have version 1.0.3 of the plugin and version 19.03.6-ce, build 369ce74 of docker.

ETA on New release? [file already closed]

Hello, I'm getting some impact by the "file already closed" issue. I see that a fix has been pushed to master but it's not published on dockerhub. Is there any ETA on that?

Edit: Looking for #24 to be released

no log driver named 'sumologic' is registered

Hello

I'm trying to get the logging driver installed on amzn-ami-2017.03.f-amazon-ecs-optimized. I followed the instructions (I ran docker plugin install store/sumologic/docker-logging-driver:1.0.2 --alias sumologic). Running docker plugins ls shows:

ID                  NAME                DESCRIPTION                 ENABLED
7d401749617c        sumologic:latest    Sumo Logic logging driver   true

(note NAME is sumologic:latest, not sumologic)

I can run a docker plugin inspect sumologic and i get back info.

However, when I try to actually use the driver... for example
docker run --log-driver=sumologic --log-opt sumo-url=<redacted> chentex/random-logger

I get: docker: Error response from daemon: logger: no log driver named 'sumologic' is registered.

Same error if I try --log-driver=sumologic:latest too. Am I missing a step?

Publish images to Amazon Public ECR

DockerHub has been tightening enforcement of their image pull rate limits, which is impacting our usage of sumologic/docker-logging-driver docker image.

SumoLogic already provides several images via AWS Public ECR: https://gallery.ecr.aws/sumologic/

However, I wasn't able to find the docker-logging-driver among these. Would it be possible to publish these images to AWS Public ECR?

Thank you!

Can't install plugin

Hi,

I tried installing this plugin, but didn't have much luck. I get the following output:

$ docker plugin install sumologic/docker-logging-driver
Error response from daemon: repository sumologic/docker-logging-driver not found: does not exist or no pull access
$ docker plugin install store/sumologic/docker-logging-driver
Error response from daemon: manifest for store/sumologic/docker-logging-driver:latest not found
$ docker plugin install store/sumologic/docker-logging-driver:v0.9.0
Error response from daemon: manifest for store/sumologic/docker-logging-driver:v0.9.0 not found

Any ideas?

Failed to log msg because of broken pipe

Hi,
I am using this sumologic-docker-logger-driver plugin version 1.0.2 with following log options

    "sumo-queue-size": "1000",
    "sumo-batch-size": "20000000",
    "sumo-compress" : "true",
    "sumo-compress-level" : "5",
    "sumo-sending-interval" : "15",

The rate of injest of logs is : 6GB / minute.
Fd limit inside the container is set to 65000.

While doing so, getting the following error
time="2018-04-23T13:00:07.118852858Z" level=error msg="Failed to log msg \"\" for logger sumologic: write /proc/self/fd/74: broken pipe"
Can you please help ?

Unable to use plugin on rancher host

We are using Rancher to manage docker. We've installed the sumologic docker logging driver but we're getting the dreaded no driver named sumologic is registered error. We are not using ECS in this case. Any idea where the failure is?

# docker version
Client:
 Version:      17.03.0-ce
 API version:  1.26
 Go version:   go1.7.5
 Git commit:   60ccb22
 Built:        Thu Feb 23 10:57:47 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.03.0-ce
 API version:  1.26 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   60ccb22
 Built:        Thu Feb 23 10:57:47 2017
 OS/Arch:      linux/amd64
 Experimental: false
# docker plugin inspect sumologic
[
    {
        "Config": {
            "Args": {
                "Description": "",
                "Name": "",
                "Settable": null,
                "Value": null
            },
            "Description": "Sumo Logic logging driver",
            "Documentation": "https://github.com/SumoLogic/sumologic-docker-logging-driver",
            "Entrypoint": [
                "/usr/bin/docker-logging-driver"
            ],
            "Env": null,
            "Interface": {
                "Socket": "sumologic.sock",
                "Types": [
                    "docker.logdriver/1.0"
                ]
            },
            "Linux": {
                "AllowAllDevices": false,
                "Capabilities": null,
                "Devices": null
            },
            "Mounts": null,
            "Network": {
                "Type": "host"
            },
            "PropagatedMount": "",
            "User": {},
            "WorkDir": "",
            "rootfs": {
                "diff_ids": [
                    "sha256:18fca34a589e793a578366e429373d70ba1665d92da440e61a0765dce81daf12"
                ],
                "type": "layers"
            }
        },
        "Enabled": true,
        "Id": "1a3e47080a69a372f98d8cb9a6ff3a2b5b4c6a3aa8ffa2a779ff3a340b7be46b",
        "Name": "sumologic:latest",
        "PluginReference": "store/sumologic/docker-logging-driver:1.0.2",
        "Settings": {
            "Args": [],
            "Devices": [],
            "Env": [],
            "Mounts": []
        }
    }
]
# docker info
Containers: 19
 Running: 18
 Paused: 0
 Stopped: 1
Images: 19
Server Version: 17.03.0-ce
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 201
 Dirperm1 Supported: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 977c511eda0925a723debdc94d09459af49d082a
runc version: a01dafd48bc1c7cc12bdb01206f9fea7dd6feb70
init version: 949e6fa
Security Options:
 apparmor
Kernel Version: 3.13.0-153-generic
Operating System: Ubuntu 14.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 3.676 GiB
Name: redacted
ID: redacted
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false
# docker run --log-driver=sumologic:latest --log-opt "sumo-url=redacted" --log-opt "sumo-category=testing/helloworld" hello-world
docker: Error response from daemon: logger: no log driver named 'sumologic:latest' is registered.
See 'docker run --help'.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.