kaiba42 / pino-elasticsearch

🌲 load pino logs into Elasticsearch

pino-elasticsearch   

Load pino logs into Elasticsearch.

Install

npm install pino-elasticsearch -g

Usage

  pino-elasticsearch

  To send pino logs to elasticsearch:

     cat log | pino-elasticsearch --node http://localhost:9200

  Flags
  -h  | --help              Display Help
  -v  | --version           display Version
  -n  | --node              the URL where Elasticsearch is running
  -i  | --index             the name of the index to use; default: pino
                            will replace %{DATE} with the YYYY-MM-DD date
  -t  | --type              the name of the type to use; default: log
  -f  | --flush-bytes       the number of bytes for each bulk insert; default: 1000
  -b  | --bulk-size         the number of documents for each bulk insert [DEPERCATED]
  -l  | --trace-level       trace level for the elasticsearch client, default 'error' (info, debug, trace).
      | --es-version        specify the major version number of Elasticsearch (eg: 5, 6, 7)
                            (this is needed only if you are using Elasticsearch <= 7)

Usage as module

const pino = require('pino')
const pinoElastic = require('pino-elasticsearch')

const streamToElastic = pinoElastic({
  index: 'an-index',
  consistency: 'one',
  node: 'http://localhost:9200',
  'es-version': 7,
  'flush-bytes': 1000
})

const logger = pino({ level: 'info' }, streamToElastic)

logger.info('hello world')
// ...

ECS support

If you want to use Elastic Common Schema, you should install @elastic/ecs-pino-format, as the ecs option of this module has been removed.

const pino = require('pino')
const ecsFormat = require('@elastic/ecs-pino-format')()
const pinoElastic = require('pino-elasticsearch')

const streamToElastic = pinoElastic({
  index: 'an-index',
  consistency: 'one',
  node: 'http://localhost:9200',
  'es-version': 7,
  'flush-bytes': 1000
})

const logger = pino({ level: 'info',  ...ecsFormat  }, streamToElastic)

logger.info('hello world')
// ...

You can then use Kibana to browse and visualize your logs.
Note: This transport works only with Elasticsearch version ≥ 5.

Dynamic index

It is possible to customize the index name for every log line just providing a function to the index option:

const pino = require('pino')
const pinoElastic = require('pino-elasticsearch')

const streamToElastic = pinoElastic({
  index: function (logTime) {
    // the logTime is a ISO 8601 formatted string of the log line
    return `awesome-app-${logTime.substring(5, 10)}`
  },
  consistency: 'one',
  node: 'http://localhost:9200'
})
// ...

The function must be sync, doesn't throw and return a string.

Authentication

If you need to use basic authentication to connect with the Elasticsearch cluster, pass the credentials in the URL:

cat log | pino-elasticsearch --node https://user:pwd@localhost:9200

Setup and Testing

Setting up pino-elasticsearch is easy, and you can use the bundled docker-compose.yml file to bring up both Elasticsearch and Kibana.

You will need docker and docker-compose, then in this project folder, launch docker-compose up.

You can test it by launching node example | pino-elasticsearch, in this project folder. You will need to have pino-elasticsearch installed globally.

Acknowledgements

This project was kindly sponsored by nearForm.

License

Licensed under MIT.