Coder Social home page Coder Social logo

log's Introduction

phuslog - Fastest structured logging

godoc goreport build stability-stable

Features

  • Dependency Free
  • Simple and Clean Interface
  • Consistent Writer
    • IOWriter, io.Writer wrapper
    • ConsoleWriter, colorful & formatting
    • FileWriter, rotating & effective
    • MultiLevelWriter, multiple level dispatch
    • SyslogWriter, memory efficient syslog
    • JournalWriter, linux systemd logging
    • EventlogWriter, windows system event
    • AsyncWriter, asynchronously writing
  • Stdlib Log Adapter
    • Logger.Std, transform to std log instances
    • Logger.Slog, transform to log/slog instances
  • Third-party Logger Interceptor
    • logr, logr interceptor
    • gin, gin logging middleware
    • gorm, gorm logger interface
    • fiber, fiber logging handler
    • grpc, grpc logger interceptor
    • grpcgateway, grpcgateway logger interceptor
  • Useful utility function
    • Goid(), the goroutine id matches stack trace
    • NewXID(), create a tracing id
    • Fastrandn(n uint32), fast pseudorandom uint32 in [0,n)
    • IsTerminal(fd uintptr), isatty for golang
    • Printf(fmt string, a ...interface{}), printf logging
  • High Performance

Interfaces

Logger

// DefaultLogger is the global logger.
var DefaultLogger = Logger{
	Level:      DebugLevel,
	Caller:     0,
	TimeField:  "",
	TimeFormat: "",
	Writer:     &IOWriter{os.Stderr},
}

// Logger represents an active logging object that generates lines of JSON output to an io.Writer.
type Logger struct {
	// Level defines log levels.
	Level Level

	// Caller determines if adds the file:line of the "caller" key.
	// If Caller is negative, adds the full /path/to/file:line of the "caller" key.
	Caller int

	// TimeField defines the time field name in output.  It uses "time" in if empty.
	TimeField string

	// TimeFormat specifies the time format in output. Uses RFC3339 with millisecond if empty.
	// If set to `TimeFormatUnix/TimeFormatUnixMs`, timestamps will be formatted.
	TimeFormat string

	// TimeLocation specifices that the location of TimeFormat used. Uses time.Local if empty.
	TimeLocation *time.Location

	// Writer specifies the writer of output. It uses a wrapped os.Stderr Writer in if empty.
	Writer Writer
}

ConsoleWriter

// ConsoleWriter parses the JSON input and writes it in a colorized, human-friendly format to Writer.
// IMPORTANT: Don't use ConsoleWriter on critical path of a high concurrency and low latency application.
//
// Default output format:
//     {Time} {Level} {Goid} {Caller} > {Message} {Key}={Value} {Key}={Value}
type ConsoleWriter struct {
	// ColorOutput determines if used colorized output.
	ColorOutput bool

	// QuoteString determines if quoting string values.
	QuoteString bool

	// EndWithMessage determines if output message in the end of line.
	EndWithMessage bool

	// Writer is the output destination. using os.Stderr if empty.
	Writer io.Writer

	// Formatter specifies an optional text formatter for creating a customized output,
	// If it is set, ColorOutput, QuoteString and EndWithMessage will be ignored.
	Formatter func(w io.Writer, args *FormatterArgs) (n int, err error)
}

// FormatterArgs is a parsed sturct from json input
type FormatterArgs struct {
	Time      string // "2019-07-10T05:35:54.277Z"
	Message   string // "a structure message"
	Level     string // "info"
	Caller    string // "main.go:123"
	Goid      string // "1"
	Stack     string // "<stack string>"
	KeyValues []struct {
		Key       string // "foo"
		Value     string // "bar"
	}
}

FileWriter

// FileWriter is an Writer that writes to the specified filename.
type FileWriter struct {
	// Filename is the file to write logs to.  Backup log files will be retained
	// in the same directory.
	Filename string

	// FileMode represents the file's mode and permission bits.  The default
	// mode is 0644
	FileMode os.FileMode

	// MaxSize is the maximum size in bytes of the log file before it gets rotated.
	MaxSize int64

	// MaxBackups is the maximum number of old log files to retain.  The default
	// is to retain all old log files
	MaxBackups int

	// TimeFormat specifies the time format of filename, uses `2006-01-02T15-04-05` as default format.
	// If set with `TimeFormatUnix`, `TimeFormatUnixMs`, times are formated as UNIX timestamp.
	TimeFormat string

	// LocalTime determines if the time used for formatting the timestamps in
	// log files is the computer's local time.  The default is to use UTC time.
	LocalTime bool

	// HostName determines if the hostname used for formatting in log files.
	HostName bool

	// ProcessID determines if the pid used for formatting in log files.
	ProcessID bool

	// EnsureFolder ensures the file directory creation before writing.
	EnsureFolder bool

	// Header specifies an optional header function of log file after rotation,
	Header func(fileinfo os.FileInfo) []byte

	// Cleaner specifies an optional cleanup function of log backups after rotation,
	// if not set, the default behavior is to delete more than MaxBackups log files.
	Cleaner func(filename string, maxBackups int, matches []os.FileInfo)
}

Highlights:

  • FileWriter implements log.Writer and io.Writer interfaces both, it is a recommended alternative to lumberjack.
  • FileWriter creates a symlink to the current logging file, it requires administrator privileges on Windows.
  • FileWriter does not rotate if you define a broad TimeFormat value(daily or monthly) until reach its MaxSize.

Getting Started

Simple Logging Example

An out of box example. playground

package main

import (
	"github.com/phuslu/log"
)

func main() {
	log.Info().Str("foo", "bar").Int("number", 42).Msg("hi, phuslog")
	log.Info().Msgf("foo=%s number=%d error=%+v", "bar", 42, "an error")
}

// Output:
//   {"time":"2020-03-22T09:58:41.828Z","level":"info","foo":"bar","number":42,"message":"hi, phuslog"}
//   {"time":"2020-03-22T09:58:41.828Z","level":"info","message":"foo=bar number=42 error=an error"}

Note: By default log writes to os.Stderr

Customize the configuration and formatting:

To customize logger filed name and format. playground

package main

import (
	"github.com/phuslu/log"
)

func main() {
	log.DefaultLogger = log.Logger{
		Level:      log.InfoLevel,
		Caller:     1,
		TimeField:  "date",
		TimeFormat: "2006-01-02",
		Writer:     &log.IOWriter{os.Stdout},
	}

	log.Info().Str("foo", "bar").Msgf("hello %s", "world")

	logger := log.Logger{
		Level:      log.InfoLevel,
		TimeField:  "ts",
		TimeFormat: log.TimeFormatUnixMs,
	}

	logger.Log().Str("foo", "bar").Msg("")
}

// Output:
//    {"date":"2019-07-04","level":"info","caller":"prog.go:16","foo":"bar","message":"hello world"}
//    {"ts":1257894000000,"foo":"bar"}

Pretty Console Writer

To log a human-friendly, colorized output, use ConsoleWriter. playground

if log.IsTerminal(os.Stderr.Fd()) {
	log.DefaultLogger = log.Logger{
		TimeFormat: "15:04:05",
		Caller:     1,
		Writer: &log.ConsoleWriter{
			ColorOutput:    true,
			QuoteString:    true,
			EndWithMessage: true,
		},
	}
}

log.Debug().Int("everything", 42).Str("foo", "bar").Msg("hello world")
log.Info().Int("everything", 42).Str("foo", "bar").Msg("hello world")
log.Warn().Int("everything", 42).Str("foo", "bar").Msg("hello world")
log.Error().Err(errors.New("an error")).Msg("hello world")

Pretty logging

Note: pretty logging also works on windows console

Formatting Console Writer

To log with user-defined format(e.g. glog), using ConsoleWriter.Formatter. playground

package main

import (
	"fmt"
	"io"

	"github.com/phuslu/log"
)

type Glog struct {
	Logger log.Logger
}

func (l *Glog) Infof(fmt string, a ...any) { l.Logger.Info().Msgf(fmt, a...) }

func (l *Glog) Warnf(fmt string, a ...any) { l.Logger.Warn().Msgf(fmt, a...) }

func (l *Glog) Errorf(fmt string, a ...any) { l.Logger.Error().Msgf(fmt, a...) }

var glog = &Glog{log.Logger{
	Level:      log.InfoLevel,
	Caller:     2,
	TimeFormat: "0102 15:04:05.999999",
	Writer: &log.ConsoleWriter{Formatter: func(w io.Writer, a *log.FormatterArgs) (int, error) {
		return fmt.Fprintf(w, "%c%s %s %s] %s\n%s", a.Level[0]-32, a.Time, a.Goid, a.Caller, a.Message, a.Stack)
	}},
}}

func main() {
	glog.Infof("hello glog %s", "Info")
	glog.Warnf("hello glog %s", "Warn")
	glog.Errorf("hello glog %s", "Error")
}

// Output:
// I0725 09:59:57.503246 19 console_test.go:183] hello glog Info
// W0725 09:59:57.504247 19 console_test.go:184] hello glog Warn
// E0725 09:59:57.504247 19 console_test.go:185] hello glog Error

Formatting Logfmt output

To log with logfmt format, also using ConsoleWriter.Formatter. playground

package main

import (
	"io"
	"os"

	"github.com/phuslu/log"
)

func main() {
	log.DefaultLogger = log.Logger{
		Level:      log.InfoLevel,
		Caller:     1,
		TimeField:  "ts",
		TimeFormat: log.TimeFormatUnixWithMs,
		Writer: &log.ConsoleWriter{
			Formatter: log.LogfmtFormatter{"ts"}.Formatter,
			Writer:    io.MultiWriter(os.Stdout, os.Stderr),
		},
	}

	log.Info().Str("foo", "bar").Int("no", 42).Msgf("a logfmt %s", "info")
}

// Output:
// ts=1257894000.000 level=info goid=1 caller="prog.go:20" foo="bar" no=42 "a logfmt info"
// ts=1257894000.000 level=info goid=1 caller="prog.go:20" foo="bar" no=42 "a logfmt info"

Rotating File Writer

To log to a daily-rotating file, use FileWriter. playground

package main

import (
	"os"
	"path/filepath"
	"time"

	"github.com/phuslu/log"
	"github.com/robfig/cron/v3"
)

func main() {
	logger := log.Logger{
		Level: log.ParseLevel("info"),
		Writer: &log.FileWriter{
			Filename:     "logs/main.log",
			FileMode:     0600,
			MaxSize:      100 * 1024 * 1024,
			MaxBackups:   7,
			EnsureFolder: true,
			LocalTime:    true,
		},
	}

	runner := cron.New(cron.WithLocation(time.Local))
	runner.AddFunc("0 0 * * *", func() { logger.Writer.(*log.FileWriter).Rotate() })
	go runner.Run()

	for {
		time.Sleep(time.Second)
		logger.Info().Msg("hello world")
	}
}

Rotating File Writer within a total size

To rotating log file hourly and keep in a total size, use FileWriter.Cleaner.

package main

import (
	"os"
	"path/filepath"
	"time"

	"github.com/phuslu/log"
	"github.com/robfig/cron/v3"
)

func main() {
	logger := log.Logger{
		Level: log.ParseLevel("info"),
		Writer: &log.FileWriter{
			Filename: "main.log",
			MaxSize:  500 * 1024 * 1024,
			Cleaner:  func(filename string, maxBackups int, matches []os.FileInfo) {
				var dir = filepath.Dir(filename)
				var total int64
				for i := len(matches) - 1; i >= 0; i-- {
					total += matches[i].Size()
					if total > 5*1024*1024*1024 {
						os.Remove(filepath.Join(dir, matches[i].Name()))
					}
				}
			},
		},
	}

	runner := cron.New(cron.WithLocation(time.UTC))
	runner.AddFunc("0 * * * *", func() { logger.Writer.(*log.FileWriter).Rotate() })
	go runner.Run()

	for {
		time.Sleep(time.Second)
		logger.Info().Msg("hello world")
	}
}

Rotating File Writer with compression

To rotating log file hourly and compressing after rotation, use FileWriter.Cleaner.

package main

import (
	"os"
	"os/exec"
	"path/filepath"
	"time"

	"github.com/phuslu/log"
	"github.com/robfig/cron/v3"
)

func main() {
	logger := log.Logger{
		Level: log.ParseLevel("info"),
		Writer: &log.FileWriter{
			Filename: "main.log",
			MaxSize:  500 * 1024 * 1024,
			Cleaner:  func(filename string, maxBackups int, matches []os.FileInfo) {
				var dir = filepath.Dir(filename)
				for i, fi := range matches {
					filename := filepath.Join(dir, fi.Name())
					switch {
					case i > maxBackups:
						os.Remove(filename)
					case !strings.HasSuffix(filename, ".gz"):
						go exec.Command("nice", "gzip", filename).Run()
					}
				}
			},
		},
	}

	runner := cron.New(cron.WithLocation(time.UTC))
	runner.AddFunc("0 * * * *", func() { logger.Writer.(*log.FileWriter).Rotate() })
	go runner.Run()

	for {
		time.Sleep(time.Second)
		logger.Info().Msg("hello world")
	}
}

Random Sample Logger:

To logging only 5% logs, use below idiom.

if log.Fastrandn(100) < 5 {
	log.Log().Msg("hello world")
}

Multiple Dispatching Writer

To log to different writers by different levels, use MultiLevelWriter.

log.DefaultLogger.Writer = &log.MultiLevelWriter{
	InfoWriter:    &log.FileWriter{Filename: "main.INFO", MaxSize: 100<<20},
	WarnWriter:    &log.FileWriter{Filename: "main.WARNING", MaxSize: 100<<20},
	ErrorWriter:   &log.FileWriter{Filename: "main.ERROR", MaxSize: 100<<20},
	ConsoleWriter: &log.ConsoleWriter{ColorOutput: true},
	ConsoleLevel:  log.ErrorLevel,
}

log.Info().Int("number", 42).Str("foo", "bar").Msg("a info log")
log.Warn().Int("number", 42).Str("foo", "bar").Msg("a warn log")
log.Error().Int("number", 42).Str("foo", "bar").Msg("a error log")

Multiple Entry Writer

To log to different writers, use MultiEntryWriter.

log.DefaultLogger.Writer = &log.MultiEntryWriter{
	&log.ConsoleWriter{ColorOutput: true},
	&log.FileWriter{Filename: "main.log", MaxSize: 100<<20},
	&log.EventlogWriter{Source: ".NET Runtime", ID: 1000},
}

log.Info().Int("number", 42).Str("foo", "bar").Msg("a info log")

Multiple IO Writer

To log to multiple io writers like io.MultiWriter, use below idiom. playground

log.DefaultLogger.Writer = &log.MultiIOWriter{
	os.Stdout,
	&log.FileWriter{Filename: "main.log", MaxSize: 100<<20},
}

log.Info().Int("number", 42).Str("foo", "bar").Msg("a info log")

Multiple Combined Logger:

To logging to different logger as you want, use below idiom. playground

package main

import (
	"github.com/phuslu/log"
)

var logger = struct {
	Console log.Logger
	Access  log.Logger
	Data    log.Logger
}{
	Console: log.Logger{
		TimeFormat: "15:04:05",
		Caller:     1,
		Writer: &log.ConsoleWriter{
			ColorOutput:    true,
			EndWithMessage: true,
		},
	},
	Access: log.Logger{
		Level: log.InfoLevel,
		Writer: &log.FileWriter{
			Filename:   "access.log",
			MaxSize:    50 * 1024 * 1024,
			MaxBackups: 7,
			LocalTime:  false,
		},
	},
	Data: log.Logger{
		Level: log.InfoLevel,
		Writer: &log.FileWriter{
			Filename:   "data.log",
			MaxSize:    50 * 1024 * 1024,
			MaxBackups: 7,
			LocalTime:  false,
		},
	},
}

func main() {
	logger.Console.Info().Msgf("hello world")
	logger.Access.Log().Msgf("handle request")
	logger.Data.Log().Msgf("some data")
}

SyslogWriter

SyslogWriter is a memory-efficient, cross-platform, dependency-free syslog writer, outperforms all other structured logging libraries.

package main

import (
	"net"
	"time"

	"github.com/phuslu/log"
)

func main() {
	go func() {
		ln, _ := net.Listen("tcp", "127.0.0.1:1601")
		for {
			conn, _ := ln.Accept()
			go func(c net.Conn) {
				b := make([]byte, 8192)
				n, _ := conn.Read(b)
				println(string(b[:n]))
			}(conn)
		}
	}()

	syslog := log.Logger{
		Level:      log.InfoLevel,
		TimeField:  "ts",
		TimeFormat: log.TimeFormatUnixMs,
		Writer: &log.SyslogWriter{
			Network: "tcp",            // "unixgram",
			Address: "127.0.0.1:1601", // "/run/systemd/journal/syslog",
			Tag:     "",
			Marker:  "@cee:",
			Dial:    net.Dial,
		},
	}

	syslog.Info().Str("foo", "bar").Int("an", 42).Msg("a syslog info")
	syslog.Warn().Str("foo", "bar").Int("an", 42).Msg("a syslog warn")
	time.Sleep(2)
}

// Output:
// <6>2022-07-24T18:48:15+08:00 127.0.0.1:59277 [11516]: @cee:{"ts":1658659695428,"level":"info","foo":"bar","an":42,"message":"a syslog info"}
// <4>2022-07-24T18:48:15+08:00 127.0.0.1:59277 [11516]: @cee:{"ts":1658659695429,"level":"warn","foo":"bar","an":42,"message":"a syslog warn"}

JournalWriter

To log to linux systemd journald, using JournalWriter.

log.DefaultLogger.Writer = &log.JournalWriter{
	JournalSocket: "/run/systemd/journal/socket",
}

log.Info().Int("number", 42).Str("foo", "bar").Msg("hello world")

EventlogWriter

To log to windows system event, using EventlogWriter.

log.DefaultLogger.Writer = &log.EventlogWriter{
	Source: ".NET Runtime",
	ID:     1000,
}

log.Info().Int("number", 42).Str("foo", "bar").Msg("hello world")

AsyncWriter

To logging asynchronously for performance stability, use AsyncWriter.

logger := log.Logger{
	Level:  log.InfoLevel,
	Writer: &log.AsyncWriter{
		ChannelSize: 100,
		Writer:      &log.FileWriter{
			Filename:   "main.log",
			FileMode:   0600,
			MaxSize:    50*1024*1024,
			MaxBackups: 7,
			LocalTime:  false,
		},
	},
}

logger.Info().Int("number", 42).Str("foo", "bar").Msg("a async info log")
logger.Warn().Int("number", 42).Str("foo", "bar").Msg("a async warn log")
logger.Writer.(io.Closer).Close()

Note: To flush data and quit safely, call AsyncWriter.Close() explicitly.

Stdlib Log Adapter

Using wrapped loggers for stdlog. playground

package main

import (
	stdlog "log"
	"os"

	"github.com/phuslu/log"
)

func main() {
	var logger *stdlog.Logger = (&log.Logger{
		Level:      log.InfoLevel,
		TimeField:  "date",
		TimeFormat: "2006-01-02",
		Caller:     1,
		Context:    log.NewContext(nil).Str("logger", "mystdlog").Int("myid", 42).Value(),
		Writer:     &log.IOWriter{os.Stdout},
	}).Std("", 0)

	logger.Print("hello from stdlog Print")
	logger.Println("hello from stdlog Println")
	logger.Printf("hello from stdlog %s", "Printf")
}

slog Adapter

Using wrapped loggers for slog. playground

package main

import (
	"log/slog"

	"github.com/phuslu/log"
)

func main() {
	var logger *slog.Logger = (&log.Logger{
		Level:      log.InfoLevel,
		TimeField:  "date",
		TimeFormat: "2006-01-02",
		Caller:     1,
	}).Slog()

	logger = logger.With("logger", "a_test_slog").With("everything", 42)

	logger.Info("hello from slog Info")
	logger.Warn("hello from slog Warn")
	logger.Error("hello from slog Error")
}

Third-party Logger Interceptor

Logger Interceptor
logr https://github.com/phuslu/log-contrib/tree/master/logr
gin https://github.com/phuslu/log-contrib/tree/master/gin
fiber https://github.com/phuslu/log-contrib/tree/master/fiber
gorm https://github.com/phuslu/log-contrib/tree/master/gorm
grpc https://github.com/phuslu/log-contrib/tree/master/grpc
grpcgateway https://github.com/phuslu/log-contrib/tree/master/grpcgateway

User-defined Data Structure

To log with user-defined struct effectively, implements MarshalObject. playground

package main

import (
	"github.com/phuslu/log"
)

type User struct {
	ID   int
	Name string
	Pass string
}

func (u *User) MarshalObject(e *log.Entry) {
	e.Int("id", u.ID).Str("name", u.Name).Str("password", "***")
}

func main() {
	log.Info().Object("user", &User{1, "neo", "123456"}).Msg("")
	log.Info().EmbedObject(&User{2, "john", "abc"}).Msg("")
}

// Output:
//   {"time":"2020-07-12T05:03:43.949Z","level":"info","user":{"id":1,"name":"neo","password":"***"}}
//   {"time":"2020-07-12T05:03:43.949Z","level":"info","id":2,"name":"john","password":"***"}

Contextual Fields

To add preserved key:value pairs to each entry, use NewContext. playground

logger := log.Logger{
	Level:   log.InfoLevel,
	Context: log.NewContext(nil).Str("ctx", "some_ctx").Value(),
}

logger.Debug().Int("no0", 0).Msg("zero")
logger.Info().Int("no1", 1).Msg("first")
logger.Info().Int("no2", 2).Msg("second")

// Output:
//   {"time":"2020-07-12T05:03:43.949Z","level":"info","ctx":"some_ctx","no1":1,"message":"first"}
//   {"time":"2020-07-12T05:03:43.949Z","level":"info","ctx":"some_ctx","no2":2,"message":"second"}

You can make a copy of log and add contextual fields. playground

package main

import (
	"github.com/phuslu/log"
)

func main() {
	sublogger := log.DefaultLogger
	sublogger.Level = log.InfoLevel
	sublogger.Context = log.NewContext(nil).Str("ctx", "some_ctx").Value()

	sublogger.Debug().Int("no0", 0).Msg("zero")
	sublogger.Info().Int("no1", 1).Msg("first")
	sublogger.Info().Int("no2", 2).Msg("second")
	log.Debug().Int("no3", 3).Msg("no context")
}

// Output:
//   {"time":"2021-06-14T06:36:42.904+02:00","level":"info","ctx":"some_ctx","no1":1,"message":"first"}
//   {"time":"2021-06-14T06:36:42.905+02:00","level":"info","ctx":"some_ctx","no2":2,"message":"second"}
//   {"time":"2021-06-14T06:36:42.906+02:00","level":"debug","no3":3,"message":"no context"}

High Performance

The most common benchmarks(disable/normal/caller/printf/interface) against slog/zap/zerolog
// go test -v -cpu=4 -run=none -bench=. -benchtime=10s -benchmem log_test.go
package main

import (
	"io"
	stdlog "log"
	"log/slog"
	"testing"

	"github.com/phuslu/log"
	"github.com/rs/zerolog"
	"go.uber.org/zap"
	"go.uber.org/zap/zapcore"
)

const msg = "The quick brown fox jumps over the lazy dog"
var obj = struct {Rate string; Low int; High float32}{"15", 16, 123.2}

func BenchmarkSlogDisable(b *testing.B) {
	logger := slog.New(slog.NewJSONHandler(io.Discard, nil))
	for i := 0; i < b.N; i++ {
		logger.Debug(msg, "rate", "15", "low", 16, "high", 123.2)
	}
}

func BenchmarkSlogNormal(b *testing.B) {
	logger := slog.New(slog.NewJSONHandler(io.Discard, nil))
	for i := 0; i < b.N; i++ {
		logger.Info(msg, "rate", "15", "low", 16, "high", 123.2)
	}
}

func BenchmarkSlogPrintf(b *testing.B) {
	slog.SetDefault(slog.New(slog.NewJSONHandler(io.Discard, nil)))
	for i := 0; i < b.N; i++ {
		stdlog.Printf("rate=%s low=%d high=%f msg=%s", "15", 16, 123.2, msg)
	}
}

func BenchmarkSlogCaller(b *testing.B) {
	logger := slog.New(slog.NewJSONHandler(io.Discard, &slog.HandlerOptions{AddSource: true}))
	for i := 0; i < b.N; i++ {
		logger.Info(msg, "rate", "15", "low", 16, "high", 123.2)
	}
}

func BenchmarkSlogInterface(b *testing.B) {
	logger := slog.New(slog.NewJSONHandler(io.Discard, nil))
	for i := 0; i < b.N; i++ {
		logger.Info(msg, "object", &obj)
	}
}

func BenchmarkZapDisable(b *testing.B) {
	logger := zap.New(zapcore.NewCore(
		zapcore.NewJSONEncoder(zap.NewProductionEncoderConfig()),
		zapcore.AddSync(io.Discard),
		zapcore.InfoLevel,
	)).Sugar()
	for i := 0; i < b.N; i++ {
		logger.Debugw(msg, "rate", "15", "low", 16, "high", 123.2)
	}
}

func BenchmarkZapNormal(b *testing.B) {
	logger := zap.New(zapcore.NewCore(
		zapcore.NewJSONEncoder(zap.NewProductionEncoderConfig()),
		zapcore.AddSync(io.Discard),
		zapcore.InfoLevel,
	)).Sugar()
	for i := 0; i < b.N; i++ {
		logger.Infow(msg, "rate", "15", "low", 16, "high", 123.2)
	}
}

func BenchmarkZapPrintf(b *testing.B) {
	logger := zap.New(zapcore.NewCore(
		zapcore.NewJSONEncoder(zap.NewProductionEncoderConfig()),
		zapcore.AddSync(io.Discard),
		zapcore.InfoLevel,
	)).Sugar()
	for i := 0; i < b.N; i++ {
		logger.Infof("rate=%s low=%d high=%f msg=%s", "15", 16, 123.2, msg)
	}
}

func BenchmarkZapCaller(b *testing.B) {
	logger := zap.New(zapcore.NewCore(
		zapcore.NewJSONEncoder(zap.NewProductionEncoderConfig()),
		zapcore.AddSync(io.Discard),
		zapcore.InfoLevel),
		zap.AddCaller(),
	).Sugar()
	for i := 0; i < b.N; i++ {
		logger.Infow(msg, "rate", "15", "low", 16, "high", 123.2)
	}
}

func BenchmarkZapInterface(b *testing.B) {
	logger := zap.New(zapcore.NewCore(
		zapcore.NewJSONEncoder(zap.NewProductionEncoderConfig()),
		zapcore.AddSync(io.Discard),
		zapcore.InfoLevel,
	)).Sugar()
	for i := 0; i < b.N; i++ {
		logger.Infow(msg, "object", &obj)
	}
}

func BenchmarkZeroLogDisable(b *testing.B) {
	zerolog.SetGlobalLevel(zerolog.InfoLevel)
	logger := zerolog.New(io.Discard).With().Timestamp().Logger()
	for i := 0; i < b.N; i++ {
		logger.Debug().Str("rate", "15").Int("low", 16).Float32("high", 123.2).Msg(msg)
	}
}

func BenchmarkZeroLogNormal(b *testing.B) {
	logger := zerolog.New(io.Discard).With().Timestamp().Logger()
	for i := 0; i < b.N; i++ {
		logger.Info().Str("rate", "15").Int("low", 16).Float32("high", 123.2).Msg(msg)
	}
}

func BenchmarkZeroLogPrintf(b *testing.B) {
	logger := zerolog.New(io.Discard).With().Timestamp().Logger()
	for i := 0; i < b.N; i++ {
		logger.Info().Msgf("rate=%s low=%d high=%f msg=%s", "15", 16, 123.2, msg)
	}
}

func BenchmarkZeroLogCaller(b *testing.B) {
	logger := zerolog.New(io.Discard).With().Caller().Timestamp().Logger()
	for i := 0; i < b.N; i++ {
		logger.Info().Str("rate", "15").Int("low", 16).Float32("high", 123.2).Msg(msg)
	}
}

func BenchmarkZeroLogInterface(b *testing.B) {
	logger := zerolog.New(io.Discard).With().Timestamp().Logger()
	for i := 0; i < b.N; i++ {
		logger.Info().Interface("object", &obj).Msg(msg)
	}
}

func BenchmarkPhusLogDisable(b *testing.B) {
	logger := log.Logger{Level: log.InfoLevel, Writer: log.IOWriter{io.Discard}}
	for i := 0; i < b.N; i++ {
		logger.Debug().Str("rate", "15").Int("low", 16).Float32("high", 123.2).Msg(msg)
	}
}

func BenchmarkPhusLogNormal(b *testing.B) {
	logger := log.Logger{Writer: log.IOWriter{io.Discard}}
	for i := 0; i < b.N; i++ {
		logger.Info().Str("rate", "15").Int("low", 16).Float32("high", 123.2).Msg(msg)
	}
}

func BenchmarkPhusLogPrintf(b *testing.B) {
	logger := log.Logger{Writer: log.IOWriter{io.Discard}}
	for i := 0; i < b.N; i++ {
		logger.Info().Msgf("rate=%s low=%d high=%f msg=%s", "15", 16, 123.2, msg)
	}
}

func BenchmarkPhusLogCaller(b *testing.B) {
	logger := log.Logger{Caller: 1, Writer: log.IOWriter{io.Discard}}
	for i := 0; i < b.N; i++ {
		logger.Info().Str("rate", "15").Int("low", 16).Float32("high", 123.2).Msg(msg)
	}
}

func BenchmarkPhusLogInterface(b *testing.B) {
	logger := log.Logger{Writer: log.IOWriter{io.Discard}}
	for i := 0; i < b.N; i++ {
		logger.Info().Interface("object", &obj).Msg(msg)
	}
}

A Performance result as below, for daily benchmark results see github actions

goos: linux
goarch: amd64
cpu: AMD EPYC 7763 64-Core Processor

BenchmarkSlogDisable-4        	1000000000	         8.438 ns/op	       0 B/op	       0 allocs/op
BenchmarkSlogNormal-4         	 8976294	      1337 ns/op	     120 B/op	       3 allocs/op
BenchmarkSlogPrintf-4         	11826330	      1020 ns/op	      80 B/op	       1 allocs/op
BenchmarkSlogCaller-4         	 5401592	      2202 ns/op	     688 B/op	       9 allocs/op
BenchmarkSlogInterface-4      	 9175490	      1306 ns/op	     112 B/op	       2 allocs/op

BenchmarkZapDisable-4         	192014136	        62.76 ns/op	     192 B/op	       1 allocs/op
BenchmarkZapNormal-4          	16954477	       712.5 ns/op	     192 B/op	       1 allocs/op
BenchmarkZapPrintf-4          	12580900	       948.9 ns/op	      80 B/op	       1 allocs/op
BenchmarkZapCaller-4          	 5984283	      2011 ns/op	     440 B/op	       3 allocs/op
BenchmarkZapInterface-4       	11327323	      1055 ns/op	     224 B/op	       2 allocs/op

BenchmarkZeroLogDisable-4     	1000000000	         9.940 ns/op	       0 B/op	       0 allocs/op
BenchmarkZeroLogNormal-4      	36321386	       331.3 ns/op	       0 B/op	       0 allocs/op
BenchmarkZeroLogPrintf-4      	18025590	       656.6 ns/op	      80 B/op	       1 allocs/op
BenchmarkZeroLogCaller-4      	 9181623	      1311 ns/op	     304 B/op	       4 allocs/op
BenchmarkZeroLogInterface-4   	19666522	       611.6 ns/op	      48 B/op	       1 allocs/op

BenchmarkPhusLogDisable-4     	1000000000	         9.599 ns/op	       0 B/op	       0 allocs/op
BenchmarkPhusLogNormal-4      	51243552	       233.3 ns/op	       0 B/op	       0 allocs/op
BenchmarkPhusLogPrintf-4      	23395036	       510.9 ns/op	       0 B/op	       0 allocs/op
BenchmarkPhusLogCaller-4      	23624828	       506.7 ns/op	       0 B/op	       0 allocs/op
BenchmarkPhusLogInterface-4   	21708657	       554.7 ns/op	       0 B/op	       0 allocs/op

PASS
ok  	bench	256.505s

In summary, phuslog offers a blend of low latency, minimal memory usage, and efficient logging across various scenarios, making it an excellent option for high-performance logging in Go applications.

A Real World Example

The example starts a geoip http server which supports change log level dynamically

package main

import (
	"encoding/json"
	"fmt"
	"net"
	"net/http"
	"os"

	"github.com/phuslu/iploc"
	"github.com/phuslu/log"
)

type Config struct {
	Listen struct {
		Tcp string
	}
	Log struct {
		Level   string
		Maxsize int64
		Backups int
	}
}

type Handler struct {
	Config       *Config
	AccessLogger log.Logger
}

func (h *Handler) ServeHTTP(rw http.ResponseWriter, req *http.Request) {
	reqID := log.NewXID()
	remoteIP, _, _ := net.SplitHostPort(req.RemoteAddr)
	geo := iploc.Country(net.ParseIP(remoteIP))

	h.AccessLogger.Log().
		Xid("req_id", reqID).
		Str("host", req.Host).
		Bytes("geo", geo).
		Str("remote_ip", remoteIP).
		Str("request_uri", req.RequestURI).
		Str("user_agent", req.UserAgent()).
		Str("referer", req.Referer()).
		Msg("access log")

	switch req.RequestURI {
	case "/debug", "/info", "/warn", "/error":
		log.DefaultLogger.SetLevel(log.ParseLevel(req.RequestURI[1:]))
	default:
		fmt.Fprintf(rw, `{"req_id":"%s","ip":"%s","geo":"%s"}`, reqID, remoteIP, geo)
	}
}

func main() {
	config := new(Config)
	err := json.Unmarshal([]byte(`{
		"listen": {
			"tcp": ":8080"
		},
		"log": {
			"level": "debug",
			"maxsize": 1073741824,
			"backups": 5
		}
	}`), config)
	if err != nil {
		log.Fatal().Msgf("json.Unmarshal error: %+v", err)
	}

	handler := &Handler{
		Config: config,
		AccessLogger: log.Logger{
			Writer: &log.FileWriter{
				Filename:   "access.log",
				MaxSize:    config.Log.Maxsize,
				MaxBackups: config.Log.Backups,
				LocalTime:  true,
			},
		},
	}

	if log.IsTerminal(os.Stderr.Fd()) {
		log.DefaultLogger = log.Logger{
			Level:      log.ParseLevel(config.Log.Level),
			Caller:     1,
			TimeFormat: "15:04:05",
			Writer: &log.ConsoleWriter{
				ColorOutput:    true,
				EndWithMessage: true,
			},
		}
		handler.AccessLogger = log.DefaultLogger
	} else {
		log.DefaultLogger = log.Logger{
			Level: log.ParseLevel(config.Log.Level),
			Writer: &log.FileWriter{
				Filename:   "main.log",
				MaxSize:    config.Log.Maxsize,
				MaxBackups: config.Log.Backups,
				LocalTime:  true,
			},
		}
	}

	server := &http.Server{
		Addr:     config.Listen.Tcp,
		ErrorLog: log.DefaultLogger.Std(log.ErrorLevel, nil, "", 0),
		Handler:  handler,
	}

	log.Fatal().Err(server.ListenAndServe()).Msg("listen failed")
}

Acknowledgment

This log is heavily inspired by zerolog, glog, gjson and lumberjack.

log's People

Contributors

fireflycons avatar huazhihao avatar juneezee avatar kmrgirish avatar mislink avatar ogimenezb avatar phuslu avatar rfyiamcool avatar suzaku avatar ttwshell avatar u2386 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

log's Issues

Docs request: clarification of optimization techniques

I would be interested in a blurb in the documentation somewhere on the techniques you've chosen to go with which provide this library with its performance over the others which you benchmark against. Often there is a tradeoff in terms of functionality or complexity when enhancing speed over alternatives. Alternatively, if there are no such concerns, and it's simply clever engineering, that's worth mentioning too, since it would help users contextualize their choice of this library over the alternatives.

Time() always format with local timezone

I notice when using func (e *Entry) Time(key string, t time.Time) *Entry , t always formatted using my local time zone, even if t has location set to UTC.
For comparison, zerolog will format with UTC if t has location set to UTC

MultiWriter based on io.Writer

Sometimes I need to write to different outputs with a selected level.
For example, given a Debug level I would like to be able to write to console, log file and syslog.

Have tried io.MultiWriter but ConsoleWriter does not implement io.Writer method.

Alternatively we could implement something like https://github.com/rs/zerolog/blob/117cb53bc66413d9a810ebed32383e53416347e3/writer.go#L88

I have tried several things and the only option I see is MultiWriter but it's level based not io.Writer based.

Any suggestions?

How about supporting global level?

Setting level to a specific logger is not very useful, normally we want to change level for all created logger. There is no way to do it currently.

FileWriter should be lockfree

*os.File.Write already has a lock for concurrency call. so we could remove current mutex of FileWriter.
A original but outdated implementation reverted at b6dc7fe

启动多个应用程序写入同一日志文件

&log.FileWriter{
    Filename:     "logs/access.log",
    FileMode:     0600,
    MaxSize:      100 << 20,
    MaxBackups:   7,
    EnsureFolder: true,
    LocalTime:    true,
    TimeFormat:   "run",
},
lrwxrwxrwx 1 root root   14 Dec 28 13:50 access.log -> access.run.log*
-rwxrwxrwx 1 root root 1549 Dec 28 13:52 access.run.log*

配置如上. 主要为了实现像 nginx 一样, 把所有日志都写在相同文件名中.
请问我开启了多个应用程序时, 会有多个进程同时写相同的日志文件, 有问题吗?
或者有什么进程安全参数设置项吗?
谢谢!

Compatibility with rs/logbench

I like this logger a lot because it has zero dependencies and an intuitive API. Therefore, I hope that it can gain more popularity among the Go community.

I have created a draft pull request rs/logbench#4 in the rs/logbench repository to include this logger. However, this logger currently does not pass the validation of logbench, more details can be found in rs/logbench#4.

To pass the logbench validation, this logger must

  1. Allow timestamp to be disabled
  2. Support different encoder for Time and Duration

Panic on using for aws s3

project code here

http://github.com/pankajsoni19/go-kafka-to-s3


admin@ip-41-2-0-28:~/go-kafka-to-s3$ panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x7891ed]

goroutine 140 [running]:
github.com/phuslu/log.(*FileWriter).Write(0x0, 0xc000f78000, 0x99, 0xa0, 0x98, 0xc000f78000, 0x18)
	/home/admin/go/pkg/mod/github.com/phuslu/[email protected]/file.go:92 +0x2d
log.(*Logger).Output(0xc000082cd0, 0x2, 0xc0000ae300, 0x80, 0x0, 0x0)
	/usr/local/go/src/log/log.go:181 +0x284
log.(*Logger).Printf(0xc000082cd0, 0xc0000ac1b0, 0x2e, 0xc000150320, 0x1, 0x1)
	/usr/local/go/src/log/log.go:188 +0x7e
github.com/aws/smithy-go/logging.StandardLogger.Logf(...)
	/home/admin/go/pkg/mod/github.com/aws/[email protected]/logging/logger.go:74
github.com/aws/aws-sdk-go-v2/aws/retry.Attempt.logf(...)
	/home/admin/go/pkg/mod/github.com/aws/[email protected]/aws/retry/middleware.go:59
github.com/aws/aws-sdk-go-v2/aws/retry.Attempt.handleAttempt(0xc7b101, 0xc7bde0, 0xc0000cc420, 0xc04c90, 0xc7b1e0, 0xc000138b40, 0xbd4620, 0xc000138a50, 0xc728a0, 0xc00000e560, ...)
	/home/admin/go/pkg/mod/github.com/aws/[email protected]/aws/retry/middleware.go:150 +0x9af
github.com/aws/aws-sdk-go-v2/aws/retry.Attempt.HandleFinalize(0xed7b58701, 0xc7bde0, 0xc0000cc420, 0xc04c90, 0xc7b1e0, 0xc000138a20, 0xbd4620, 0xc000138960, 0xc728a0, 0xc00000e560, ...)
	/home/admin/go/pkg/mod/github.com/aws/[email protected]/aws/retry/middleware.go:86 +0x305
github.com/aws/smithy-go/middleware.decoratedFinalizeHandler.HandleFinalize(...)
	/home/admin/go/pkg/mod/github.com/aws/[email protected]/middleware/step_finalize.go:200
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding.(*DisableGzip).HandleFinalize(0x121ae80, 0xc7b1e0, 0xc000138a20, 0xbd4620, 0xc000138960, 0xc728a0, 0xc00000e580, 0xc00000e5a0, 0xc000f577d0, 0xc00000e5a0, ...)
	/home/admin/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/internal/[email protected]/accept_encoding_gzip.go:67 +0x167
github.com/aws/smithy-go/middleware.decoratedFinalizeHandler.HandleFinalize(...)
	/home/admin/go/pkg/mod/github.com/aws/[email protected]/middleware/step_finalize.go:200
github.com/aws/smithy-go/middleware.(*FinalizeStep).HandleMiddleware(0xc0000b2048, 0xc7b1e0, 0xc000138a20, 0xbd4620, 0xc000138960, 0xc728c0, 0xc00000e2a0, 0xa81a664ba2bfe8a1, 0xc76c51a3c24b8b70, 0xb8cb68db0a658c3c, ...)
	/home/admin/go/pkg/mod/github.com/aws/[email protected]/middleware/step_finalize.go:114 +0x1b7
github.com/aws/smithy-go/middleware.decoratedHandler.Handle(...)
	/home/admin/go/pkg/mod/github.com/aws/[email protected]/middleware/middleware.go:57
github.com/aws/smithy-go/middleware.buildWrapHandler.HandleBuild(...)
	/home/admin/go/pkg/mod/github.com/aws/[email protected]/middleware/step_build.go:184
github.com/aws/aws-sdk-go-v2/aws/middleware.(*requestUserAgent).HandleBuild(0xc0001500d0, 0xc7b1e0, 0xc000138a20, 0xbd4620, 0xc000138960, 0xc72840, 0xc0001501d0, 0x4461900b0dca02ef, 0x10, 0xb51260, ...)
	/home/admin/go/pkg/mod/github.com/aws/[email protected]/aws/middleware/user_agent.go:232 +0x3b7
github.com/aws/smithy-go/middleware.decoratedBuildHandler.HandleBuild(...)
	/home/admin/go/pkg/mod/github.com/aws/[email protected]/middleware/step_build.go:200
github.com/aws/aws-sdk-go-v2/aws/signer/v4.(*contentSHA256Header).HandleBuild(0x121ae80, 0xc7b1e0, 0xc000138a20, 0xbd4620, 0xc000138960, 0xc72860, 0xc00000e4a0, 0x121ae80, 0xb6e4c0, 0xc0001389f0, ...)
	/home/admin/go/pkg/mod/github.com/aws/[email protected]/aws/signer/v4/middleware.go:195 +0x1dd
github.com/aws/smithy-go/middleware.decoratedBuildHandler.HandleBuild(...)
	/home/admin/go/pkg/mod/github.com/aws/[email protected]/middleware/step_build.go:200
github.com/aws/aws-sdk-go-v2/aws/signer/v4.(*computePayloadSHA256).HandleBuild(0x121ae80, 0xc7b1e0, 0xc000138870, 0xbd4620, 0xc000138960, 0xc72860, 0xc00000e4c0, 0x0, 0x0, 0x7f65e4342500, ...)
	/home/admin/go/pkg/mod/github.com/aws/[email protected]/aws/signer/v4/middleware.go:156 +0x27c
github.com/aws/smithy-go/middleware.decoratedBuildHandler.HandleBuild(...)
	/home/admin/go/pkg/mod/github.com/aws/[email protected]/middleware/step_build.go:200
github.com/aws/smithy-go/transport/http.(*ComputeContentLength).HandleBuild(0x121ae80, 0xc7b1e0, 0xc000138870, 0xbd4620, 0xc000138960, 0xc72860, 0xc00000e4e0, 0x237b1360e7940a64, 0xc000f5a160, 0xe91575e0efd69d15, ...)
	/home/admin/go/pkg/mod/github.com/aws/[email protected]/transport/http/middleware_content_length.go:55 +0x17f
github.com/aws/smithy-go/middleware.decoratedBuildHandler.HandleBuild(...)
	/home/admin/go/pkg/mod/github.com/aws/[email protected]/middleware/step_build.go:200
github.com/aws/aws-sdk-go-v2/aws/middleware.ClientRequestID.HandleBuild(0xc7b1e0, 0xc000138870, 0xbd4620, 0xc000138960, 0xc72860, 0xc00000e500, 0x20, 0x20, 0x7f66176217d0, 0xc00000e520, ...)
	/home/admin/go/pkg/mod/github.com/aws/[email protected]/aws/middleware/middleware.go:42 +0x1ed
github.com/aws/smithy-go/middleware.decoratedBuildHandler.HandleBuild(...)
	/home/admin/go/pkg/mod/github.com/aws/[email protected]/middleware/step_build.go:200
github.com/aws/smithy-go/middleware.(*BuildStep).HandleMiddleware(0xc0000b2040, 0xc7b1e0, 0xc000138870, 0xbd4620, 0xc000138960, 0xc728c0, 0xc00000e2c0, 0xc000f580a0, 0x0, 0x0, ...)
	/home/admin/go/pkg/mod/github.com/aws/[email protected]/middleware/step_build.go:114 +0x1b7
github.com/aws/smithy-go/middleware.decoratedHandler.Handle(...)
	/home/admin/go/pkg/mod/github.com/aws/[email protected]/middleware/middleware.go:57
github.com/aws/smithy-go/middleware.serializeWrapHandler.HandleSerialize(...)
	/home/admin/go/pkg/mod/github.com/aws/[email protected]/middleware/step_serialize.go:192
github.com/aws/aws-sdk-go-v2/service/s3/internal/customizations.(*removeBucketFromPathMiddleware).HandleSerialize(0x121ae80, 0xc7b1e0, 0xc000138870, 0xb3d4e0, 0xc0000aa140, 0xbd4620, 0xc000138960, 0xc72980, 0xc000150140, 0xc0001161b0, ...)
	/home/admin/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/[email protected]/internal/customizations/remove_bucket_middleware.go:27 +0x27a
github.com/aws/smithy-go/middleware.decoratedSerializeHandler.HandleSerialize(...)
	/home/admin/go/pkg/mod/github.com/aws/[email protected]/middleware/step_serialize.go:208
github.com/aws/aws-sdk-go-v2/service/s3/internal/customizations.(*updateEndpoint).HandleSerialize(0xc00000e220, 0xc7b1e0, 0xc000138870, 0xb3d4e0, 0xc0000aa140, 0xbd4620, 0xc000138960, 0xc72900, 0xc00000e3c0, 0xc000138840, ...)
	/home/admin/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/[email protected]/internal/customizations/update_endpoint.go:175 +0x213
github.com/aws/smithy-go/middleware.decoratedSerializeHandler.HandleSerialize(...)
	/home/admin/go/pkg/mod/github.com/aws/[email protected]/middleware/step_serialize.go:208
github.com/aws/aws-sdk-go-v2/service/internal/s3shared.(*EnableDualstack).HandleSerialize(0xc00000e1e0, 0xc7b1e0, 0xc000138870, 0xb3d4e0, 0xc0000aa140, 0xbd4620, 0xc000138960, 0xc72900, 0xc00000e3e0, 0x1, ...)
	/home/admin/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/internal/[email protected]/update_endpoint.go:75 +0x1b1
github.com/aws/smithy-go/middleware.decoratedSerializeHandler.HandleSerialize(...)
	/home/admin/go/pkg/mod/github.com/aws/[email protected]/middleware/step_serialize.go:208
github.com/aws/aws-sdk-go-v2/service/s3.(*awsRestxml_serializeOpPutObject).HandleSerialize(0x121ae80, 0xc7b1e0, 0xc000138870, 0xb3d4e0, 0xc0000aa140, 0xbd4620, 0xc000138720, 0xc72900, 0xc00000e400, 0xc000138840, ...)
	/home/admin/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/[email protected]/serializers.go:6936 +0x3a9
github.com/aws/smithy-go/middleware.decoratedSerializeHandler.HandleSerialize(...)
	/home/admin/go/pkg/mod/github.com/aws/[email protected]/middleware/step_serialize.go:208
github.com/aws/aws-sdk-go-v2/service/s3/internal/customizations.(*processARNResource).HandleSerialize(0xc00000e1c0, 0xc7b1e0, 0xc000138870, 0xb3d4e0, 0xc0000aa140, 0xbd4620, 0xc000138720, 0xc72900, 0xc00000e420, 0x121ae80, ...)
	/home/admin/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/[email protected]/internal/customizations/process_arn_resource.go:48 +0x1382
github.com/aws/smithy-go/middleware.decoratedSerializeHandler.HandleSerialize(...)
	/home/admin/go/pkg/mod/github.com/aws/[email protected]/middleware/step_serialize.go:208
github.com/aws/aws-sdk-go-v2/service/s3.(*ResolveEndpoint).HandleSerialize(0xc00000e140, 0xc7b1e0, 0xc0001386f0, 0xb3d4e0, 0xc0000aa140, 0xbd4620, 0xc000138720, 0xc72900, 0xc00000e440, 0x7a0b25, ...)
	/home/admin/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/[email protected]/endpoints.go:111 +0x5c7
github.com/aws/smithy-go/middleware.decoratedSerializeHandler.HandleSerialize(...)
	/home/admin/go/pkg/mod/github.com/aws/[email protected]/middleware/step_serialize.go:208
github.com/aws/smithy-go/middleware.(*SerializeStep).HandleMiddleware(0xc000150080, 0xc7b1e0, 0xc0001386f0, 0xb3d4e0, 0xc0000aa140, 0xc728c0, 0xc00000e2e0, 0x3e8000081a4, 0x3e8, 0x203000, ...)
	/home/admin/go/pkg/mod/github.com/aws/[email protected]/middleware/step_serialize.go:122 +0x1fb
github.com/aws/smithy-go/middleware.decoratedHandler.Handle(...)
	/home/admin/go/pkg/mod/github.com/aws/[email protected]/middleware/middleware.go:57
github.com/aws/smithy-go/middleware.initializeWrapHandler.HandleInitialize(...)
	/home/admin/go/pkg/mod/github.com/aws/[email protected]/middleware/step_initialize.go:184
github.com/aws/aws-sdk-go-v2/service/s3.(*validateOpPutObject).HandleInitialize(0x121ae80, 0xc7b1e0, 0xc0001386f0, 0xb3d4e0, 0xc0000aa140, 0xc72960, 0xc0001500f0, 0x549046, 0xbafa40, 0xc0001386f0, ...)
	/home/admin/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/[email protected]/validators.go:1630 +0xdb
github.com/aws/smithy-go/middleware.decoratedInitializeHandler.HandleInitialize(...)
	/home/admin/go/pkg/mod/github.com/aws/[email protected]/middleware/step_initialize.go:200
github.com/aws/smithy-go/middleware.(*setLogger).HandleInitialize(0xc0001500c0, 0xc7b1e0, 0xc0001386c0, 0xb3d4e0, 0xc0000aa140, 0xc728e0, 0xc00000e340, 0x121ae80, 0xb6e4c0, 0xc000138690, ...)
	/home/admin/go/pkg/mod/github.com/aws/[email protected]/middleware/logging.go:45 +0xb5
github.com/aws/smithy-go/middleware.decoratedInitializeHandler.HandleInitialize(...)
	/home/admin/go/pkg/mod/github.com/aws/[email protected]/middleware/step_initialize.go:200
github.com/aws/aws-sdk-go-v2/aws/middleware.RegisterServiceMetadata.HandleInitialize(0xbe1fde, 0x2, 0xbe20a8, 0x2, 0xc0000b1ee6, 0x9, 0xbe4466, 0x9, 0xc7b1e0, 0xc0001382a0, ...)
	/home/admin/go/pkg/mod/github.com/aws/[email protected]/aws/middleware/metadata.go:40 +0xe7
github.com/aws/smithy-go/middleware.decoratedInitializeHandler.HandleInitialize(...)
	/home/admin/go/pkg/mod/github.com/aws/[email protected]/middleware/step_initialize.go:200
github.com/aws/aws-sdk-go-v2/service/internal/s3shared.(*ARNLookup).HandleInitialize(0xc0000b2058, 0xc7b1e0, 0xc0001382a0, 0xb3d4e0, 0xc0000aa140, 0xc728e0, 0xc00000e380, 0xc00000e3a0, 0xc000f497c0, 0xc00000e3a0, ...)
	/home/admin/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/internal/[email protected]/arn_lookup.go:39 +0xca
github.com/aws/smithy-go/middleware.decoratedInitializeHandler.HandleInitialize(...)
	/home/admin/go/pkg/mod/github.com/aws/[email protected]/middleware/step_initialize.go:200
github.com/aws/smithy-go/middleware.(*InitializeStep).HandleMiddleware(0xc0000b2038, 0xc7b1e0, 0xc0001382a0, 0xb3d4e0, 0xc0000aa140, 0xc728c0, 0xc00000e300, 0x0, 0xc728c0, 0xc00000e300, ...)
	/home/admin/go/pkg/mod/github.com/aws/[email protected]/middleware/step_initialize.go:114 +0x1b7
github.com/aws/smithy-go/middleware.decoratedHandler.Handle(...)
	/home/admin/go/pkg/mod/github.com/aws/[email protected]/middleware/middleware.go:57
github.com/aws/smithy-go/middleware.(*Stack).HandleMiddleware(0xc0000c8000, 0xc7b1e0, 0xc0001382a0, 0xb3d4e0, 0xc0000aa140, 0xc729a0, 0xc0001500e0, 0x0, 0xc729a0, 0xc0001500e0, ...)
	/home/admin/go/pkg/mod/github.com/aws/[email protected]/middleware/stack.go:109 +0x151
github.com/aws/smithy-go/middleware.decoratedHandler.Handle(...)
	/home/admin/go/pkg/mod/github.com/aws/[email protected]/middleware/middleware.go:57
github.com/aws/aws-sdk-go-v2/service/s3.(*Client).invokeOperation(0xc000130dc0, 0xc7b160, 0xc0000b0018, 0xbe4466, 0x9, 0xb3d4e0, 0xc0000aa140, 0x0, 0x0, 0x0, ...)
	/home/admin/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/[email protected]/api_client.go:165 +0x533
github.com/aws/aws-sdk-go-v2/service/s3.(*Client).PutObject(0xc000130dc0, 0xc7b160, 0xc0000b0018, 0xc0000aa140, 0x0, 0x0, 0x0, 0x12, 0xc00045fea0, 0xc00045fed8)
	/home/admin/go/pkg/mod/github.com/aws/aws-sdk-go-v2/service/[email protected]/api_op_PutObject.go:82 +0xfb
main.upload(0xc000143860, 0xc0000b1eb0, 0xb, 0xc0000ac0f0, 0x27, 0x0, 0x0)
	/home/admin/go-kafka-to-s3/main.go:235 +0x34c
main.setupRotation.func2.1(0xc0000ac060, 0x24, 0xc000143860, 0xc0000b1eb0, 0xb)
	/home/admin/go-kafka-to-s3/main.go:192 +0x172
created by main.setupRotation.func2
	/home/admin/go-kafka-to-s3/main.go:182 +0x1cd
^C
[1]+  Exit 2                  ./go-kafka-to-s3

Disable JSON in log.FileWriter

I have trying to implement logger in my app, have few things to be cleared as those are not present in readme or didn't get it how to do.
Those are:

  1. Disable JSON format and log like nginx log in log.FileWriter or like its there in ConsoleWriter basically formating in FileWriter
  2. Write JSON to console, can this be achieved by ConsoleWriter.Formatter
  3. It has been mentioned for log.MultiWriter that It can write level to different file, can we have custom/default log level and lock file to level only in that file or console.
  4. readme mentioned this log lib is inspired by lumberjack also but can lumberjack be used to rotate log file in log.FileWriter. Not sure about performance with cron for rotating logs.

is cbor logger needed?

Currently, there're Logger(for json) and TSVLogger(for tsv/csv) in this lib.
I'm not sure a cbor logger is needed or not, if you have strong reasons, please comment.

[QUESTION] - How to use graylog with this library

I'm trying to use Graylog as writer. Instead of logging as individual field log is printed as single string field.

Graylog library: https://github.com/Graylog2/go-gelf

How would I implement it for graylog

image

The code I'm using is this

gelfWriter, err := gelf.NewUDPWriter(grayLogAddr)
	if err != nil {
		panic(err)
	}
	log.DefaultLogger = log.Logger{
		Level:      log.InfoLevel,
		Caller:     0,
		TimeField:  "date",
		TimeFormat: "2006-01-02 15:04:05",
		Writer:     &log.IOWriter{gelfWriter},
	}
r := Request{
			requestId: rid,
			method:    c.Method(),
			host:      c.Hostname(),
			path:      c.Path(),
			request:   string(payload),
			response:  string(response),
			status:    c.Response().StatusCode(),
			latency:   fmt.Sprintf("%s", time.Since(start)),
		}
		log.Info().EmbedObject(&r).Msg("")

Single Quotation Mark

Given the following example:

var logger = struct {
	Console plog.Logger
	Pretty  plog.Logger
	File    plog.Logger
}{
	Console: plog.Logger{
		Level:  plog.TraceLevel,
		Caller: 1,
		Writer: &plog.IOWriter{os.Stdout},
	},
	Pretty: plog.Logger{
		Level:  plog.TraceLevel,
		Caller: 1,
		Writer: &plog.ConsoleWriter{
			ColorOutput:    true,
			QuoteString:    true,
			EndWithMessage: true,
		},
	},
	File: plog.Logger{
		Level:  plog.TraceLevel,
		Caller: 1,
		Writer: &plog.FileWriter{
			Filename:     "plog_test.log",
			FileMode:     0600,
			MaxSize:      100 * 1024 * 1024,
			MaxBackups:   7,
			EnsureFolder: true,
			LocalTime:    true,
		},
	},
}

func testSpecial() {
	logger.Console.Info().Msg("can't handle single quotation mark")
	logger.Pretty.Info().Msg("can't handle single quotation mark")
	logger.File.Info().Msg("can't handle single quotation mark")
}

I have the following outputs:
Console:
{"time":"2021-06-12T18:10:42.379+02:00","level":"info","caller":"main.go:233","goid":1,"message":"can\u0027t handle single quotation mark"}
Pretty:
2021-06-12T18:10:42.379+02:00 INF 1 main.go:234 > can't handle single quotation mark
File:
{"time":"2021-06-12T18:10:42.379+02:00","level":"info","caller":"main.go:235","goid":1,"message":"can\u0027t handle single quotation mark"}

It seems it can\u0027t handle it correctly... lol

FileWriter Filename

This could be considered as an improvement? maybe?

When you output to file, you have name.timestamp.ext that's ok... but when there are many cases when this is a little bit unconfutable.
I think I should be name.ext and when rotated name.timestamp.ext

  • If you are testing
  • If you have a console command options
  • If you need to stop and start the project a couple of times
  • ...

When rotation happens, because of whatever condition, having timestamp is ok, but if there are no conditions that have been meet it should be just the name and append data.

For compatibility issues we could implement an AutoRotateOnInitialize (bool), or OnlyTimestampOnRotate (bool) a simpler name...

Incorrect caller when using slog

Hi!

package main

import (
	"log/slog"

	plog "github.com/phuslu/log"
)

func main() {
	plog.DefaultLogger = plog.Logger{
		Level: plog.DebugLevel,
		Caller: 1,
	}

	var logger *slog.Logger
	logger = plog.DefaultLogger.Slog()

	plog.Info().Msg("msg from phuslu/log")
	logger.Info("info msg from slog")
	logger.Error("error msg from slot")
}

Output:

{"time":"2023-09-06T15:36:41.406+08:00","level":"info","caller":"main.go:18","goid":1,"message":"msg from phuslu/log"}
{"time":"2023-09-06T15:36:41.406+08:00","caller":"logger_go1.21.go:30","goid":1,"message":"info msg from slog"}
{"time":"2023-09-06T15:36:41.406+08:00","caller":"logger_go1.21.go:30","goid":1,"message":"error msg from slot"}

And missing level when using log/slog, it would be great to have support for slog levels!

gin logger documentation: what is ginlogger.Config?

The documentation for the gin logger mentions this:

	// Custom logger
	r.Use(ginlogger.SetLogger(ginlogger.Config{
		Logger: &log.Logger{
			Writer: &log.FileWriter{
				Filename: "access.log",
				MaxSize:  1024 * 1024 * 1024,
			},
		},
		Context: log.NewContext(nil).Str("foo", "bar").Value(),
		Skip:    func(c *gin.Context) bool {
			if c.Request.URL.Path == "/backdoor" {
				return true
			}
			return false
		},
	}))

What is ginlogger.Config ?

The documentation mentioned is at:

https://github.com/phuslu/log-contrib/tree/master/gin

W3C extended log file format fromatter

With very little effort I was able to implement a custom Formatter to make the ConsoleWriter write log lines in a W3C Extended Log File Format compliant way (see: https://www.w3.org/TR/WD-logfile.html)

Although JSON structured logging is all the rage today (and, in fact, I personally prefer JSON big time) there's a lot of log analyzers out there that are unable to read JSON, but are perfectly capable of importing W3C log files.

The problem is that there is no Formatter in the FileWriter writer, which is exactly where it would need to be, if we are to be able to save log files in W3C format.

The way I see it, this could be achieved 2 ways:

  • add a Formatter to the existing FileWriter interface
  • create a whole new W3CLogFileWriter interface (probably not a good idea)

Also, W3C log files require a "header" to be written at the beginning of every file, so ideally each time the file is rotated this header should be automatically written to the file, before we can actually add log lines to it.

Here's a W3C log file header example (taken from Microsoft IIS):

#Software: Internet Information Services 8.0 
#Version: 1.0 
#Date: 2021-05-02 17:42:15 
#Fields: time c-ip cs-method cs-uri-stem sc-status cs-version

This could probably be achieved with some custom wizardry in the Cleaner function of the FileWriter interface, but while we're at it, why not make everyone life's easier, and provide a string property for it, and have the Writer automagically add it at the beginning of each new file, right when the log its rotated?

JSON logging is still far far better, I think we can all agree on that. But W3C log file format is not dead, and is required by more users than I originally thought possible. Any feedback on this would be greatly appreciated. Thank you in advance.

有内置日志采样功能的计划吗?

// Will let 5 debug messages per period of 1 second.
// Over 5 debug message, 1 every 100 debug messages are logged.
// Other levels are not sampled.
sampled := log.Sample(zerolog.LevelSampler{
    DebugSampler: &zerolog.BurstSampler{
        Burst: 5,
        Period: 1*time.Second,
        NextSampler: &zerolog.BasicSampler{N: 100},
    },
})
sampled.Debug().Msg("hello world")

// Output: {"time":1494567715,"level":"debug","message":"hello world"}

您好吖.

很喜欢干净整洁的库, 高性能无依赖是天花板. 功能也很全面.

不过, 看了一圈没发现上面的日志采样功能, 在一些高并发场景我们可能更希望1秒有几个日志即可. 或意外产生错误时很可能疯狂记录错误日志, 此时也希望仅采样错误来记录.

如果已有此功能或简单方案, 请贴下范例.

若无, 请问有计划吗?

我感觉这样就完全可以替代我心中理想的日志库了.

谢谢~~

如何把所有日志放一个文件

`log.DefaultLogger.Writer = &log.MultiEntryWriter{

	&log.FileWriter{
		Filename:  "main.log",
		MaxSize:   100 << 20,
		LocalTime: true,
	},
}`

每次启动会生一个新的"main.2021-10-11T10-32-41.log" 类似的文件,如何控制,都写在同一个文件下,而不是新生成。

log With()

Great logger, simple and easy to use. Are you looking to add a with() function, as this is most important when logging web transactions and correlation of single web requests, through the core application.

Thanks,
Bob

Long filename in `caller`

It would be much convenient if we could log long filename and line number, and stdlib supports this.

FileWriter comment mentions MaxAge but that field does not exist

The comment for FileWriter mentions MaxAge but that field does not exist.

Any files with an encoded timestamp older than MaxAge days are deleted, regardless of MaxBackups

MaxAge would be nice to have if this library is meant to be a drop-in replacement for Lumberjack.

How to use it with echo ?

I am on echo v4

Logger = log.Logger{
		Level: log.InfoLevel,
		Writer: &log.AsyncWriter{
			ChannelSize: 100,
			Writer:      fileWriter,
		},
	}

logConfig := middleware.DefaultLoggerConfig
logConfig.Output = Logger   // error here
e.Use(middleware.LoggerWithConfig(logConfig))  

Error is

cannot use Logger.Writer (variable of type log.Writer) as io.Writer value in assignment: missing method Writecompiler

Callers - vary stack depth

Hi @phuslu

I have a case where I'm wrapping logger inside another abstraction, and I want caller to log the point at which the abstraction is called, rather than Info() etc.

How about making the Caller member actually by the number of stack frames to look back instead of just a switch?
e.g. 2/-2 would look one frame further. Is there a better way than simply overloading the meaning of Caller? For instance a new property in case existing users are relying on Caller simply being not zero and not explicitly 1/-1

I could look into it

it does not rotate if you have a TimeFormat defines and a MaxSize then reach MaxSize with the same TimeFormat

I would like to comment or expand FileWriter so we can al better understand it.

  • Files are saved according to name.timestamp.ext.
  • File is always created with the defined or default TimeFormat.
  • Once program is initiated it only autorotates when condition for MaxSize is meet and has a different name.
  • If the filename is the same it never autorotates (Should this be like this?)
  • If TimeFormat is default and goes by minutes each minute it generates a new log file. (Should it be like this?)
  • There is no way to compres. (Maybe we can implement it or add AfterRotate func for user to do what he wants)

I thinks this deserves a thought to see if it marches what people usually expect.

When rotating if file exists and MaxSize is meet we should append _n
Don't know how to go about file name... if it does not meet MaxSize should we use the same name whether it takes 30m or 30 days and only add timestamp on rotate?

ConsoleWriter cant print Err

use consolewriter will lost Err() info.
like:

package main

import (
	"errors"

	"github.com/phuslu/log"
)

var glog = (&log.Logger{
	Level:      log.InfoLevel,
	Caller:     1,
	TimeFormat: "0102 15:04:05.999999",
	Writer: &log.ConsoleWriter{
		ColorOutput:    false,
		QuoteString:    true,
		EndWithMessage: true,
	},
})

func main() {
	glog.Error().Err(errors.New("an error")).Msgf("hello glog")
}

#output: 
1110 23:00:00 ERR 1 prog.go:21 > hello glog

Gzip examples writes truncated file

Following the example in #19

async := &log.AsyncWriter{
    ChannelSize: 1,
    Writer: log.IOWriter{
        Writer: gzip.NewWriter(&log.FileWriter{
            Filename:   "foo.log.gz",
            MaxSize:    50 * 1024 * 1024,
            MaxBackups: 7,
        }),
    },
}
logger := log.Logger{
    Writer: async,
}
logger.Info().Msg("hello phuslu")
async.Close()
ls -la
total 40
-rw-r--r--  1 cgrindst  staff    10B May 20 11:07 foo.log.2022-05-20T15-07-25.gz
lrwxr-xr-x  1 cgrindst  staff    30B May 20 11:07 foo.log.gz@ -> foo.log.2022-05-20T15-07-25.gz

file foo.log.2022-05-20T15-07-25.gz 
foo.log.2022-05-20T15-07-25.gz: gzip compressed data, truncated

gunzip foo.log.2022-05-20T15-07-25.gz 
gunzip: foo.log.2022-05-20T15-07-25.gz: unexpected end of file
gunzip: foo.log.2022-05-20T15-07-25.gz: uncompress failed

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.