cybozu-go / log Goto Github PK

View Code? Open in Web Editor NEW

28.0 28.0 7.0 107 KB

Logging framework for Cybozu Go products

License: MIT License

Go 100.00%

log's People

Contributors

Stargazers

Watchers

Forkers

mattn windbg justforkin 42wim nokute78 asa-taka itchyny

log's Issues

[ToDo] Use forthcoming strings.ToValidUTF8

golang/go#25805

This library uses string([]rune(s)) technique to remove invalid Unicode characters
in multiple places. They should be replaced with strings.ToValidUTF8 which will
be added in Go 1.13.

$ git grep '\[\]rune'
json.go:                        t = string([]rune(t))
json.go:                        s = string([]rune(s))
logfmt.go:                      t = string([]rune(t))
logfmt.go:                      s = string([]rune(s))
plain.go:                       t = string([]rune(t))
plain.go:                       s = string([]rune(s))

dummy for add a new label and all Neco related repos

Output to systemd journald

https://www.freedesktop.org/software/systemd/man/systemd.journal-fields.html
https://github.com/coreos/go-systemd

General improvements

Use sync.Pool and format logs before acquiring the lock for better performance.
Accept any kind of types.
Switch log formats between plain, logfmt, and JSON line.
Correct the end of the file if broken in fileOpener.Open.
Drop fluentd support.

Formatter implementations extend the buffer regardless of its capacity

Describe the bug
Formatter implementations, JSONFormat, Logfmt, and PlainFormat extend the input buffer to arbitrary size, in contradiction to the expectation of the Formatter interface.

Environments

Version: 1.6.1
OS: macOS 12.2.1
Go: go1.17.8 darwin/amd64

To Reproduce
Steps to reproduce the behavior:
Run following code.

package main

import (
	"encoding/json"
	"fmt"
	"os"
	"strconv"
	"time"

	"github.com/cybozu-go/log"
)

func main() {
	var err error
	l := log.NewLogger()
	f := log.JSONFormat{Utsname: "test"}
	t := time.Now()

	data := map[string]interface{}{}
	for i := 0; i < 500; i++ {
		data["x"+strconv.Itoa(i)] = i
	}

	buf := make([]byte, 0, 1024)
	fmt.Printf("len: %d, cap: %d\n", len(buf), cap(buf))

	buf, err = f.Format(buf, l, t, log.LvDebug, "test", data)
	if err != nil {
		fmt.Fprintf(os.Stderr, "%s\n", err)
		os.Exit(1)
	}
	fmt.Printf("%s\n", buf)
	fmt.Printf("len: %d, cap: %d\n", len(buf), cap(buf))

	err = json.Unmarshal(buf, &data)
	if err != nil {
		fmt.Fprintf(os.Stderr, "%s\n", err)
		os.Exit(1)
	}
	fmt.Printf("%v\n", data)
}

Get following output.

len: 0, cap: 1024
{"topic":"main","logged_at":"2022-03-16T01:29:43.568542Z","severity":"debug","utsname":"test","message":"test","x54":54,"x60":60,"x271":271,"x325":325,"x326":326,"x357":357,"x71":71,"x140":140,"x218":218,"x251":251,"x290":290,"x149":149,"x195":195,"x300":300,"x478":478,"x482":482,"x187":187,"x196":196,"x248":248,"x388":388,"x25":25,"x57":57,"x118":118,"x215":215,"x371":371,"x460":460,"x275":275,"x277":277,"x369":369,"x69":69,"x77":77,"x264":264,"x61":61,"x308":308,"x352":352,"x453":453,"x479":479,"x18":18,"x100":100,"x410":410,"x31":31,"x170":170,"x297":297,"x429":429,"x66":66,"x336":336,"x83":83,"x184":184,"x304":304,"x446":446,"x128":128,"x197":197,"x310":310,"x370":370,"x434":434,"x306":306,"x245":245,"x262":262,"x339":339,"x438":438,"x439":439,"x212":212,"x432":432,"x239":239,"x376":376,"x405":405,"x337":337,"x134":134,"x235":235,"x246":246,"x333":333,"x97":97,"x202":202,"x292":292,"x469":469,"x11":11,"x327":327,"x343":343,"x427":427,"x497":497,"x90":90,"x377":377,"x129":129,"x492":492,"x283":283,"x313":313,"x471":471,"x5":5,"x39":39,"x183":183,"x243":243,"x398":398,"x457":457,"x59":59,"x110":110,"x417":417,"x164":164,"x255":255,"x299":299,"x477":477,"x194":194,"x267":267,"x380":380,"x456":456,"x94":94,"x127":127,"x78":78,"x123":123,"x143":143,"x413":413,"x440":440,"x488":488,"x394":394,"x412":412,"x485":485,"x51":51,"x139":139,"x153":153,"x342":342,"x346":346,"x390":390,"x147":147,"x384":384,"x496":496,"x34":34,"x150":150,"x217":217,"x130":130,"x137":137,"x259":259,"x360":360,"x386":386,"x391":391,"x395":395,"x318":318,"x27":27,"x80":80,"x224":224,"x354":354,"x385":385,"x236":236,"x418":418,"x494":494,"x242":242,"x375":375,"x58":58,"x289":289,"x358":358,"x365":365,"x14":14,"x53":53,"x287":287,"x288":288,"x158":158,"x226":226,"x250":250,"x257":257,"x356":356,"x162":162,"x266":266,"x443":443,"x63":63,"x132":132,"x319":319,"x76":76,"x124":124,"x172":172,"x6":6,"x9":9,"x119":119,"x315":315,"x359":359,"x155":155,"x347":347,"x397":397,"x433":433,"x445":445,"x475":475,"x444":444,"x72":72,"x203":203,"x279":279,"x38":38,"x392":392,"x449":449,"x495":495,"x75":75,"x201":201,"x301":301,"x348":348,"x367":367,"x448":448,"x455":455,"x491":491,"x47":47,"x256":256,"x324":324,"x353":353,"x407":407,"x454":454,"x470":470,"x481":481,"x4":4,"x50":50,"x101":101,"x409":409,"x419":419,"x451":451,"x151":151,"x163":163,"x180":180,"x247":247,"x344":344,"x117":117,"x316":316,"x73":73,"x192":192,"x211":211,"x372":372,"x402":402,"x7":7,"x49":49,"x191":191,"x291":291,"x295":295,"x307":307,"x87":87,"x165":165,"x231":231,"x244":244,"x366":366,"x148":148,"x303":303,"x12":12,"x32":32,"x174":174,"x265":265,"x302":302,"x447":447,"x206":206,"x227":227,"x254":254,"x414":414,"x422":422,"x452":452,"x363":363,"x464":464,"x56":56,"x169":169,"x294":294,"x381":381,"x204":204,"x186":186,"x232":232,"x240":240,"x345":345,"x387":387,"x408":408,"x86":86,"x98":98,"x136":136,"x146":146,"x378":378,"x474":474,"x126":126,"x330":330,"x487":487,"x458":458,"x468":468,"x16":16,"x79":79,"x198":198,"x282":282,"x355":355,"x399":399,"x499":499,"x293":293,"x321":321,"x331":331,"x23":23,"x175":175,"x216":216,"x225":225,"x238":238,"x332":332,"x95":95,"x131":131,"x214":214,"x268":268,"x437":437,"x467":467,"x40":40,"x62":62,"x171":171,"x103":103,"x105":105,"x489":489,"x55":55,"x65":65,"x142":142,"x305":305,"x320":320,"x426":426,"x24":24,"x335":335,"x37":37,"x84":84,"x113":113,"x154":154,"x252":252,"x483":483,"x96":96,"x273":273,"x362":362,"x19":19,"x219":219,"x263":263,"x424":424,"x278":278,"x8":8,"x311":311,"x393":393,"x421":421,"x463":463,"x13":13,"x48":48,"x281":281,"x430":430,"x436":436,"x498":498,"x89":89,"x138":138,"x188":188,"x222":222,"x88":88,"x428":428,"x209":209,"x322":322,"x441":441,"x30":30,"x109":109,"x189":189,"x329":329,"x473":473,"x35":35,"x241":241,"x298":298,"x26":26,"x181":181,"x269":269,"x285":285,"x182":182,"x465":465,"x17":17,"x22":22,"x210":210,"x270":270,"x423":423,"x431":431,"x160":160,"x234":234,"x296":296,"x383":383,"x396":396,"x403":403,"x42":42,"x114":114,"x435":435,"x472":472,"x0":0,"x120":120,"x229":229,"x361":361,"x404":404,"x480":480,"x20":20,"x99":99,"x177":177,"x340":340,"x115":115,"x476":476,"x28":28,"x104":104,"x112":112,"x116":116,"x167":167,"x258":258,"x411":411,"x43":43,"x64":64,"x81":81,"x106":106,"x107":107,"x220":220,"x70":70,"x145":145,"x341":341,"x400":400,"x490":490,"x179":179,"x207":207,"x314":314,"x44":44,"x253":253,"x350":350,"x29":29,"x323":323,"x364":364,"x373":373,"x176":176,"x425":425,"x466":466,"x85":85,"x102":102,"x152":152,"x420":420,"x280":280,"x416":416,"x166":166,"x349":349,"x382":382,"x462":462,"x486":486,"x193":193,"x205":205,"x351":351,"x368":368,"x108":108,"x144":144,"x309":309,"x10":10,"x21":21,"x33":33,"x91":91,"x199":199,"x286":286,"x74":74,"x122":122,"x135":135,"x493":493,"x213":213,"x46":46,"x178":178,"x200":200,"x274":274,"x312":312,"x133":133,"x159":159,"x141":141,"x221":221,"x338":338,"x374":374,"x328":328,"x401":401,"x2":2,"x3":3,"x41":41,"x190":190,"x233":233,"x249":249,"x442":442,"x161":161,"x276":276,"x317":317,"x450":450,"x459":459,"x461":461,"x45":45,"x67":67,"x156":156,"x168":168,"x173":173,"x379":379,"x1":1,"x185":185,"x223":223,"x15":15,"x93":93,"x111":111,"x121":121,"x260":260,"x92":92,"x261":261,"x272":272,"x68":68,"x82":82,"x125":125,"x157":157,"x228":228,"x237":237,"x389":389,"x52":52,"x208":208,"x484":484,"x36":36,"x230":230,"x284":284,"x334":334,"x406":406,"x415":415}

len: 5392, cap: 6784
map[logged_at:2022-03-16T01:29:43.568542Z message:test severity:debug topic:main utsname:test x0:0 x1:1 x10:10 x100:100 x101:101 x102:102 x103:103 x104:104 x105:105 x106:106 x107:107 x108:108 x109:109 x11:11 x110:110 x111:111 x112:112 x113:113 x114:114 x115:115 x116:116 x117:117 x118:118 x119:119 x12:12 x120:120 x121:121 x122:122 x123:123 x124:124 x125:125 x126:126 x127:127 x128:128 x129:129 x13:13 x130:130 x131:131 x132:132 x133:133 x134:134 x135:135 x136:136 x137:137 x138:138 x139:139 x14:14 x140:140 x141:141 x142:142 x143:143 x144:144 x145:145 x146:146 x147:147 x148:148 x149:149 x15:15 x150:150 x151:151 x152:152 x153:153 x154:154 x155:155 x156:156 x157:157 x158:158 x159:159 x16:16 x160:160 x161:161 x162:162 x163:163 x164:164 x165:165 x166:166 x167:167 x168:168 x169:169 x17:17 x170:170 x171:171 x172:172 x173:173 x174:174 x175:175 x176:176 x177:177 x178:178 x179:179 x18:18 x180:180 x181:181 x182:182 x183:183 x184:184 x185:185 x186:186 x187:187 x188:188 x189:189 x19:19 x190:190 x191:191 x192:192 x193:193 x194:194 x195:195 x196:196 x197:197 x198:198 x199:199 x2:2 x20:20 x200:200 x201:201 x202:202 x203:203 x204:204 x205:205 x206:206 x207:207 x208:208 x209:209 x21:21 x210:210 x211:211 x212:212 x213:213 x214:214 x215:215 x216:216 x217:217 x218:218 x219:219 x22:22 x220:220 x221:221 x222:222 x223:223 x224:224 x225:225 x226:226 x227:227 x228:228 x229:229 x23:23 x230:230 x231:231 x232:232 x233:233 x234:234 x235:235 x236:236 x237:237 x238:238 x239:239 x24:24 x240:240 x241:241 x242:242 x243:243 x244:244 x245:245 x246:246 x247:247 x248:248 x249:249 x25:25 x250:250 x251:251 x252:252 x253:253 x254:254 x255:255 x256:256 x257:257 x258:258 x259:259 x26:26 x260:260 x261:261 x262:262 x263:263 x264:264 x265:265 x266:266 x267:267 x268:268 x269:269 x27:27 x270:270 x271:271 x272:272 x273:273 x274:274 x275:275 x276:276 x277:277 x278:278 x279:279 x28:28 x280:280 x281:281 x282:282 x283:283 x284:284 x285:285 x286:286 x287:287 x288:288 x289:289 x29:29 x290:290 x291:291 x292:292 x293:293 x294:294 x295:295 x296:296 x297:297 x298:298 x299:299 x3:3 x30:30 x300:300 x301:301 x302:302 x303:303 x304:304 x305:305 x306:306 x307:307 x308:308 x309:309 x31:31 x310:310 x311:311 x312:312 x313:313 x314:314 x315:315 x316:316 x317:317 x318:318 x319:319 x32:32 x320:320 x321:321 x322:322 x323:323 x324:324 x325:325 x326:326 x327:327 x328:328 x329:329 x33:33 x330:330 x331:331 x332:332 x333:333 x334:334 x335:335 x336:336 x337:337 x338:338 x339:339 x34:34 x340:340 x341:341 x342:342 x343:343 x344:344 x345:345 x346:346 x347:347 x348:348 x349:349 x35:35 x350:350 x351:351 x352:352 x353:353 x354:354 x355:355 x356:356 x357:357 x358:358 x359:359 x36:36 x360:360 x361:361 x362:362 x363:363 x364:364 x365:365 x366:366 x367:367 x368:368 x369:369 x37:37 x370:370 x371:371 x372:372 x373:373 x374:374 x375:375 x376:376 x377:377 x378:378 x379:379 x38:38 x380:380 x381:381 x382:382 x383:383 x384:384 x385:385 x386:386 x387:387 x388:388 x389:389 x39:39 x390:390 x391:391 x392:392 x393:393 x394:394 x395:395 x396:396 x397:397 x398:398 x399:399 x4:4 x40:40 x400:400 x401:401 x402:402 x403:403 x404:404 x405:405 x406:406 x407:407 x408:408 x409:409 x41:41 x410:410 x411:411 x412:412 x413:413 x414:414 x415:415 x416:416 x417:417 x418:418 x419:419 x42:42 x420:420 x421:421 x422:422 x423:423 x424:424 x425:425 x426:426 x427:427 x428:428 x429:429 x43:43 x430:430 x431:431 x432:432 x433:433 x434:434 x435:435 x436:436 x437:437 x438:438 x439:439 x44:44 x440:440 x441:441 x442:442 x443:443 x444:444 x445:445 x446:446 x447:447 x448:448 x449:449 x45:45 x450:450 x451:451 x452:452 x453:453 x454:454 x455:455 x456:456 x457:457 x458:458 x459:459 x46:46 x460:460 x461:461 x462:462 x463:463 x464:464 x465:465 x466:466 x467:467 x468:468 x469:469 x47:47 x470:470 x471:471 x472:472 x473:473 x474:474 x475:475 x476:476 x477:477 x478:478 x479:479 x48:48 x480:480 x481:481 x482:482 x483:483 x484:484 x485:485 x486:486 x487:487 x488:488 x489:489 x49:49 x490:490 x491:491 x492:492 x493:493 x494:494 x495:495 x496:496 x497:497 x498:498 x499:499 x5:5 x50:50 x51:51 x52:52 x53:53 x54:54 x55:55 x56:56 x57:57 x58:58 x59:59 x6:6 x60:60 x61:61 x62:62 x63:63 x64:64 x65:65 x66:66 x67:67 x68:68 x69:69 x7:7 x70:70 x71:71 x72:72 x73:73 x74:74 x75:75 x76:76 x77:77 x78:78 x79:79 x8:8 x80:80 x81:81 x82:82 x83:83 x84:84 x85:85 x86:86 x87:87 x88:88 x89:89 x9:9 x90:90 x91:91 x92:92 x93:93 x94:94 x95:95 x96:96 x97:97 x98:98 x99:99]

Expected behavior
The Format() method of formatters implementing Formatter is expected to return (nil, ErrTooLarge), when the buffer capacity is not sufficient. This is clearly documented. The result of the above code should emit this error, instead of extending the buffer capacity.

Additional context
The loggers checks cap(buf) against the maximum required estimated size based on each value type, but this looks like a misuse of cap(buf)-len(buf).
Also, loggers checks len(s)*2 + 2 on appending strings, but the escaped string can grow up to 4 times larger than the original byte length (e.x. "\x01"), or 6 times larger in JSON ("\u0001").

Logging to file

As is often the case with log rotation, the logger should
re-open the log file if it receives a signal such as SIGUSR1.

This package should provide a function to help implementing such a feature.

Handle EPIPE

Users of logger tend to ignore errors returned from logging functions, like this:

func doSomething() {
    err := f()
    if err != nil {
        // ignoring possible error returned by log.Error
        log.Error("failed", map[string]interface{}{
            log.FnError: err.Error(),
        })
    }
}

When the underlying output stream went broken and log.Error returns EPIPE,
logs will not be written. Further, with journald, we see that log.Error will eventually
block forever if continuously write logs after EPIPE.

To prevent disasters, we should change the default behavior so that cybozu-go/log
detects EPIPE internally and aborts the program.

Error is not handled in Logger.WriteThrough

In #12, error handler is added to logger, but Logger.WriteThrough did not use it.

Data race

$ go test -race ./...
==================
WARNING: DATA RACE
Write at 0x0000006fdc60 by goroutine 31:
  github.com/cybozu-go/log.TestPlainFormat1()
      /home/ymmt/go/src/github.com/cybozu-go/log/plain_test.go:67 +0x73
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:610 +0xc9

Previous read at 0x0000006fdc60 by goroutine 7:
  github.com/cybozu-go/log.Logfmt.Format()
      /home/ymmt/go/src/github.com/cybozu-go/log/logfmt.go:40 +0x4cc
  github.com/cybozu-go/log.(*Logfmt).Format()
      <autogenerated>:22 +0x11e
  github.com/cybozu-go/log.(*Logger).Log()
      /home/ymmt/go/src/github.com/cybozu-go/log/logger.go:269 +0x31d
  github.com/cybozu-go/log.(*Logger).Writer.func1()
      /home/ymmt/go/src/github.com/cybozu-go/log/logger.go:241 +0x139
  github.com/cybozu-go/log.(*logWriter).Write()
      /home/ymmt/go/src/github.com/cybozu-go/log/logger.go:209 +0xd0
  log.(*Logger).Output()
      /usr/local/go/src/log/log.go:166 +0x397
  log.Print()
      /usr/local/go/src/log/log.go:283 +0xa1
  github.com/cybozu-go/log.TestDefaultLogger()
      /home/ymmt/go/src/github.com/cybozu-go/log/default_test.go:32 +0x3ad
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:610 +0xc9

Goroutine 31 (running) created at:
  testing.(*T).Run()
      /usr/local/go/src/testing/testing.go:646 +0x52f
  testing.RunTests.func1()
      /usr/local/go/src/testing/testing.go:793 +0xb9
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:610 +0xc9
  testing.RunTests()
      /usr/local/go/src/testing/testing.go:799 +0x4ba
  testing.(*M).Run()
      /usr/local/go/src/testing/testing.go:743 +0x12f
  main.main()
      github.com/cybozu-go/log/_test/_testmain.go:94 +0x1b8

Goroutine 7 (finished) created at:
  testing.(*T).Run()
      /usr/local/go/src/testing/testing.go:646 +0x52f
  testing.RunTests.func1()
      /usr/local/go/src/testing/testing.go:793 +0xb9
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:610 +0xc9
  testing.RunTests()
      /usr/local/go/src/testing/testing.go:799 +0x4ba
  testing.(*M).Run()
      /usr/local/go/src/testing/testing.go:743 +0x12f
  main.main()
      github.com/cybozu-go/log/_test/_testmain.go:94 +0x1b8
==================
PASS
Found 1 data race(s)

Deprecate and archive this repository

What

Since its creation, the Go ecosystem has evolved to have a lot of logging options.
Today, we have go-logr/log or uber-go/zap, and Go 1.21 will add log/slog.

I believe this package should come to an end and declare its status explicitly.

How

Add the deprecation notice to README.md and the package doc. cf. https://github.com/golang/go/wiki/Deprecated
Archive this repository.

Checklist

Finish implementation of the issue
Test all functions
Have enough logs to trace activities
Notify developers of necessary actions

JSONFormat outputs non-JSON on some non-printable characters

Describe the bug
JSONFormat is expected to output JSON regardless of the input data, but outputs non-JSON on some special characters; \x00, \a, \v, \U0001FBFF.

Environments

Version: 1.6.1
OS: macOS 12.2.1
Go: go1.17.8 darwin/amd64

To Reproduce
Steps to reproduce the behavior:
Run following code.

package main

import (
	"encoding/json"
	"fmt"
	"os"
	"time"

	"github.com/cybozu-go/log"
)

func main() {
	var err error
	l := log.NewLogger()
	f := log.JSONFormat{Utsname: "test"}
	t := time.Now()

	buf := make([]byte, 0, 256)
	buf, err = f.Format(buf, l, t, log.LvDebug, "test", map[string]interface{}{
		"x": "\x00\a\v\U0001FBFF",
	})
	if err != nil {
		fmt.Fprintf(os.Stderr, "%s\n", err)
		os.Exit(1)
	}
	fmt.Printf("%s\n", buf)

	var data interface{}
	err = json.Unmarshal(buf, &data)
	if err != nil {
		fmt.Fprintf(os.Stderr, "%s\n", err)
		os.Exit(1)
	}
	fmt.Printf("%v\n", data)
}

Get following output.

{"topic":"main","logged_at":"2022-03-15T06:39:16.059075Z","severity":"debug","utsname":"test","message":"test","x":"\x00\a\v\U0001fbff"}

invalid character 'x' in string escape code
exit status 1

Expected behavior
JSONFormat emits valid JSON even if the input data contains non-printable characters, and the output is safely decoded back to the data.

Additional context
Currently this formatter uses strconv.AppendQuote to encode string values. This method is intended to output Go string literal representation, which allows wider range of character representation than JSON. For example, "\x00\a\v\U0001FBFF" is a valid Go string literal, but its quoted value "\"\\x00\\a\\v\\U0001fbff\"" is not a valid JSON.

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.