MaxMind DB Reader for Go
License: ISC License
This is a Go reader for the MaxMind DB format. Although this can be used to read GeoLite2 and GeoIP2 databases, geoip2 provides a higher-level API for doing so.
This is not an official MaxMind API.
go get github.com/oschwald/maxminddb-golang
See GoDoc for documentation and examples.
See GoDoc or
example_test.go for examples.
Contributions welcome! Please fork the repository and open a pull request with your changes.
This is free software, licensed under the ISC License.
![dependabot-preview[bot] avatar](https://avatars.githubusercontent.com/in/2141?v=4 "dependabot-preview[bot]")
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
These are currently missing.
If I want to use golang language to rewrite MaxMind-Write-Perl, What should be noted？thanks
When used as a verb, it should be "look up" rather than "lookup". With Go's naming conventions, this would be LookUp. However, this is awkward. Some other verb may be more appropriate (Get, Query, etc.).
Although this is an API change, we can continue to support Lookup until we do a 2.x.y release.
Hello,
what does the error "invalid node in search tree" mean?
thanx
Godeps/_workspace/src/github.com/oschwald/maxminddb-golang/mmap_unix.go:10: undefined: syscall.Mmap
Godeps/_workspace/src/github.com/oschwald/maxminddb-golang/mmap_unix.go:14: undefined: syscall.Munmap
Related wrong code:
https://github.com/oschwald/maxminddb-golang/blob/master/traverse.go#L26
The GoLang net.IP.To4() function detects if an IPv4 exists inside the IPv6 Byte array.
Since this library uses a zeroed instead of the predefined one:
var v4InV6Prefix = []byte{0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0xff, 0xff}
The net.IPNet.IP.To4() always returns a nil instead of 4 Byte array.
To reproduce the Fix I am currently using I created the next example:
https://play.golang.org/p/y-A-SC8fs9u
To understand better, this is the To4() function source
// To4 converts the IPv4 address ip to a 4-byte representation.
// If ip is not an IPv4 address, To4 returns nil.
func (ip IP) To4() IP {
if len(ip) == IPv4len {
return ip
}
if len(ip) == IPv6len &&
isZeros(ip[0:10]) &&
ip[10] == 0xff &&
ip[11] == 0xff {
return ip[12:16]
}
return nil
}To fix this issue I propose to make a small adjustment to the sources at the relevant function:
https://play.golang.org/p/nj68eVsVbRv
This apparently does not work at the moment.
panic: reflect: call of reflect.Value.Type on zero Value
goroutine 1 [running]:
panic(0x81d3e0, 0xc82012aa80)
/usr/local/go/src/runtime/panic.go:464 +0x3e6
reflect.Value.Type(0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/reflect/value.go:1671 +0x7b
github.com/oschwald/maxminddb-golang.(*decoder).unmarshalMap(0xc8200b48e0, 0x2, 0x432, 0x0, 0x0, 0x0, 0x42e, 0x0, 0x0)
/home/greg/MaxMind/go/src/github.com/oschwald/maxminddb-golang/decoder.go:241 +0x2b1
github.com/oschwald/maxminddb-golang.(*decoder).decodeFromType(0xc8200b48e0, 0x7, 0x2, 0x432, 0x0, 0x0, 0x0, 0x7efcc5c5088d, 0x0, 0x0)
/home/greg/MaxMind/go/src/github.com/oschwald/maxminddb-golang/decoder.go:104 +0x4bf
github.com/oschwald/maxminddb-golang.(*decoder).decode(0xc8200b48e0, 0x431, 0x8664c0, 0xc82012c600, 0x196, 0xc82012c600, 0x0, 0x0)
/home/greg/MaxMind/go/src/github.com/oschwald/maxminddb-golang/decoder.go:43 +0x1c2
github.com/oschwald/maxminddb-golang.(*decoder).decodeStruct(0xc8200b48e0, 0x6, 0x431, 0x891800, 0xc82012c600, 0x199, 0x42e, 0x0, 0x0)
/home/greg/MaxMind/go/src/github.com/oschwald/maxminddb-golang/decoder.go:493 +0x557
github.com/oschwald/maxminddb-golang.(*decoder).unmarshalMap(0xc8200b48e0, 0x6, 0x42f, 0x891800, 0xc82012c600, 0x199, 0x9, 0x0, 0x0)
/home/greg/MaxMind/go/src/github.com/oschwald/maxminddb-golang/decoder.go:243 +0x254
github.com/oschwald/maxminddb-golang.(*decoder).decodeFromType(0xc8200b48e0, 0x7, 0x6, 0x42f, 0x891800, 0xc82012c600, 0x199, 0x0, 0x0, 0x0)
/home/greg/MaxMind/go/src/github.com/oschwald/maxminddb-golang/decoder.go:104 +0x4bf
github.com/oschwald/maxminddb-golang.(*decoder).decode(0xc8200b48e0, 0x42e, 0x866100, 0xc82012c600, 0x16, 0x18, 0x0, 0x0)
/home/greg/MaxMind/go/src/github.com/oschwald/maxminddb-golang/decoder.go:43 +0x1c2
github.com/oschwald/maxminddb-golang.(*Reader).Decode(0xc8200b48c0, 0x42e, 0x866100, 0xc82012c600, 0x0, 0x0)
/home/greg/MaxMind/go/src/github.com/oschwald/maxminddb-golang/reader.go:151 +0x135
main.createCachedDb(0xc82007d200, 0x0, 0x0)
Golang 1.12.5, on Fedora Rawhide x86_64, maxminddb 1.3.0:
Testing in: /builddir/build/BUILD/maxminddb-golang-1.3.0/_build/src
PATH: /builddir/build/BUILD/maxminddb-golang-1.3.0/_build/bin:/builddir/.local/bin:/builddir/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/sbin
GOPATH: /builddir/build/BUILD/maxminddb-golang-1.3.0/_build:/usr/share/gocode
GO111MODULE: off
command: go test -buildmode pie -compiler gc -ldflags "-X github.com/oschwald/maxminddb-golang/version=1.3.0 -extldflags '-Wl,-z,relro -Wl,--as-needed -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld '"
testing: github.com/oschwald/maxminddb-golang
github.com/oschwald/maxminddb-golang
unexpected fault address 0x7febf45bb623
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0x7febf45bb623 pc=0x563fe18a7806]
goroutine 52 [running]:
runtime.throw(0x563fe1aeb66f, 0x5)
/usr/lib/golang/src/runtime/panic.go:617 +0x74 fp=0xc0000a9890 sp=0xc0000a9860 pc=0x563fe1877b94
runtime.sigpanic()
/usr/lib/golang/src/runtime/signal_unix.go:397 +0x405 fp=0xc0000a98c0 sp=0xc0000a9890 pc=0x563fe188cdb5
runtime.memmove(0xc000019890, 0x7febf45bb623, 0xc)
/usr/lib/golang/src/runtime/memmove_amd64.s:173 +0x136 fp=0xc0000a98c8 sp=0xc0000a98c0 pc=0x563fe18a7806
runtime.slicebytetostring(0x0, 0x7febf45bb623, 0xc, 0x7381, 0xc000019890, 0xc)
/usr/lib/golang/src/runtime/string.go:106 +0x82 fp=0xc0000a98f8 sp=0xc0000a98c8 pc=0x563fe1891a12
github.com/oschwald/maxminddb-golang.(*decoder).decodeMap(0xc0000a9c68, 0x2, 0x5901, 0x563fe1c02e00, 0xc0004653b0, 0x15, 0x1, 0x58e4, 0x0, 0x0)
/builddir/build/BUILD/maxminddb-golang-1.3.0/_build/src/github.com/oschwald/maxminddb-golang/decoder.go:495 +0x28f fp=0xc0000a99b0 sp=0xc0000a98f8 pc=0x563fe1ad87cf
github.com/oschwald/maxminddb-golang.(*decoder).unmarshalMap(0xc0000a9c68, 0x2, 0x58e5, 0x563fe1c01840, 0xc0004693f0, 0x194, 0x1, 0x2, 0x58e5, 0x0)
/builddir/build/BUILD/maxminddb-golang-1.3.0/_build/src/github.com/oschwald/maxminddb-golang/decoder.go:324 +0x3d5 fp=0xc0000a9ab8 sp=0xc0000a99b0 pc=0x563fe1ad6e15
github.com/oschwald/maxminddb-golang.(*decoder).decodeFromType(0xc0000a9c68, 0x7, 0x2, 0x58e5, 0x563fe1be3fe0, 0xc0004693f0, 0x16, 0x1, 0xc0003c7320, 0xc0000a9bb0, ...)
/builddir/build/BUILD/maxminddb-golang-1.3.0/_build/src/github.com/oschwald/maxminddb-golang/decoder.go:123 +0x580 fp=0xc0000a9b38 sp=0xc0000a9ab8 pc=0x563fe1ad4d10
github.com/oschwald/maxminddb-golang.(*decoder).decode(0xc0000a9c68, 0x58e4, 0x563fe1be3fe0, 0xc0004693f0, 0x16, 0x0, 0x58e4, 0x0, 0x0)
/builddir/build/BUILD/maxminddb-golang-1.3.0/_build/src/github.com/oschwald/maxminddb-golang/decoder.go:54 +0xee fp=0xc0000a9bc0 sp=0xc0000a9b38 pc=0x563fe1ad415e
github.com/oschwald/maxminddb-golang.(*verifier).verifyDataSection(0xc0000a9d80, 0xc000382f90, 0x0, 0x0)
/builddir/build/BUILD/maxminddb-golang-1.3.0/_build/src/github.com/oschwald/maxminddb-golang/verifier.go:137 +0x18f fp=0xc0000a9d30 sp=0xc0000a9bc0 pc=0x563fe1adce9f
github.com/oschwald/maxminddb-golang.(*verifier).verifyDatabase(0xc0000a9d80, 0x0, 0x0)
/builddir/build/BUILD/maxminddb-golang-1.3.0/_build/src/github.com/oschwald/maxminddb-golang/verifier.go:94 +0x93 fp=0xc0000a9d68 sp=0xc0000a9d30 pc=0x563fe1adc913
github.com/oschwald/maxminddb-golang.(*Reader).Verify(0xc0002daa00, 0xc0002c0400, 0x0)
/builddir/build/BUILD/maxminddb-golang-1.3.0/_build/src/github.com/oschwald/maxminddb-golang/verifier.go:18 +0x71 fp=0xc0000a9d98 sp=0xc0000a9d68 pc=0x563fe1adbf21
github.com/oschwald/maxminddb-golang.TestVerifyOnGoodDatabases(0xc0002c0400)
/builddir/build/BUILD/maxminddb-golang-1.3.0/_build/src/github.com/oschwald/maxminddb-golang/verifier_test.go:35 +0x111 fp=0xc0000a9fa8 sp=0xc0000a9d98 pc=0x563fe1ae8521
testing.tRunner(0xc0002c0400, 0x563fe1c61588)
/usr/lib/golang/src/testing/testing.go:865 +0xc2 fp=0xc0000a9fd0 sp=0xc0000a9fa8 pc=0x563fe191ff72
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc0000a9fd8 sp=0xc0000a9fd0 pc=0x563fe18a65b1
created by testing.(*T).Run
/usr/lib/golang/src/testing/testing.go:916 +0x35c
goroutine 1 [chan receive]:
testing.(*T).Run(0xc0002c0400, 0x563fe1af1c31, 0x19, 0x563fe1c61588, 0x563fe18c1301)
/usr/lib/golang/src/testing/testing.go:917 +0x383
testing.runTests.func1(0xc00015c000)
/usr/lib/golang/src/testing/testing.go:1157 +0x7a
testing.tRunner(0xc00015c000, 0xc0000a7e30)
/usr/lib/golang/src/testing/testing.go:865 +0xc2
testing.runTests(0xc000096380, 0x563fe1e3b7c0, 0x23, 0x23, 0x0)
/usr/lib/golang/src/testing/testing.go:1155 +0x2ab
testing.(*M).Run(0xc00015a000, 0x0)
/usr/lib/golang/src/testing/testing.go:1072 +0x164
main.main()
_testmain.go:120 +0x140
exit status 2
FAIL github.com/oschwald/maxminddb-golang 0.045s
Is the Reader interface thread-safe?
A note should be added to the API documentation declaring whether it is (or not), or describing any special rules to guarantee correct operation.
Thanks for making this software available!!!
Right now, the reader assumes every interface is an empty interface, which is wrong and can lead to panics when using other interfaces in the type being decoded to. We should do something like indirect() in encoding/json.
when to release the maxminddb-writer-golang? thanks!
Go 1.18 includes the new net/netip package with value types related to IP addresses and networks.
Perhaps, new methods should be added, for example Reader.LookupAddr(ip netip.Addr, result any) and Reader.Prefixes with a corresponding iterator, that would encourage the use of the new IP types. I think, it could also reduce some of the boilerplate such as:
Lines 247 to 250 in ccd731c
and a lot of len(ip) == net.IPv4len and ip.To4() == nil.
Hi, I had a hard trying to find out what kind of data you can get from a IP look up as there was no documentation on it. Couldn't find any info on Maxmind's site either after a quick peek. I ended up using an empty interface{} for Lookup() and print that, to see the available data.
Could you please add some documentation for that? Or I could send you a PR with a file called doc.go or something and make a list of the data for now?
For reference:
map[string]interface {}{"continent":map[string]interface {}{"code":"NA", "geoname_id":0x5f722d, "names":map[string]interface {}{"fr":"Amérique du Nord", "ja":"北アメリカ", "pt-BR":"América do Norte", "ru":"Северная Америка", "zh-CN":"北美洲", "de":"Nordamerika", "en":"North America", "es":"Norteamérica"}}, "country":map[string]interface {}{"iso_code":"US", "names":map[string]interface {}{"pt-BR":"Estados Unidos", "ru":"США", "zh-CN":"美国", "de":"USA", "en":"United States", "es":"Estados Unidos", "fr":"États-Unis", "ja":"アメリカ合衆国"}, "geoname_id":0x5f65e1}, "registered_country":map[string]interface {}{"names":map[string]interface {}{"fr":"États-Unis", "ja":"アメリカ合衆国", "pt-BR":"Estados Unidos", "ru":"США", "zh-CN":"美国", "de":"USA", "en":"United States", "es":"Estados Unidos"}, "geoname_id":0x5f65e1, "iso_code":"US"}}
Provide a method that reads from a []byte instead of opening a file and mmap-ing it. This is useful for downloading a database and keeping it entirely in memory.
I've had no other issues with go 1.12, but the test suite for this package now fails on three out of six of fedora's supported architectures (ppc64le, aarch64, s390x). x86_64, i686, and armv7hl continue to work fine.
This is the crash output from the ppc64le run (from go test) for the 1.3.0 release:
unexpected fault address 0x7fffb449e94b
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0x7fffb449e94b pc=0x10c3f7590]
goroutine 52 [running]:
runtime.throw(0x10c40d701, 0x5)
/usr/lib/golang/src/runtime/panic.go:617 +0x68 fp=0xc0000af6e0 sp=0xc0000af6a0 pc=0x10c1b80b8
runtime.sigpanic()
/usr/lib/golang/src/runtime/signal_unix.go:397 +0x464 fp=0xc0000af720 sp=0xc0000af6e0 pc=0x10c1d0954
github.com/oschwald/maxminddb-golang.(*decoder).decodeCtrlData(0xc0000afc10, 0x3349, 0x10c550400, 0xc00044e000, 0xc000446000, 0xc000466000, 0x19)
/builddir/build/BUILD/maxminddb-golang-1.3.0/_build/src/github.com/oschwald/maxminddb-golang/decoder.go:62 +0x50 fp=0xc0000af7b8 sp=0xc0000af740 pc=0x10c3f7590
github.com/oschwald/maxminddb-golang.(*decoder).decodeKey(0xc0000afc10, 0x3349, 0x15, 0x10c53f840, 0xc000446000, 0x98, 0x10c54ee40, 0xc000466000)
/builddir/build/BUILD/maxminddb-golang-1.3.0/_build/src/github.com/oschwald/maxminddb-golang/decoder.go:673 +0x4c fp=0xc0000af840 sp=0xc0000af7b8 pc=0x10c3fc0fc
github.com/oschwald/maxminddb-golang.(*decoder).decodeMap(0xc0000afc10, 0x2, 0x3349, 0x10c550400, 0xc00044e000, 0x15, 0x1, 0x3325, 0x10c3f83cc, 0x0)
/builddir/build/BUILD/maxminddb-golang-1.3.0/_build/src/github.com/oschwald/maxminddb-golang/decoder.go:484 +0x120 fp=0xc0000af908 sp=0xc0000af840 pc=0x10c3fb120
github.com/oschwald/maxminddb-golang.(*decoder).unmarshalMap(0xc0000afc10, 0x2, 0x3326, 0x10c54ee40, 0xc0004479a0, 0x194, 0x1, 0x2, 0x3326, 0x0)
/builddir/build/BUILD/maxminddb-golang-1.3.0/_build/src/github.com/oschwald/maxminddb-golang/decoder.go:324 +0x320 fp=0xc0000afa20 sp=0xc0000af908 pc=0x10c3f9b30
github.com/oschwald/maxminddb-golang.(*decoder).decodeFromType(0xc0000afc10, 0x7, 0x2, 0x3326, 0x10c531700, 0xc0004479a0, 0x16, 0x1, 0x1, 0xd34263aa77ce097, ...)
/builddir/build/BUILD/maxminddb-golang-1.3.0/_build/src/github.com/oschwald/maxminddb-golang/decoder.go:123 +0x4a0 fp=0xc0000afab0 sp=0xc0000afa20 pc=0x10c3f7d80
github.com/oschwald/maxminddb-golang.(*decoder).decode(0xc0000afc10, 0x3325, 0x10c531700, 0xc0004479a0, 0x16, 0x0, 0x3325, 0x0, 0x0)
/builddir/build/BUILD/maxminddb-golang-1.3.0/_build/src/github.com/oschwald/maxminddb-golang/decoder.go:54 +0xd4 fp=0xc0000afb48 sp=0xc0000afab0 pc=0x10c3f7344
github.com/oschwald/maxminddb-golang.(*verifier).verifyDataSection(0xc0000afd48, 0xc000388f90, 0x0, 0x0)
/builddir/build/BUILD/maxminddb-golang-1.3.0/_build/src/github.com/oschwald/maxminddb-golang/verifier.go:137 +0x178 fp=0xc0000afcc8 sp=0xc0000afb48 pc=0x10c3ff4c8
github.com/oschwald/maxminddb-golang.(*verifier).verifyDatabase(0xc0000afd48, 0x0, 0x0)
/builddir/build/BUILD/maxminddb-golang-1.3.0/_build/src/github.com/oschwald/maxminddb-golang/verifier.go:94 +0xa0 fp=0xc0000afd10 sp=0xc0000afcc8 pc=0x10c3fef70
github.com/oschwald/maxminddb-golang.(*Reader).Verify(0xc0002d8aa0, 0xc0002c8200, 0x0)
/builddir/build/BUILD/maxminddb-golang-1.3.0/_build/src/github.com/oschwald/maxminddb-golang/verifier.go:18 +0x80 fp=0xc0000afd50 sp=0xc0000afd10 pc=0x10c3fe670
github.com/oschwald/maxminddb-golang.TestVerifyOnGoodDatabases(0xc0002c8200)
/builddir/build/BUILD/maxminddb-golang-1.3.0/_build/src/github.com/oschwald/maxminddb-golang/verifier_test.go:35 +0x114 fp=0xc0000aff70 sp=0xc0000afd50 pc=0x10c40a004
testing.tRunner(0xc0002c8200, 0x10c5ae540)
/usr/lib/golang/src/testing/testing.go:862 +0xdc fp=0xc0000affb0 sp=0xc0000aff70 pc=0x10c26506c
runtime.goexit()
/usr/lib/golang/src/runtime/asm_ppc64x.s:856 +0x4 fp=0xc0000affb0 sp=0xc0000affb0 pc=0x10c1eab24
created by testing.(*T).Run
/usr/lib/golang/src/testing/testing.go:913 +0x304
goroutine 1 [chan receive]:
testing.(*T).Run(0xc0002c8200, 0x10c413d0f, 0x19, 0x10c5ae540, 0x10c777f01)
/usr/lib/golang/src/testing/testing.go:914 +0x320
testing.runTests.func1(0xc000162000)
/usr/lib/golang/src/testing/testing.go:1154 +0x8c
testing.tRunner(0xc000162000, 0xc0000addd8)
/usr/lib/golang/src/testing/testing.go:862 +0xdc
testing.runTests(0xc00009c3a0, 0x10c777740, 0x23, 0x23, 0x0)
/usr/lib/golang/src/testing/testing.go:1152 +0x2a0
testing.(*M).Run(0xc000160000, 0x0)
/usr/lib/golang/src/testing/testing.go:1069 +0x174
main.main()
_testmain.go:120 +0x150
exit status 2
FAIL github.com/oschwald/maxminddb-golang 0.049s
If I can provide any other useful information, please let me know.
Is that possible? If yes, could you give an example?
Hello!
With a relatively recent maxmind country-level DB (let me know if you want my exact copy), we have found at least one IP address that decodes to a different country code based on which de-serialization type is used. This is a bit concerning (we expect it has something to do with registered vs represented countries but we haven't nailed it down).
Here's the difference:
var data map[string]map[string]any
reader.Lookup("95.140.157.134", &data)
fmt.Println(data["country"]["iso_code"])
This says Russia.
var data struct {
Country struct {
IsoCode string `maxminddb:"iso_code"`
} `maxminddb:"country"`
}
reader.Lookup("95.140.157.134", &data)
fmt.Println(data.Country.IsoCode)
This says Poland.
What's going on here?
Currently, neither of these are filled in.
Is it possible to read multiple mmdb files at the same time? Is this something that will supported in the future?
panic: runtime error: index out of range
goroutine 411988086 [running]:
panic(0xa92dc0, 0xc42000a0e0)
/usr/local/go/src/runtime/panic.go:500 +0x1a1
github.com/oschwald/maxminddb-golang.(*decoder).decodeCtrlData(0xc4202e6020, 0x79d7b07, 0x0, 0x7f267c0650dc, 0x4)
/Users/yangrong/go/src/github.com/oschwald/maxminddb-golang/decoder.go:48 +0xbe
github.com/oschwald/maxminddb-golang.(*decoder).decode(0xc4202e6020, 0x79d7b07, 0xaf1660, 0xc42a377b80, 0x199, 0x199, 0x199, 0xaf1660)
/Users/yangrong/go/src/github.com/oschwald/maxminddb-golang/decoder.go:37 +0x39
github.com/oschwald/maxminddb-golang.(*decoder).unmarshalPointer(0xc4202e6020, 0x1b, 0x304a, 0xaf1660, 0xc42a377b80, 0x199, 0x199, 0x738732, 0xc4202e6020)
/Users/yangrong/go/src/github.com/oschwald/maxminddb-golang/decoder.go:277 +0x86
github.com/oschwald/maxminddb-golang.(*decoder).decodeFromType(0xc4202e6020, 0x1, 0x1b, 0x304a, 0xa41de0, 0xc42a377b80, 0x16, 0x7c5c4, 0x0, 0xa41de0)
/Users/yangrong/go/src/github.com/oschwald/maxminddb-golang/decoder.go:103 +0x2a8
github.com/oschwald/maxminddb-golang.(*decoder).decode(0xc4202e6020, 0x3049, 0xa41de0, 0xc42a377b80, 0x16, 0xc4202e6000, 0xc420d88fec, 0x4)
/Users/yangrong/go/src/github.com/oschwald/maxminddb-golang/decoder.go:43 +0xa0
github.com/oschwald/maxminddb-golang.(*Reader).Decode(0xc4202e6000, 0x3049, 0xa41de0, 0xc42a377b80, 0x0, 0xc4204737b0)
/Users/yangrong/go/src/github.com/oschwald/maxminddb-golang/reader.go:152 +0x13a
github.com/oschwald/maxminddb-golang.(*Reader).retrieveData(0xc4202e6000, 0x7c5c4, 0xa41de0, 0xc42a377b80, 0x7c5c4, 0x0)
/Users/yangrong/go/src/github.com/oschwald/maxminddb-golang/reader.go:236 +0x74
github.com/oschwald/maxminddb-golang.(*Reader).Lookup(0xc4202e6000, 0xc420d88fe0, 0x10, 0x10, 0xa41de0, 0xc42a377b80, 0xc, 0xc420d88fe0)
/Users/yangrong/go/src/github.com/oschwald/maxminddb-golang/reader.go:114 +0xa6
github.com/oschwald/geoip2-golang.(*Reader).Country(0xc42015a5e0, 0xc420d88fe0, 0x10, 0x10, 0x10, 0xc420d88fd0, 0xd)
/Users/yangrong/go/src/github.com/oschwald/geoip2-golang/reader.go:235 +0x1b7
$ mmdblookup --version
mmdblookup version 1.4.2
$ mmdblookup --file dbip-asn.mmdb --ip 1.1.1.1
{
"autonomous_system_number":
13335 <uint32>
"autonomous_system_organization":
"Cloudflare, Inc." <utf8_string>
}
mmdb file:
https://db-ip.com/db/download/ip-to-asn-lite
Failed on my code:
package main
import (
"fmt"
"net"
mmdb "github.com/oschwald/geoip2-golang"
)
func main() {
mmdbASN, err := mmdb.Open("dbip-asn.mmdb")
if err != nil {
panic(err) // HERE the panic
}
ipParsed := net.ParseIP("1.1.1.1")
asnData, asnErr := mmdbASN.ASN(ipParsed)
if asnErr != nil {
fmt.Println(asnData)
}
return
}$ panic: geoip2: reader does not support the "DBIP-ASN-Lite (compat=GeoLite2-ASN)" database type
goroutine 1 [running]:
main.main()
main.go:14 +0x11b
Could we add a method for the Networks to return the current IPNet, without decoding the result interface.
By now I would like to iterate over all the MMDB database and skip the duplicate address cased by redirection such as 2002::/16. As we know the decoding is a time cost process. Certainly I can pass a nil member structure into the Network method, but I supposed this is not a graceful way. It would be better if the Networks could provide with a method to return the current CIDR directly.
Hi there. I'm wondering if it's possible (and if so is there any examples) to look through the DB and input all the IPs and ISO2Codes to a map or similar so that it's faster to read? Thanks.
hi, man
on windows 8, go get github.com/oschwald/maxminddb-golang failed
get the error:
maxminddb-golang\reader.go:58: undefined: syscall.Mmap
reader.go:58: undefined: syscall.PROT_READ
reader.go:58: undefined: syscall.MAP_SHARED
...
is that mean windows is not support yet?
The error from os.Close is stored to err, which is the one created here https://github.com/oschwald/maxminddb-golang/blob/master/reader_other.go#L16 .
That means that the assigment here
maxminddb-golang/reader_other.go
Line 22 in 302864b
Either this https://github.com/oschwald/maxminddb-golang/blob/master/reader_other.go#L15 should be
func Open(file string) (r *Reader, err error) {
but since that would mask actual error with one from os.Close, I think better option would be to just replace
defer func() {
if rerr := mapFile.Close(); rerr != nil {
err = rerr
}
}()
with
defer func() {
mapFile.Close()
}()
As written right now the code probably does not do what was intended.
Hello,
A newbie question:
https://github.com/oschwald/maxminddb-golang/blob/master/reader_other.go
In this, I see an os.Open call fd, which is then used for mmap. However, I don’t see an os.Close call. Is that not needed.
Thanks
Sundar.
On a database where the node count in the metadata is much larger than the actual number of nodes:
panic: runtime error: slice bounds out of range
goroutine 1 [running]:
panic(0x5419e0, 0xc820010170)
/usr/local/go/src/runtime/panic.go:464 +0x3e6
github.com/oschwald/maxminddb-golang.FromBytes(0x7fe4b60da000, 0x1de7ce, 0x1de7ce, 0x1de7ce, 0x0, 0x0)
/home/greg/MaxMind/go/src/github.com/oschwald/maxminddb-golang/reader.go:63 +0x419
github.com/oschwald/maxminddb-golang.Open(0x7ffe60ffce54, 0x23, 0x0, 0x0, 0x0)
/home/greg/MaxMind/go/src/github.com/oschwald/maxminddb-golang/reader_other.go:34 +0x218
main.main()
/home/greg/MaxMind/go/src/github.com/maxmind/mmdbverify/main.go:22 +0x10c
Seeing some odd results when trying to read all IPv4 subnets from the GeoIP2-Anonymous-IP.mmdb file:
db, err := maxminddb.Open("/usr/share/GeoIP2/GeoIP2-Anonymous-IP.mmdb")
if err != nil {
log.Fatal(err)
}
defer db.Close()
record := geoip2.AnonymousIP{}
networks := db.Networks()
for networks.Next() {
subnet, err := networks.Network(&record)
if err != nil {
log.Fatal(err)
}
if subnet.IP.To4() != nil {
fmt.Printf("%s: Anonymous:%v Tor:%v\n", subnet, record.IsAnonymous, record.IsTorExitNode)
}
}
go run...| grep 1.169.30.6/32
1.169.30.6/32: Anonymous:true Tor:true
I also note that I get 1311159 results when there's 1354761 total in the file.
I get different (expected?) results with geoip2-golang and mmdblookup
db, err := geoip2.Open("/usr/share/GeoIP2/GeoIP2-Anonymous-IP.mmdb")
if err != nil {
log.Fatal(err)
}
defer db.Close()
ip := net.ParseIP(os.Args[1])
record, err := db.AnonymousIP(ip)
if err != nil {
log.Fatal(err)
}
fmt.Printf("record is %#v\n", record)
>go run anontest.go 1.169.30.6
record is &geoip2.AnonymousIP{IsAnonymous:false, IsAnonymousVPN:false, IsHostingProvider:false, IsPublicProxy:false, IsTorExitNode:false}
>mmdblookup -f /usr/share/GeoIP2/GeoIP2-Anonymous-IP.mmdb --ip 1.169.30.6
{
}
Firstly I want to say thank you for the library.
Secondly, I have noticed that it is using 800-1000 megs of memory. Does that sound correct?
In the legacy API I could use GeoIP_last_netmask to get the netmask that the data covered.
For my application (EDNS-CLIENT-SUBNET support in a DNS server) this is very helpful.
When the ip address is not found in the database the library doesn't return an error.
I was able to reproduce this problem using the latest GeoLite2-City.mmdb and the ip address 102.53.124.191. From what I can see at https://www.maxmind.com/en/geoip2-city-database-accuracy MaxMind doesn't promise to resolve all ip addresses so it seems useful to return an error when the ip address can not be found.
Here is code that shows the issue:
package main
import (
"fmt"
"log"
"net"
"os"
"github.com/oschwald/maxminddb-golang"
)
type City struct {
City struct {
GeoNameID uint `maxminddb:"geoname_id"`
Names map[string]string `maxminddb:"names"`
} `maxminddb:"city"`
Country struct {
GeoNameID uint `maxminddb:"geoname_id"`
IsoCode string `maxminddb:"iso_code"`
Names map[string]string `maxminddb:"names"`
} `maxminddb:"country"`
Location struct {
Latitude float64 `maxminddb:"latitude"`
Longitude float64 `maxminddb:"longitude"`
} `maxminddb:"location"`
}
func main() {
geodbfile := "GeoLite2-City.mmdb"
if os.Getenv("GEODB") != "" {
geodbfile = os.Getenv("GEODB")
}
db, err := maxminddb.Open(geodbfile)
if err != nil {
log.Fatal(err)
}
defer db.Close()
ip := net.ParseIP("102.53.124.191")
var city City
err = db.Lookup(ip, &city)
if err == nil {
fmt.Println("Didn't get an error but expected one")
}
fmt.Println(city, err)
}
When viewed on godoc.org, it doesn't show the struct used for looking up data, making the example rather pointless.
I cannot deploy to Google App Engine because this package uses unsafe.
Is there some way to get rid of the unsafe package?
Is there a reason to use golang.org/x/sys/unix.Mmap instead of syscall.Mmap? I swapped the Mmap and Munmap calls to syscall and they seemed to work fine.
This removes the golang.org/x/sys/unix dependency:
--- a/github.com/oschwald/maxminddb-golang/mmap_unix.go
+++ b/github.com/oschwald/maxminddb-golang/mmap_unix.go
@@ -4,14 +4,12 @@ package maxminddb
import (
"syscall"
-
- "golang.org/x/sys/unix"
)
func mmap(fd int, length int) (data []byte, err error) {
- return unix.Mmap(fd, 0, length, syscall.PROT_READ, syscall.MAP_SHARED)
+ return syscall.Mmap(fd, 0, length, syscall.PROT_READ, syscall.MAP_SHARED)
}
func munmap(b []byte) (err error) {
- return unix.Munmap(b)
+ return syscall.Munmap(b)
}I can create a PR if you'd like.
Hey @oschwald, thanks for the excellent library. I'm hoping to get your thoughts on how we could re-organize the code so that the golang.org/x/sys dependency is not pulled if when it won't be used.
I found the performance between the memory-mapped approach and the FromBytes approach to reading the country database to be negligible.
For simplicity, I'd like to avoid the memory-mapping (and it's associated dependencies in golang.org/x/sys) . The extra dependency on functionality which won't be used is pretty heavy—golang.org/x/sys/unix adds ~160K lines of code that I must vendor in my application.
Hi @oschwald
When I am in a situation with high concurrency and slightly insufficient CPU resources, occasionally an unexpected fault address issue occurs, leading to a panic.
This kind of memory-level error makes it impossible for recover to catch it.
Below is the stack information I found, which ultimately points to the readLeft method. Can you help take a look or is this a known issue?
Thank you very much.
Hey!
I am looking at improving performance on an app using heavily geoip lookup and I am wondering if there was something between using reflect and a custom deserializer (as introduced in #74). reflect can be slow and it should be possible to use something like DecodeFieldAsString(offset uintptr, fields ...string) string.
I didn't try to use the deserializer interface as it is not really documented. An alternative would be an example on how to use it. The associated tests are strictly technical, so it is hard to understand how difficult this is.
It does install v1.6.0, but the code doesnt match the current code in github.com
For example traverse.go contains no method called NetworkWithin()
$ cat ~/go/pkg/mod/github.com/oschwald/maxminddb-golang\@v1.6.0/traverse.go | grep 'Network'
// Networks represents a set of subnets that we are iterating over.
type Networks struct {
// Networks returns an iterator that can be used to traverse all networks in
func (r *Reader) Networks() *Networks {
return &Networks{
// Next prepares the next network for reading with the Network method. It
func (n *Networks) Next() bool {
// Network returns the current network or an error if there is a problem
func (n *Networks) Network(result interface{}) (*net.IPNet, error) {
func (n *Networks) Err() error {
go get -v -u github.com/oschwald/maxminddb-golang didnt help
Had to go get -v -u github.com/oschwald/maxminddb-golang@master to force it to ignore versioning:
After which I have:
$ ls -alF ~/go/pkg/mod/github.com/oschwald/
total 36
drwxr-xr-x 1 rhaksi 197608 0 Jun 12 15:51 ./
drwxr-xr-x 1 rhaksi 197608 0 Jun 12 15:37 ../
drwxr-xr-x 1 rhaksi 197608 0 Jun 12 15:37 '[email protected]'/
drwxr-xr-x 1 rhaksi 197608 0 Jun 12 15:37 '[email protected]'/
drwxr-xr-x 1 rhaksi 197608 0 Jun 12 15:51 '[email protected]'/
It is not supported there. See #10.
Hello,
I m using this library in a HTTP server and i use the golang race detector (http://blog.golang.org/race-detector).
When i'm using the same instance of the Reader , the race detector detect a race condition on startNode() with the ipv4start variable.
thanx
Hi, I was working with something using this library. When I came to unit test, we had a problem where mmdb files cannot be found. Is there a way to mock mmdb.Open() function?
I think this is a new issue with the indirect() change.
2016/07/11 09:14:40 panic: reflect: reflect.Value.Set using unaddressable value
Stack trace:
goroutine 34 [running]:
runtime/debug.Stack(0x0, 0x0, 0x0)
/usr/local/go/src/runtime/debug/stack.go:24 +0x80
github.com/valyala/fasthttp.(*workerPool).workerFunc.func1(0xc820096280, 0xc820151ed0)
/home/greg/MaxMind/go/src/github.com/valyala/fasthttp/workerpool.go:207 +0x53
panic(0x7a5a80, 0xc8201239f0)
/usr/local/go/src/runtime/panic.go:426 +0x4e9
reflect.flag.mustBeAssignable(0x16)
/usr/local/go/src/reflect/value.go:229 +0x221
reflect.Value.Set(0x8905e0, 0x0, 0x16, 0x8905e0, 0xc820132300, 0x16)
/usr/local/go/src/reflect/value.go:1328 +0x25
github.com/oschwald/maxminddb-golang.(*decoder).indirect(0xc820012c00, 0x8905e0, 0x0, 0x16, 0x0, 0x0, 0x0)
/home/greg/MaxMind/go/src/github.com/oschwald/maxminddb-golang/decoder.go:147 +0x176
github.com/oschwald/maxminddb-golang.(*decoder).decodeFromType(0xc820012c00, 0x7, 0x1, 0xe577, 0x785ba0, 0xc8201239e0, 0x16, 0x75f2d9c, 0x0, 0x0)
/home/greg/MaxMind/go/src/github.com/oschwald/maxminddb-golang/decoder.go:87 +0x69
github.com/oschwald/maxminddb-golang.(*decoder).decode(0xc820012c00, 0xe576, 0x785ba0, 0xc8201239e0, 0x16, 0x10, 0x0, 0x0)
/home/greg/MaxMind/go/src/github.com/oschwald/maxminddb-golang/decoder.go:43 +0x1c2
github.com/oschwald/maxminddb-golang.(*Reader).Decode(0xc820012be0, 0xe576, 0x785ba0, 0xc8201239e0, 0x0, 0x0)
/home/greg/MaxMind/go/src/github.com/oschwald/maxminddb-golang/reader.go:152 +0x135
github.com/oschwald/maxminddb-golang.(*Reader).retrieveData(0xc820012be0, 0x1498ca, 0x785ba0, 0xc8201239e0, 0x0, 0x0)
/home/greg/MaxMind/go/src/github.com/oschwald/maxminddb-golang/reader.go:236 +0x90
github.com/oschwald/maxminddb-golang.(*Reader).Lookup(0xc820012be0, 0xc820123290, 0x10, 0x10, 0x785ba0, 0xc8201239e0, 0x0, 0x0)
/home/greg/MaxMind/go/src/github.com/oschwald/maxminddb-golang/reader.go:114 +0xa1
github.maxmind.com/maxmind/mm_website.git/go/geoip2-daemon/model.(*DbReader).lookupCity(0xc8200175f0, 0xc820123290, 0x10, 0x10, 0x7fe6fbc68340, 0xc820128230, 0x403d38, 0x0, 0x0)
/home/greg/MaxMind/go/src/github.maxmind.com/maxmind/mm_website.git/go/geoip2-daemon/model/db.go:148 +0x19f
github.maxmind.com/maxmind/mm_website.git/go/geoip2-daemon/model.(*DbReader).CityV20(0xc8200175f0, 0xc820123290, 0x10, 0x10, 0xffffffffffffff01, 0xc820123290, 0x0, 0x0)
It seems I got invalid result for the following IP: 185.100.86.147.
Tested with both country and city geolite2 dbs, the expected country is FI however i got RO.
Hard to tell if it's a bug in maxmind's geolite2 db though it works fine with the latest legacy db.
# geoiplookup -f /usr/share/GeoIP/GeoIP.dat 185.100.86.147
GeoIP Country Edition: FI, Finland
Could you please investigate this issue?
I would like to switch all the code in this library to the ISC license. The reason for this is to make this library compatible with code that is licensed under the GPL v2.
@oxtoacart, @alicebob, and @erikdubbelboer, as you all contributed non-trivial patches to this library, would you confirm in this issue that this is acceptable to you? Thanks!
I've recently observed this fatal error seemingly related to uintFromBytes (vendored at v1.2.0), and was wondering if there was anything that stood out from the stack trace.
$ go version
go version go1.8.3 linux/amd64unexpected fault address 0x7fba5014a240
fatal error: fault
[signal SIGBUS: bus error code=0x2 addr=0x7fba5014a240 pc=0x82d5bd]
goroutine 946477698 [running]:
runtime.throw(0x14a1d41, 0x5)
/usr/local/go/src/runtime/panic.go:596 +0x95 fp=0xc462aa4618 sp=0xc462aa45f8
runtime.sigpanic()
/usr/local/go/src/runtime/signal_unix.go:287 +0xf4 fp=0xc462aa4668 sp=0xc462aa4618
<project>/vendor/github.com/oschwald/maxminddb-golang.uintFromBytes(0x0, 0x7fba5014a240, 0x3, 0x36e136, 0x10)
/go/src/<project>/vendor/github.com/oschwald/maxminddb-golang/decoder.go:663 +0x1d fp=0xc462aa4670 sp=0xc462aa4668
<project>/vendor/github.com/oschwald/maxminddb-golang.(*Reader).readNode(0xc4219e1400, 0x60, 0x0, 0x402672, 0x14a8694, 0xa)
/go/src/<project>/vendor/github.com/oschwald/maxminddb-golang/reader.go:228 +0xa2 fp=0xc462aa46e0 sp=0xc462aa4670
<project>/vendor/github.com/oschwald/maxminddb-golang.(*Reader).findAddressInTree(0xc4219e1400, 0xc435169ccc, 0x4, 0x4, 0x4, 0x4, 0x10)
/go/src/<project>/vendor/github.com/oschwald/maxminddb-golang/reader.go:187 +0xb9 fp=0xc462aa4750 sp=0xc462aa46e0
<project>/vendor/github.com/oschwald/maxminddb-golang.(*Reader).lookupPointer(0xc4219e1400, 0xc435169cc0, 0x10, 0x10, 0x4, 0x4, 0xc435169d00)
/go/src/<project>/vendor/github.com/oschwald/maxminddb-golang/reader.go:169 +0x9b fp=0xc462aa47c0 sp=0xc462aa4750
<project>/vendor/github.com/oschwald/maxminddb-golang.(*Reader).Lookup(0xc4219e1400, 0xc435169cc0, 0x10, 0x10, 0x11c86a0, 0xc435169d10, 0x144db80, 0xc430313ee0)
/go/src/<project>/vendor/github.com/oschwald/maxminddb-golang/reader.go:110 +0x4d fp=0xc462aa4808 sp=0xc462aa47c0
It takes first item:
func main() {
db, err := maxminddb.Open("GeoIP2-City-Europe.mmdb")
if err != nil {
panic(err)
}
defer db.Close()
var res interface{}
db.Decode(0, &res)
fmt.Println(res)
}How do I know which offset is needed for the second item?
in the process of using Sometimes we needs to determine if the status of mmdb is closed or not.
errors.New("cannot call Lookup on a closed database")
Now that I'm out there judging strings, there should be a more elegant way to do it?
Looking forward to the author's recovery~
Hey!
First of all: great lib!
I would have a couple question though: Is this package enough to get a country name / city name for a single IP? Or is it only possible with the more complex geoip2 package?
My needs would only be to get the country and city names, preferably in native language, but if that's not possible English works just as well. The problem is, I can't find any reference/docs about the available structure nodes/names. The example only shows country/iso_code. Is there a list about this somewhere?
Update: I've just gone gambling and found them! :)
To start with: thank you for this great library, it is really useful. I am an amateur dev so apologies if the question is obvious.
When doing a .Lookup(), the data is extracted either into any or a struct created manually. Is there a reason why there is no type associated with the result that would encompass all the fields available as a result?
In other words, why isn't there a default struct such as the one in the example
var record struct {
Country struct {
ISOCode string `maxminddb:"iso_code"`
} `maxminddb:"country"`
} // Or any appropriate structwith all the available fields?
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.