oschwald / maxminddb-rust

Rust MaxMind DB Reader

Rust MaxMind DB Reader

This library reads the MaxMind DB format, including the GeoIP2 and GeoLite2 databases.

Building

To build everything:

cargo build

Testing

This crate manages its test data within a git submodule. To run the tests, you will first need to run the following command.

git submodule update --init

Usage

Add this to your Cargo.toml:

[dependencies]
maxminddb = "0.17"

and this to your crate root:

extern crate maxminddb;

API Documentation

The API docs are on GitHub Pages.

Example

See examples/lookup.rs for a basic example.

Benchmarks

The projects include benchmarks using Criterion.rs.

First you need to have a working copy of the GeoIP City database. You can fetch it from here.

Place it in the root folder as GeoIP2-City.mmdb.

Once this is done, run

cargo bench

If gnuplot is installed, Criterion.rs can generate an HTML report displaying the results of the benchmark under target/criterion/report/index.html.

Result of doing 100 random IP lookups:

Contributing

Contributions welcome! Please fork the repository and open a pull request with your changes.

License

This is free software, licensed under the ISC license.

maxminddb-rust's Issues

No traits defined on maxminddb::geoip2::City

Hi,
I'm getting compiler errors telling me that "the trait 'serde::ser::Serialize' is not implemented for 'maxminddb::geoip2::City'" when trying to serialize a City type with serde_json.

Upon closer inspections, it seems that my City doesn't have any traits implemented.

I'm using:
rustc 1.27.2
maxminddb 0.9.0
serde_json 1.0.24

Any idea what is going on?

lookup fails on private IP address

Hi,

when I lookup a private address I get the main thread to panic, here is an example

fn main() {
let reader = maxminddb::Reader::open_readfile("./GeoIP2-ISP.mmdb").unwrap()
let isp: geoip2::Isp = reader.lookup("127.0.0.1").unwrap;
}

If I run this code I get

thread 'main' panicked at 'called Result::unwrap() on an Err value: AddressNotFoundError("Address not found in database")', src/libcore/result.rs:999:5 note: run with RUST_BACKTRACE=1 environment variable to display a backtrace.

Can't build with panic = "abort"

Using this crate with the following option in Cargo.toml would cause a compile error

[profile.release]
panic = "abort"

~/rust/testmmdb$ cargo build --release
   Compiling cfg-if v0.1.10
   Compiling serde v1.0.115
   Compiling log v0.4.11
   Compiling maxminddb v0.14.0
error: the linked panic runtime `panic_unwind` is not compiled with this crate's panic strategy `abort`

error: aborting due to previous error

error: could not compile `maxminddb`.

Perhaps we can remove this line ?

Support decoding unsigned 128-bit integers to u128

When this reader was first created, there wasn't a u128 data type. As such, the current implementation just decodes them to bytes.

Update repository info

Would be nice to add repository topics like go version and add info about unsafe-str-decode and bench with it.

InvalidDatabaseError: Unknown data type: 120

Downloading the newest version of maxmind's GeoIP2-ISP database gives the error 'InvalidDatabaseError: Unknown data type: 120' when loaded using this lib. I don't know if you have access to this (can't share it, I think) and it might be a corrupt database that is the issue but I wanted to report it in case anyone has any idea. Going to look into it more myself first on monday.

Custom fields in mmdb file will result in those fields not being de-serialized due to use of structs

I'm using a custom serialized mmdb file, validated with https://github.com/maxmind/mmdbinspect (which uses the go library from you) , but any IP I lookup only returns continent code and LAT/LONG.

For example looking up the IP address 87.1.125.194 yields the proper location using mmdbinspect but yields only the 2 fields above with maxminddb-rust

Metadata { binary_format_major_version: 2, binary_format_minor_version: 0, build_epoch: 1654507000, database_type: "Magellan", description: {"en": "Magellan"}, ip_version: 6, languages: [], node_count: 17427449, record_size: 32 }

This is the meta data of the loaded databases, which loads fine and raises no errors.

And this is the code used to lookup the IP addresses, which is just the basic code from the examples:

    for ip in ips.iter() {
        println!("-----");
        println!("{}", ip);
        let ip: std::net::IpAddr = ip.parse().unwrap();
        let city: maxminddb::geoip2::City = geo_reader.lookup(ip).unwrap();
        println!("{:?}", city);
    }

I can probably share the database in private since it's over 100MB to debug if needed, but since the go library works with the file in question, I'm wondering if there's any known gap between this rust library and the go one.

Taking ownership of Vec<u8>, unsafe usage in open_readfile

#15 Made a change that allows creating a reader by transferring ownership of an Vec into the reader. This was changed in #16 and introduced some unsafe code into open_readfile, which we should try to get rid of.

It makes sense that unsafe code is needed for an mmap, but it shouldn't be used to simply load a file into memory. If we re-introduce a way to transfer a Vec into the reader we could just use that to read from disk without any unsafe code.

We might be able to use Cow for this.

Feature request: encoding/writer functionality

Thanks for writing this - interesting read and implementation.
I'm hoping to find a library to use in some small projects for converting between the various GeoIP formats and updating databases from external sources - any chance writer functionality is planned for the library in supported formats?

Replace travis-cargo

travis-cargo no longer appears to be maintained and several features are broken. We primarily use it for the doc generation. It might be simplest to switch back to a hand-rolled .travis.yml, but it would be worth exploring alternatives such as trust.

Benchmarks

Hey @oschwald,

thanks for this crate.
I made a few benchmarks using criterion and the results can be found over here. If you like, you can copy over the files to your repository.

Bottom line is that maxminddb-rust is quite fast with 30.000 lookups/s on my machine (Macbook Pro 2015, 16GB RAM).
With rayon it goes all the way up to 100.000 lookups/s.

License Apache and/or MIT

Hello,
Any reason to not release this crate using standard Apache 2.0 or MIT license?

Thx

Decoder works with missing fields

Currently if a struct field is missing in the database, the reader will throw an error, e.g., DecodingError(Unknown struct field represented_country).

Related issue with the Rust JSON decoder. Ideally the behavior of this API would be similar to whatever the final behavior of that API is.

Switch to Serde

Currently rustc_serialize is used internally.

Question: Compiling to ARM

Howdy,

Just wondering if it's possible to compile this to ARM? Maybe I am doing something wrong

I tried to compile with cargo build --target aarch64-unknown-linux-gnu

Result:

cargo build --release --target aarch64-unknown-linux-gnu
    Updating crates.io index
  Downloaded serde_derive v1.0.116
  Downloaded syn v1.0.42
  Downloaded serde v1.0.116
  Downloaded proc-macro2 v1.0.23
  Downloaded 4 crates (387.7 KB) in 1.13s
   Compiling proc-macro2 v1.0.23
   Compiling unicode-xid v0.2.1
   Compiling syn v1.0.42
   Compiling serde_derive v1.0.116
   Compiling serde v1.0.116
   Compiling log v0.4.11
   Compiling cfg-if v0.1.10
   Compiling quote v1.0.7
   Compiling maxminddb v0.14.0 (/Users/vsyrakis/projs/maxminddb-rust)
error: linking with `cc` failed: exit code: 1
  |
  = note: "cc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-Wl,--eh-frame-hdr" "-L" "/Users/vsyrakis/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/aarch64-unknown-linux-gnu/lib" "/Users/vsyrakis/projs/maxminddb-rust/target/aarch64-unknown-linux-gnu/release/deps/maxminddb.maxminddb.70b95j1z-cgu.0.rcgu.o" "/Users/vsyrakis/projs/maxminddb-rust/target/aarch64-unknown-linux-gnu/release/deps/maxminddb.maxminddb.70b95j1z-cgu.1.rcgu.o" "/Users/vsyrakis/projs/maxminddb-rust/target/aarch64-unknown-linux-gnu/release/deps/maxminddb.maxminddb.70b95j1z-cgu.10.rcgu.o" "/Users/vsyrakis/projs/maxminddb-rust/target/aarch64-unknown-linux-gnu/release/deps/maxminddb.maxminddb.70b95j1z-cgu.11.rcgu.o" "/Users/vsyrakis/projs/maxminddb-rust/target/aarch64-unknown-linux-gnu/release/deps/maxminddb.maxminddb.70b95j1z-cgu.12.rcgu.o" "/Users/vsyrakis/projs/maxminddb-rust/target/aarch64-unknown-linux-gnu/release/deps/maxminddb.maxminddb.70b95j1z-cgu.13.rcgu.o" "/Users/vsyrakis/projs/maxminddb-rust/target/aarch64-unknown-linux-gnu/release/deps/maxminddb.maxminddb.70b95j1z-cgu.14.rcgu.o" "/Users/vsyrakis/projs/maxminddb-rust/target/aarch64-unknown-linux-gnu/release/deps/maxminddb.maxminddb.70b95j1z-cgu.15.rcgu.o" "/Users/vsyrakis/projs/maxminddb-rust/target/aarch64-unknown-linux-gnu/release/deps/maxminddb.maxminddb.70b95j1z-cgu.2.rcgu.o" "/Users/vsyrakis/projs/maxminddb-rust/target/aarch64-unknown-linux-gnu/release/deps/maxminddb.maxminddb.70b95j1z-cgu.3.rcgu.o" "/Users/vsyrakis/projs/maxminddb-rust/target/aarch64-unknown-linux-gnu/release/deps/maxminddb.maxminddb.70b95j1z-cgu.4.rcgu.o" "/Users/vsyrakis/projs/maxminddb-rust/target/aarch64-unknown-linux-gnu/release/deps/maxminddb.maxminddb.70b95j1z-cgu.5.rcgu.o" "/Users/vsyrakis/projs/maxminddb-rust/target/aarch64-unknown-linux-gnu/release/deps/maxminddb.maxminddb.70b95j1z-cgu.6.rcgu.o" "/Users/vsyrakis/projs/maxminddb-rust/target/aarch64-unknown-linux-gnu/release/deps/maxminddb.maxminddb.70b95j1z-cgu.7.rcgu.o" "/Users/vsyrakis/projs/maxminddb-rust/target/aarch64-unknown-linux-gnu/release/deps/maxminddb.maxminddb.70b95j1z-cgu.8.rcgu.o" "/Users/vsyrakis/projs/maxminddb-rust/target/aarch64-unknown-linux-gnu/release/deps/maxminddb.maxminddb.70b95j1z-cgu.9.rcgu.o" "-o" "/Users/vsyrakis/projs/maxminddb-rust/target/aarch64-unknown-linux-gnu/release/deps/libmaxminddb.so" "-Wl,--version-script=/var/folders/m4/03w2g5951fzg8w7dsvn1qrv80000gn/T/rustc4ZFrN2/list" "/Users/vsyrakis/projs/maxminddb-rust/target/aarch64-unknown-linux-gnu/release/deps/maxminddb.y95r4q68tm7xkuo.rcgu.o" "/Users/vsyrakis/projs/maxminddb-rust/target/aarch64-unknown-linux-gnu/release/deps/maxminddb.19k1k1rf9qu3trsy.rcgu.o" "-shared" "-Wl,-zrelro" "-Wl,-znow" "-Wl,-O1" "-nodefaultlibs" "-L" "/Users/vsyrakis/projs/maxminddb-rust/target/aarch64-unknown-linux-gnu/release/deps" "-L" "/Users/vsyrakis/projs/maxminddb-rust/target/release/deps" "-L" "/Users/vsyrakis/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-Wl,-Bstatic" "-Wl,--whole-archive" "/var/folders/m4/03w2g5951fzg8w7dsvn1qrv80000gn/T/rustc4ZFrN2/libserde-ee32710b218a5468.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/var/folders/m4/03w2g5951fzg8w7dsvn1qrv80000gn/T/rustc4ZFrN2/liblog-65873595f5705df7.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/var/folders/m4/03w2g5951fzg8w7dsvn1qrv80000gn/T/rustc4ZFrN2/libcfg_if-7953ba159f8e7f1f.rlib" "-Wl,--no-whole-archive" "-Wl,--start-group" "-Wl,--whole-archive" "/var/folders/m4/03w2g5951fzg8w7dsvn1qrv80000gn/T/rustc4ZFrN2/libstd-4b444b20e16fa790.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/var/folders/m4/03w2g5951fzg8w7dsvn1qrv80000gn/T/rustc4ZFrN2/libpanic_unwind-b081523fbd37302b.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/var/folders/m4/03w2g5951fzg8w7dsvn1qrv80000gn/T/rustc4ZFrN2/libhashbrown-732478e5890bb19b.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/var/folders/m4/03w2g5951fzg8w7dsvn1qrv80000gn/T/rustc4ZFrN2/librustc_std_workspace_alloc-506221938f181c12.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/var/folders/m4/03w2g5951fzg8w7dsvn1qrv80000gn/T/rustc4ZFrN2/libbacktrace-ccd35fce129b018b.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/var/folders/m4/03w2g5951fzg8w7dsvn1qrv80000gn/T/rustc4ZFrN2/libbacktrace_sys-332ada1ae0a2c29e.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/var/folders/m4/03w2g5951fzg8w7dsvn1qrv80000gn/T/rustc4ZFrN2/librustc_demangle-754461d570cf4b14.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/var/folders/m4/03w2g5951fzg8w7dsvn1qrv80000gn/T/rustc4ZFrN2/libunwind-0c74709359820092.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/var/folders/m4/03w2g5951fzg8w7dsvn1qrv80000gn/T/rustc4ZFrN2/libcfg_if-8dd162cdf2826afa.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/var/folders/m4/03w2g5951fzg8w7dsvn1qrv80000gn/T/rustc4ZFrN2/liblibc-f1b9594dad3ab4b5.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/var/folders/m4/03w2g5951fzg8w7dsvn1qrv80000gn/T/rustc4ZFrN2/liballoc-6ceb1876b4ee5d23.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/var/folders/m4/03w2g5951fzg8w7dsvn1qrv80000gn/T/rustc4ZFrN2/librustc_std_workspace_core-147dc72ae5a8b57d.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/var/folders/m4/03w2g5951fzg8w7dsvn1qrv80000gn/T/rustc4ZFrN2/libcore-d35a752fdcd79eb4.rlib" "-Wl,--no-whole-archive" "-Wl,--end-group" "/var/folders/m4/03w2g5951fzg8w7dsvn1qrv80000gn/T/rustc4ZFrN2/libcompiler_builtins-1887fc639a1f6ebc.rlib" "-Wl,-Bdynamic" "-ldl" "-lrt" "-lpthread" "-lgcc_s" "-lc" "-lm" "-lrt" "-lpthread" "-lutil" "-ldl" "-lutil"
  = note: ld: unknown option: --as-needed
          clang: error: linker command failed with exit code 1 (use -v to see invocation)


error: aborting due to previous error

error: could not compile `maxminddb`.

To learn more, run the command again with --verbose.

I think the issue is note: ld: unknown option: --as-needed, but I'm not really sure what I can do to change this.
I'm on OSX, maybe that's why.

But I also tried this and got the following error on the rust docker container rust:1, from here: https://hub.docker.com/r/rustlang/rust/tags

Result:

cargo build --release --target aarch64-unknown-linux-gnu
   Compiling cfg-if v0.1.10
   Compiling proc-macro2 v1.0.23
   Compiling syn v1.0.42
   Compiling log v0.4.11
   Compiling serde v1.0.116
   Compiling serde_derive v1.0.116
   Compiling quote v1.0.7
   Compiling maxminddb v0.14.0 (/tmp/maxminddb-rust)
error: linking with `cc` failed: exit code: 1
  |
  = note: "cc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-Wl,--eh-frame-hdr" "-L" "/usr/local/rustup/toolchains/1.46.0-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib" "/tmp/maxminddb-rust/target/aarch64-unknown-linux-gnu/release/deps/maxminddb.maxminddb.70b95j1z-cgu.0.rcgu.o" "/tmp/maxminddb-rust/target/aarch64-unknown-linux-gnu/release/deps/maxminddb.maxminddb.70b95j1z-cgu.1.rcgu.o" "/tmp/maxminddb-rust/target/aarch64-unknown-linux-gnu/release/deps/maxminddb.maxminddb.70b95j1z-cgu.10.rcgu.o" "/tmp/maxminddb-rust/target/aarch64-unknown-linux-gnu/release/deps/maxminddb.maxminddb.70b95j1z-cgu.11.rcgu.o" "/tmp/maxminddb-rust/target/aarch64-unknown-linux-gnu/release/deps/maxminddb.maxminddb.70b95j1z-cgu.12.rcgu.o" "/tmp/maxminddb-rust/target/aarch64-unknown-linux-gnu/release/deps/maxminddb.maxminddb.70b95j1z-cgu.13.rcgu.o" "/tmp/maxminddb-rust/target/aarch64-unknown-linux-gnu/release/deps/maxminddb.maxminddb.70b95j1z-cgu.14.rcgu.o" "/tmp/maxminddb-rust/target/aarch64-unknown-linux-gnu/release/deps/maxminddb.maxminddb.70b95j1z-cgu.15.rcgu.o" "/tmp/maxminddb-rust/target/aarch64-unknown-linux-gnu/release/deps/maxminddb.maxminddb.70b95j1z-cgu.2.rcgu.o" "/tmp/maxminddb-rust/target/aarch64-unknown-linux-gnu/release/deps/maxminddb.maxminddb.70b95j1z-cgu.3.rcgu.o" "/tmp/maxminddb-rust/target/aarch64-unknown-linux-gnu/release/deps/maxminddb.maxminddb.70b95j1z-cgu.4.rcgu.o" "/tmp/maxminddb-rust/target/aarch64-unknown-linux-gnu/release/deps/maxminddb.maxminddb.70b95j1z-cgu.5.rcgu.o" "/tmp/maxminddb-rust/target/aarch64-unknown-linux-gnu/release/deps/maxminddb.maxminddb.70b95j1z-cgu.6.rcgu.o" "/tmp/maxminddb-rust/target/aarch64-unknown-linux-gnu/release/deps/maxminddb.maxminddb.70b95j1z-cgu.7.rcgu.o" "/tmp/maxminddb-rust/target/aarch64-unknown-linux-gnu/release/deps/maxminddb.maxminddb.70b95j1z-cgu.8.rcgu.o" "/tmp/maxminddb-rust/target/aarch64-unknown-linux-gnu/release/deps/maxminddb.maxminddb.70b95j1z-cgu.9.rcgu.o" "-o" "/tmp/maxminddb-rust/target/aarch64-unknown-linux-gnu/release/deps/libmaxminddb.so" "-Wl,--version-script=/tmp/rustcMfa47O/list" "/tmp/maxminddb-rust/target/aarch64-unknown-linux-gnu/release/deps/maxminddb.y95r4q68tm7xkuo.rcgu.o" "/tmp/maxminddb-rust/target/aarch64-unknown-linux-gnu/release/deps/maxminddb.19k1k1rf9qu3trsy.rcgu.o" "-shared" "-Wl,-zrelro" "-Wl,-znow" "-Wl,-O1" "-nodefaultlibs" "-L" "/tmp/maxminddb-rust/target/aarch64-unknown-linux-gnu/release/deps" "-L" "/tmp/maxminddb-rust/target/release/deps" "-L" "/usr/local/rustup/toolchains/1.46.0-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-Wl,-Bstatic" "-Wl,--whole-archive" "/tmp/rustcMfa47O/libserde-d7f282ca7f563c08.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/tmp/rustcMfa47O/liblog-ec0666624e9ed146.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/tmp/rustcMfa47O/libcfg_if-f3da84320fe405e4.rlib" "-Wl,--no-whole-archive" "-Wl,--start-group" "-Wl,--whole-archive" "/tmp/rustcMfa47O/libstd-4b444b20e16fa790.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/tmp/rustcMfa47O/libpanic_unwind-b081523fbd37302b.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/tmp/rustcMfa47O/libhashbrown-732478e5890bb19b.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/tmp/rustcMfa47O/librustc_std_workspace_alloc-506221938f181c12.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/tmp/rustcMfa47O/libbacktrace-ccd35fce129b018b.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/tmp/rustcMfa47O/libbacktrace_sys-332ada1ae0a2c29e.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/tmp/rustcMfa47O/librustc_demangle-754461d570cf4b14.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/tmp/rustcMfa47O/libunwind-0c74709359820092.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/tmp/rustcMfa47O/libcfg_if-8dd162cdf2826afa.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/tmp/rustcMfa47O/liblibc-f1b9594dad3ab4b5.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/tmp/rustcMfa47O/liballoc-6ceb1876b4ee5d23.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/tmp/rustcMfa47O/librustc_std_workspace_core-147dc72ae5a8b57d.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/tmp/rustcMfa47O/libcore-d35a752fdcd79eb4.rlib" "-Wl,--no-whole-archive" "-Wl,--end-group" "/tmp/rustcMfa47O/libcompiler_builtins-1887fc639a1f6ebc.rlib" "-Wl,-Bdynamic" "-ldl" "-lrt" "-lpthread" "-lgcc_s" "-lc" "-lm" "-lrt" "-lpthread" "-lutil" "-ldl" "-lutil"
  = note: /usr/bin/ld: /tmp/maxminddb-rust/target/aarch64-unknown-linux-gnu/release/deps/maxminddb.maxminddb.70b95j1z-cgu.0.rcgu.o: Relocations in generic ELF (EM: 183)
          /usr/bin/ld: /tmp/maxminddb-rust/target/aarch64-unknown-linux-gnu/release/deps/maxminddb.maxminddb.70b95j1z-cgu.0.rcgu.o: error adding symbols: file in wrong format
          collect2: error: ld returned 1 exit status


error: aborting due to previous error

error: could not compile `maxminddb`.

To learn more, run the command again with --verbose.

Any direction would be helpful. Perhaps I can make the changes and raise a PR if it's straightforward enough.

Edit: it does seem to compile for aarch64-unknown-linux-musl... so that's something