mteodoro / mmutils Goto Github PK
View Code? Open in Web Editor NEWTools for working with MaxMind GeoIP csv and dat files
License: MIT License
Tools for working with MaxMind GeoIP csv and dat files
License: MIT License
Hi there
I'd first like to thank you for your csv2dat.py script. I have used it to merge the lat/long positions of our internal 10/8 network subnets into the maxmind CSV files and now we can do things like have maps showing our internal devices using websites/etc as well as Internet :-)
However, now we're seeing applications that only support the newer MMDB format. Have you thought about adding support for that as an option? So far there only seems to be a perl "writer" from Maxmind - but nothing for python :-(
In any case, thanks for your work!
Jason
Hi there!
Maybe i'll lucky and someone can help me with this deprecated project.
I try to build geolitecity from csv (Source: GeoLiteCity-flat-sorted.csv.zip) with csv2dat.py -w GeoLiteCity.dat mmcity GeoLiteCity-flat-sorted.csv
. When i done, i check it with, for example, ip 191.243.53.129 .
In debug file (when we use csv2dat.py
with -d
key) i see:
2148767 ["14702888 (('BR', 'Parana', 'Ampere', '', '-25.9167', '-53.4667', '', '20'), IPv4Network('191.243.48.0/22'))", "17135708 (('BR', 'Santa Catarina', 'Abelardo Luz', '89830', '-26.5983', '-52.2232', '', '50'), IPv4Network('191.243.52.0/22'))"]
2148768 ["16892194 (('BR', 'Minas Gerais', 'Pompeu', '35640', '-19.1495', '-44.9149', '', '10'), IPv4Network('191.243.56.0/22'))", 2148769]
BUT when i ask my GeoLiteCity.dat file "tell me, who is 191.243.53.129" with python (pygeoip, GeoIP modules) or geoiplookup
i got: US, Pennsylvania, N/A, Philadelphia, 19123, 39.964298, -75.144402, 504, 500
So, maybe someone can explain me, why it done? Why info in .dat file is wrong? And how fix it?
P.S. I'm desperate
Hi there,
I tried to use this project to convert GeoIP2 csv to dat file. But it did not work. Do you have plan to add this feature ?
I need one, thank you.
can you set a license for the scripts? in doubt i would recommend MIT.
Hi there
I want to use csv2dat.py to merge into GeoIP geographic locations for our 10/8 network (200 sites, I've got lat/long/city by subnet)
So I duplicated the CSV format of GeoLiteCity-Location.csv/GeoLiteCity-Blocks.csv appended my data - but csv2dat crashes as follows when I run it. BTW, if I run it on the original GeoIP csv files, it works fine, and I even made a 10-line version where I hand-picked some data points from GeoIP plus mine and that worked too. I suspect there's a simple bad char or something in my much larger data set - but it's too big to parse by eyeball. How could I add some debugging code to at least tell me what line in locations/blocks is causing the fault?
Here's the runtime error. Thanks!
csv2geoipDAT.py -w mmcity.dat -l locations.csv mmcity blocks.csv
Traceback (most recent call last):
File "/usr/local/bin/csv2geoipDAT.py", line 475, in
rval = main()
File "/usr/local/bin/csv2geoipDAT.py", line 471, in main
return cmd(opts, args)
File "/usr/local/bin/csv2geoipDAT.py", line 439, in build_dat
r.load(opts, args)
File "/usr/local/bin/csv2geoipDAT.py", line 196, in load
self[net] = data
File "/usr/local/bin/csv2geoipDAT.py", line 170, in setitem
if not node.lhs:
AttributeError: 'tuple' object has no attribute 'lhs'
c:\Temp\mmdb-convert-master>mmdb-convert.py GeoLite2-City.mmdb > GeoIPCity.dat
Traceback (most recent call last):
File "C:\Temp\mmdb-convert-master\mmdb-convert.py", line 463, in
metadata, the_tree, _ = parse_mm_file(content)
File "C:\Temp\mmdb-convert-master\mmdb-convert.py", line 327, in parse_mm_file
tree = parse_search_tree(s[:tree_size], mm['record_size'].int_val())
File "C:\Temp\mmdb-convert-master\mmdb-convert.py", line 123, in parse_search_tree
left, right = to_leftright(s[p:p+record_bytes])
File "C:\Temp\mmdb-convert-master\mmdb-convert.py", line 80, in to_int28
a, b = unpack("!LL", s + b'\x00')
NameError: name 'unpack' is not defined
I've run the conversion a few times, and I find that the script doesn't handle when there is duplicate IPs.
Line 205 gets an except when there is a duplicate start IP address
Line 189 gets an except if you define overlapping IP address
Traceback (most recent call last):
File "./csv2dat.py", line 475, in <module>
rval = main()
File "./csv2dat.py", line 471, in main
return cmd(opts, args)
File "./csv2dat.py", line 439, in build_dat
r.load(opts, args)
File "./csv2dat.py", line 194, in load
for nets, data in self.gen_nets(opts, args):
File "./csv2dat.py", line 317, in gen_nets
lo, hi = row[:2]
ValueError: need more than 0 values to unpack
`Please help on this error.
/System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7 /Users/admin/PycharmProjects/geoip/csv2dat.py -w GeoIPCity.dat mmorg GeoIPCity.csv
Traceback (most recent call last):
File "/Users/admin/PycharmProjects/geoip/csv2dat.py", line 475, in
rval = main()
File "/Users/admin/PycharmProjects/geoip/csv2dat.py", line 471, in main
return cmd(opts, args)
File "/Users/admin/PycharmProjects/geoip/csv2dat.py", line 439, in build_dat
r.load(opts, args)
File "/Users/admin/PycharmProjects/geoip/csv2dat.py", line 194, in load
for nets, data in self.gen_nets(opts, args):
File "/Users/admin/PycharmProjects/geoip/csv2dat.py", line 261, in gen_nets
for lo, hi, asn in gen_csv(fileinput.input(args)):
ValueError: too many values to unpack
It would be nice to restructure package that way it could be installed via setup tools and used as standalone library as the standalone tool it does at the moment.
We could make PR for this, if maintainer interested in it.
Hi
My csv file contains characters (isp names) out range of ascii.
After transforming, my application could read the dat file, but result in unreadable code.
Thank you for help.
Quick question - Is this error related to my environment, the import file, or something else? This file does not contain any IPv6 addresses...
root# ./csv2dat.py -w Citrix_mynewdb.dat mmcity myoldcsv.csv
Traceback (most recent call last):
File "./csv2dat.py", line 270, in
class ASNv6RadixTree(ASNRadixTree):
File "./csv2dat.py", line 274, in ASNv6RadixTree
edition = pygeoip.const.ASNUM_EDITION_V6
AttributeError: 'module' object has no attribute 'ASNUM_EDITION_V6'
[root]#
I'm really looking to use this utility to modify info from GeoIP Country (GEO-108) and GeoIP ISP (GEO-124) csv files and produce the corresponding dat files. For the country we only access the .country_code_by_addr and for the isp we only access the .org_by_addr methods of the python api.
Please, any info or help would be appreciated. Thank you for making this utility available in the first place.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.