t2mune / mrtparse Goto Github PK
View Code? Open in Web Editor NEWMRT format data parser
License: Apache License 2.0
MRT format data parser
License: Apache License 2.0
used Version: 1.4
used mrt files: samples/quagga_bgp, samples/quagga_rib, one from RIPE RIS (updates dump), two from routeviews.org (updates dump, rib dump)
observed behaviour:
with (-A|-G) flag (api (grouping) format): mrt2exabgp.py doesn't finish by itself; it prints the correct routes with rib dumps (nothing for updates dumps->other issue) but it stays alive till the user finishes it with strg+c for example
Regards,
Daniel
Hi,
First of all I want to say that I appreciate a lot this tool and that it's been helping me to read the BGP Updates messages. Nevertheless, I am having some trouble now. Here is what happens:
I am downloading one of the updates from: http://routeviews.org/bgpdata/2018.03/UPDATES/
For example: updates.20180322.1430.bz2
I run: python2.7 mrt2bgpdump.py -m -O output.txt updates.20180322.1430.bz2
And it just crashes with the following output:
Traceback (most recent call last):
File "mrt2bgpdump.py", line 302, in
main()
File "mrt2bgpdump.py", line 298, in main
b.bgp4mp(m, count)
File "mrt2bgpdump.py", line 206, in bgp4mp
self.bgp_attr(attr)
File "mrt2bgpdump.py", line 268, in bgp_attr
self.as4_aggr = '%d %s' % (attr.as4_aggr['asn'], attr.as4_aggr['ip'])
KeyError: 'ip'
In line 267 and 268 we have:
elif attr.type == BGP_ATTR_T['AS4_AGGREGATOR']:
self.as4_aggr = '%d %s' % (attr.as4_aggr['asn'], attr.as4_aggr['ip'])
So for some reason the dictionary is not recognizing the key.
It seems that in all the previous tables I was using this attribute doesn't appear (at least I checked a couple and effectively it's not there). But now that I am taking Updates from routeviews, it creates some trouble. I commented the line after the if, and with a print I could see that the cases in which this attribute appears is quite low, but I did not know what else I could do, so I just leave you this message.
I hope I am not making any horrible mistake!
Thanks for your time in advance,
Best regards,
Julian.
Hello , I wish install full routing table in a Router but before Convert the RAW BGP data from RIPE into a format that ExaBGP can process, however i get a issue.
I tried with mrtparse and http://data.ris.ripe.net/rrc16/latest-bview.gz and then with mrt2exabgp -G -P ./latest-bview.gz > fullbgptable.py. The session is UP, not received prefix, but i got the same error message in the router.
sh bgp summary
Neighbor Spk AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down St/PfxRcd
192.168.1.111 0 200 1608 12 4 0 0 00:00:24 0
RP/0/0/CPU0:ios#RP/0/0/CPU0:Oct 21 02:05:15.424 : bgp[1052]: %ROUTING-BGP-3-MALFORM_UPDATE : Malformed UPDATE message received from neighbor 192.168.1.111 (VRF: default) - message length 269 bytes, error flags 0x00000200, action taken "TreatAsWdr". Error details: "Error 0x00000200, Field "Attr-data", Attribute 2 (Flags 0x40, Length 18), Data [40021202040000cc6000000da300000d]". NLRIs: [IPv4 Unicast] 217.14.120.0/22 205.167.22.0/23 205.167.188.0/23 206.221.126.0/23 199.254.216.0/24 200.41.49.0/24 200.41.53.0/24...
The code after mrtparse is:
#!/usr/bin/env python
import sys
import time
msgs = [
'announce attributes origin IGP as-path [199524 2914 ] next-hop 198.32.243.133 nlri 0.0.0.0/0',
'announce attributes origin IGP as-path [199524 2914 ] next-hop 2001:504:0:6:0:19:9524:1 nlri ::/0 2001:218::/32 2001:218:2002::/48 2001:218:8000::/38 2001:418::/32 2001:728::/32 2>
'announce attributes origin IGP as-path [52320 13335 ] med 0 aggregator (13335:108.162.211.10) community [52320:31912] next-hop 198.32.242.254 nlri 1.0.0.0/24 1.1.1.0/24 8.27.64.0/>
'announce attributes origin IGP as-path [6939 4826 38803 ] med 0 next-hop 198.32.242.176 nlri 1.0.4.0/22 1.0.4.0/24 1.0.5.0/24 1.0.6.0/24 1.0.7.0/24 43.252.108.0/22 43.252.108.0/24>
'announce attributes origin IGP as-path [52320 3491 2516 7670 18144 ] med 0 aggregator (18144:219.118.225.18) community [52320:11912] next-hop 198.32.242.254 nlri 1.0.64.0/18 58.18>
'announce attributes origin IGP as-path [6939 38040 23969 ] med 0 next-hop 198.32.242.176 nlri 1.0.128.0/17 1.0.128.0/18 1.0.128.0/19 1.0.130.0/23 1.0.132.0/24 1.0.133.0/24 1.0.136>
'announce attributes origin IGP as-path [6939 4651 23969 ] med 0 next-hop 198.32.242.176 nlri 1.0.128.0/24 1.0.129.0/24 1.0.137.0/24 1.0.144.0/20 1.0.160.0/22 1.0.164.0/24 1.0.165.>
'announce attributes origin IGP as-path [52320 4134 58543 ] med 0 community [52320:21912] next-hop 198.32.242.254 nlri 1.1.8.0/24 121.201.0.0/18 121.201.64.0/18',
.....
while msgs:
msg = msgs.pop(0)
if isinstance(msg, str):
sys.stdout.write(msg + '\n')
sys.stdout.flush()
else:
time.sleep(msg)
while True:
time.sleep(1)
How can i get data in correct format ?
Bird already supports Extended NLRI Encodings in order to offer Advertisement of Multiple Paths in BGP. When bird is configured with the multiple paths, the nlris are extended encoded on the mrt path.
Is it possible to add the support of extended nlri encodings?
$ lsb_release -a
LSB Version: core-11.1.0ubuntu4-noarch:printing-11.1.0ubuntu4-noarch:security-11.1.0ubuntu4-noarch
Distributor ID: Ubuntu
Description: Ubuntu 22.04.4 LTS
Release: 22.04
Codename: jammy
$ python -V
Python 3.10.12
$ pip3 freeze | grep mrt
mrtparse==2.2.0
$ mrt2exabgp -P latest-bview.gz
#!/usr/bin/env python
import sys
import time
msgs = [
Traceback (most recent call last):
File "/usr/bin/mrt2exabgp", line 498, in <module>
main()
File "/usr/bin/mrt2exabgp", line 495, in main
conv_format(args, flags, d)
File "/usr/bin/mrt2exabgp", line 192, in conv_format
m = m.mrt
^^^^^
AttributeError: 'Reader' object has no attribute 'mrt'
Hi, I have been able to advertise the full internet table with exabgp using the old 'announce route' syntax however I believe there is a more efficient way to do this using a newer 'api' format. I can see in the article that there is an 'announce attributes' syntax. https://github.com/Exa-Networks/exabgp/wiki/Large-Configuration-File and that mrtparse allows you to generate this config using the flags -G and -P
I have used the command:
python /usr/local/lib/python2.7/dist-packages/mrtparse/examples/mrt2exabgp.py -G -P latest-bview.gz > fullbgptable.py
When I run this command the process hangs and does that send any output to the file. If I omit the -G then the command works but I end up with the old syntax.
If anyone has any experience with getting this working then any help would be appreciated.
Thanks.
Using http://data.ris.ripe.net/rrc00/latest-bview.gz:
root@pgen:/usr/local/etc/exabgp# /home/jlixfeld/mrtparse/examples/mrt2exabgp.py -A /home/jlixfeld/latest-bview.gz
announce route 0.0.0.0/0 origin IGP as-path [45896 3356 ] community [3356:3 3356:9 3356:575 3356:2003] next-hop 111.91.233.1
announce route ::/0 origin IGP as-path [29608 5511 5511 5511 5511 36925 ] med 11 community [5511:500 5511:518 5511:999 5511:5511 29608:30400] next-hop 2a01:678::2
announce route 1.0.4.0/22 origin IGP as-path [45896 4826 38803 56203 ] community [19996:2001] next-hop 111.91.233.1
announce route 1.0.4.0/24 origin IGP as-path [45896 4826 38803 56203 ] community [19996:2001] next-hop 111.91.233.1
...
...
...
Traceback (most recent call last):
File "/home/jlixfeld/mrtparse/examples/mrt2exabgp.py", line 498, in
main()
File "/home/jlixfeld/mrtparse/examples/mrt2exabgp.py", line 495, in main
conv_format(args, flags, d)
File "/home/jlixfeld/mrtparse/examples/mrt2exabgp.py", line 191, in conv_format
for m in d:
File "/usr/local/lib/python2.7/dist-packages/mrtparse-1.6-py2.7.egg/mrtparse/init.py", line 609, in next
self.unpack_data()
File "/usr/local/lib/python2.7/dist-packages/mrtparse-1.6-py2.7.egg/mrtparse/init.py", line 640, in unpack_data
data = self.f.read(self.mrt.len)
File "/usr/lib/python2.7/gzip.py", line 268, in read
self._read(readsize)
File "/usr/lib/python2.7/gzip.py", line 315, in _read
self._read_eof()
File "/usr/lib/python2.7/gzip.py", line 354, in _read_eof
hex(self.crc)))
IOError: CRC check failed 0x579c0579 != 0xd1fd5a7dL
I'm not well enough versed in Python to really be able to understand how to troubleshoot this. Any assistance would be appreciated!
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "", line 1, in
File "C:\Users\salmacmp\AppData\Local\Temp\pip-req-build-fs_0nxwc\setup.py", line 32, in
import mrtparse
File "C:\Users\salmacmp\AppData\Local\Temp\pip-req-build-fs_0nxwc\mrtparse_init_.py", line 31, in
signal.signal(signal.SIGPIPE, signal.SIG_DFL)
AttributeError: module 'signal' has no attribute 'SIGPIPE'
Hi,
I used print_all.py
to print the output of a MRT file from RIPE RouteCollector rrc21 (http://data.ris.ripe.net/rrc21/2020.04/updates.20200422.0125.gz). It seems that IPv4 prefixes at the end of BGP message are getting decoded as IPv6. For example, following is an example taken from the output of print_all.py
:
MRT Header
Timestamp: 1587518739(2020-04-22 01:25:39)
Type: 16(BGP4MP)
Subtype: 4(BGP4MP_MESSAGE_AS4)
Length: 267
BGP4MP_MESSAGE_AS4
Peer AS Number: 42275
Local AS Number: 12654
Interface Index: 0
Address Family: 2(IPv6)
Peer IP Address: 2001:7f8:54::1:204
Local IP Address: 2001:7f8:54::1:99
BGP Message
Marker: -- ignored --
Length: 223
Type: 2(UPDATE)
Withdrawn Routes Length: 0
Total Path Attribute Length: 111
Path Attribute Flags/Type/Length: 0x40/1/1
ORIGIN: 0(IGP)
Path Attribute Flags/Type/Length: 0x40/2/30
AS_PATH
Path Segment Type: 2(AS_SEQUENCE)
Path Segment Length: 7
Path Segment Value: 42275 25091 57463 263444 6762 16509 14618
Path Attribute Flags/Type/Length: 0x40/3/4
NEXT_HOP: 85.208.68.1
Path Attribute Flags/Type/Length: 0xc0/8/64
COMMUNITY: 0:8304 0:49981 16:1134 6762:1 6762:31 6762:40 6762:10150 24115:57463 25091:22414 25091:24115 64700:57463 64704:1300 65400:0 65400:65400 65444:102 65444:2000
NLRI: 305::/20
NLRI: 305::/24
NLRI: 305:1000::/21
NLRI: 305:1000::/24
NLRI: 350::/12
NLRI: 3e0::/12
NLRI: 3d0::/12
NLRI: 305:400::/24
NLRI: 305:900::/24
NLRI: 305:1100::/24
NLRI: 305:1200::/24
NLRI: 305:a00::/24
NLRI: 305:b00::/24
NLRI: 305:700::/24
NLRI: 305:1300::/24
NLRI: 305:200::/24
NLRI: 305:600::/24
NLRI: 305:300::/24
NLRI: 305:1500::/24
NLRI: 305:500::/24
NLRI: 305:800::/24
NLRI: 305:100::/24
NLRI: 305:1400::/24
It seems that the parser is using the AFI (IPv6) provided in the BGP4MP_MESSAGE header to decode the prefixes at the end of the BGP message whereas RFC 6396 under 4.4.2 states that:
The Address Family value only applies to the IP addresses contained
in the MRT header. The BGP4MP_MESSAGE Subtype is otherwise
transparent to the contents of the actual message that may contain
any valid AFI/SAFI values. Only one BGP message SHALL be encoded in
the BGP4MP_MESSAGE Subtype.
used Version: 1.4
used mrt files: samples/quagga_bgp, one from RIPE RIS (updates dump), one from routeviews.org (updates dump)
observed behaviour:
without any flag (config format): prints the config header and footer but no static routes
with (-A|-G) flag (api (grouping) format): prints nothing (and doesn't finish by itself; strg+c at 0% CPU over long time->other issue)
my guess:
It seems like mrt2exabgp.py can't extract routes from the BGP4MP_MESSAGE type.
Regards,
Daniel
I must be doing something wrong, I'm using the latest git checkout.
I am trying to pull out all the Cogent routing information for AS174...
wget http://data.ris.ripe.net/rrc00/2020.11/bview.20201101.0000.gz
examples/mrt2exabgp.py -p 174 -s -A bview.20201111.0000.gz
The error I'm getting is this...
$ examples/mrt2exabgp.py -p 174 -s -A bview.20201111.0000.gz
Traceback (most recent call last):
File "examples/mrt2exabgp.py", line 514, in <module>
main()
File "examples/mrt2exabgp.py", line 511, in main
conv_format(args, flags, d)
File "examples/mrt2exabgp.py", line 185, in conv_format
print_route_td(args, params, m.data)
File "examples/mrt2exabgp.py", line 250, in print_route_td
line += get_bgp_attr(args, params, m, attr)
File "examples/mrt2exabgp.py", line 397, in get_bgp_attr
path_seg = attr['as_path'][0]
KeyError: 'as_path'
Any help would be greatly appreciated!
In https://github.com/t2mune/mrtparse/blob/master/examples/mrt2bgpdump.py
I believe there is an issue on line 285:
seg_t = list(seg['type'])[0]
for seg in attr['value']:
if seg_t == AS_PATH_SEG_T['AS_SET']:
I believe it should be
for seg in attr['value']:
seg_t = list(seg['type'])[0]
if seg_t == AS_PATH_SEG_T['AS_SET']:
As in line 242 since seg does not exist prior.
Hi there,
In the init.py file there's a typo on line 930 when path_seg['len']
is called. I believe it should be path_seg['length']
instead, as there is no attribute len
in a BGP message.
The "announce attribute ..." syntax of the API grouping format was changed a little. It has to be "...attributes..." with s now in the current version.
Regards,
Daniel
Hey!
It would be nice to be able to do "pip install mrtparse".
hi sir, when I run this code:$ cd " clone directory "
the ubunto said = syntax error near unexpected token `newline'
what's the problem? ' i have done all of the step i mean download that code.
I'm running print_all.py for a Route Views update file and am getting the error below
File "/Users/rodrigodalri/Downloads/BGPstability/mrtparse/init.py", line 454, in chk_buf
if len(self.buf) - self.p < n:
TypeError: object of type 'NoneType' has no len()
Any idea what that might be?
Hello Dear All,
When I run below command, I am receiving error.
Could you please help me?
[root@centos examples]# time python3 mrt2exabgp.py -G -P latest-bview.gz > fullbgptable.py
^CTraceback (most recent call last):
File "mrt2exabgp.py", line 514, in
main()
File "mrt2exabgp.py", line 511, in main
conv_format(args, flags, d)
File "mrt2exabgp.py", line 179, in conv_format
for m in d:
File "/usr/local/lib/python3.6/site-packages/mrtparse/init.py", line 94, in next
self.unpack_msg(mrt)
File "/usr/local/lib/python3.6/site-packages/mrtparse/init.py", line 138, in unpack_msg
self.unpack_td_v2(buf, mrt)
File "/usr/local/lib/python3.6/site-packages/mrtparse/init.py", line 185, in unpack_td_v2
rib.unpack()
File "/usr/local/lib/python3.6/site-packages/mrtparse/init.py", line 381, in unpack
self.p += entry.unpack()
File "/usr/local/lib/python3.6/site-packages/mrtparse/init.py", line 410, in unpack
self.p += attr.unpack()
File "/usr/local/lib/python3.6/site-packages/mrtparse/init.py", line 703, in unpack
self.unpack_next_hop()
File "/usr/local/lib/python3.6/site-packages/mrtparse/init.py", line 766, in unpack_next_hop
self.data['value'] = self.val_addr(AFI_T['IPv4'])
File "/usr/local/lib/python3.6/site-packages/mrtparse/base.py", line 301, in val_addr
addr = socket.inet_ntop(_af, buf + b'\x00'*(plen_max // 8 - n))
KeyboardInterrupt
real 0m5.786s
user 0m5.513s
sys 0m0.058s
[root@centos examples]#
you seem to create the function val_dict
only for the reason to return a default if the value is not in the dict. Without the default value the following lines are equivalent:
>>> MSG_ST[9][4]
'BGP_SYNC'
>>> val_dict(MSG_ST, 9, 4)
'BGP_SYNC'
This is a bit unpythonic and the actual meaning of the function is a bit hided. There are two better solutions for val_dict
:
First assign the default value direct on accessing content. This has the advantage that the user see that it returns a default on Key Error, but it is more to write and not as nice as MSG_ST[9][4]
:
>>> MSG_ST.get(9, dict()).get(4, 'unknown')
'BGP_SYNC'
>>> MSG_ST.get(9, dict()).get(90, 'unknown')
'unknown'
>>> MSG_ST.get(90, dict()).get(4, 'unknown')
'unknown'
If you want that behaviour in any case the best, nicest and cleanest solution would be to simple subclass the dict class:
>>> import collections
>>> BGP_ST = collections.defaultdict(lambda: "unknown", BGP_ST)
>>> BGP_ST[4]
'BGP_SYNC'
>>> BGP_ST[90]
'unknown'
for dicts which returns dicts you have to use:
>>> MSG_ST = collections.defaultdict(lambda: dict(), MSG_ST)
>>> MSG_ST[9]
{0: 'BGP_NULL', 1: 'BGP_UPDATE', 2: 'BGP_PREF_UPDATE', 3: 'BGP_STATE_CHANGE', 4: 'BGP_SYNC', 5: 'BGP_OPEN', 6: 'BGP_NOTIFY', 7: 'BGP_KEEPALIVE', 'BGP_KEEPALIVE': 7, 'BGP_NOTIFY': 6, 'BGP_NULL': 0, 'BGP_OPEN': 5, 'BGP_SYNC': 4, 'BGP_STATE_CHANGE': 3, 'BGP_UPDATE': 1, 'BGP_PREF_UPDATE': 2}
>>> MSG_ST[90]
{}
Of course the dics gets anyway obsolete with #5
Hello, Thank you very much for writing this very useful tool.
I noticed that exabgp_conf.py can be generate very large configuration files which will most often will then take ages to load. A faster way to do would be to send the routes using the API. The process has been documented by another ExaBGP user here: https://github.com/FastVPSEestiOu/fastnetmon/blob/master/docs/EXABGP_INTEGRATION.md
or more elegantly https://github.com/job/irrexplorer/blob/master/exabgp.conf
Also, I added an "undocumented" (i.e.: needing some improvement but working) command to the API allowing to reduce parsing by grouping updates with the same attributes. From memory it is something like "announce attribute ... route ... ". I would need to double check if this is of interest.
Good continuation.
I was trying to use mrtparse for exabgp in our lab and found that the prefix length key is set to just length
, where in the json dump the key is prefix_length. I can submit a PR if you like.
(.venv) [jhall@fooji bgp]$ pip list | grep mrt
mrtparse 2.1.0
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.