Comments (8)
Thanks, that's really helpful. Unfortunately the device we're targeting has a 4.14 kernel that doesn't support kernel-side filtering, but the custom parser is really useful.
Using this, I've written this version of get_addr()
:
import ipaddress
import pathlib
import socket
import struct
from typing import List, Union
import pyroute2
from pyroute2 import netlink
from pyroute2.netlink import rtnl
from pyroute2.netlink.rtnl import rtmsg
IFA_ADDRESS = 1
IFA_LOCAL = 2
def ifc_address_parser(family: int, index: int) -> 'netlink.nlmsg':
"""
Generate a netlink parser that returns IP addresses in the given family and
for the interface with the given index.
"""
@profile
def parser(data, offset, length):
length, type, flags, sequence_number = struct.unpack_from('IHHI', data, offset)
header=dict(
length=length, type=type, flags=flags, sequence_number=sequence_number, error=None
)
if type == netlink.NLMSG_DONE:
msg = netlink.nlmsg()
msg['header'] = header
msg.length = length
return msg
if type != rtnl.RTM_NEWADDR:
return None
ifc_index = struct.unpack_from("I", data, offset + 20)[0]
if ifc_index != index:
return None
ifa_family, ifa_prefixlen = struct.unpack_from("BB", data, offset + 16)
if family and ifa_family != family:
return None
cursor = offset + 24 # offset + sizeof(struct header) + sizeof(struct ifaddrmsg)
while cursor < offset + length:
nla_length, nla_type = struct.unpack_from('HH', data, cursor)
if nla_type in [IFA_ADDRESS, IFA_LOCAL]:
addr = data[cursor+4:cursor+nla_length]
msg = netlink.nlmsg()
msg['header'] = header
msg.length = length
if ifa_family == socket.AF_INET:
ipaddr = (int(b) for b in addr)
ipaddr = '.'.join(str(x) for x in ipaddr)
msg['ADDRESS'] = ipaddr
msg['PREFIXLEN'] = ifa_prefixlen
else:
ipaddr = ipaddress.IPv6Interface((addr, ifa_prefixlen))
msg["ADDRESS"] = ipaddr
return msg
nla_length = (nla_length + 3) & ~3
cursor += nla_length
return None
return parser
def get_addr(ipr: pyroute2.IPRoute, family: int, index: int) -> List[Union[ipaddress.IPv4Address, ipaddress.IPv6Address]]:
msg = rtmsg.rtmsg()
if family:
msg['family'] = family
msg['index'] = 10
addresses = ipr.nlm_request(
msg,
msg_type=rtnl.RTM_GETADDR,
msg_flags=netlink.NLM_F_DUMP | netlink.NLM_F_REQUEST,
parser=ifc_address_parser(family, index)
)
return [address["ADDRESS"] for address in addresses]
def interface_index(interface):
index_file = pathlib.Path("/sys/class/net") / interface / "ifindex"
if not index_file.exists():
return None
return int(index_file.read_text(encoding="utf-8"))
ipr = pyroute2.IPRoute()
get_addr(ipr, socket.AF_INET, interface_index("wlo1"))
(delete the @profile
if you're not using line_profiler
). Performance here is a bit platform-dependent, but the speed-up compared to IPRoute.get_addr()
is about 60% on x86/Python3.10 and >80% on aarch64/Python3.6 (the target platform here).
Unfortunately I can't use line_profiler
on the target platform (or not easily, at any rate) but on x86, it's now spending more than 1/4 of its time in the constructor of netlink.nlmsg()
. I haven't had a chance to look into why that's an expensive operation, though my guess would be that dictionary constructors are part of it.
Another thing that would be really useful in this area is if IPDB.register_callback()
had an option not to parse messages but to return the raw message buffer, so that similar optimised parsing routines could be used. Would you be open to a PR that does this? (I know it seems an edge case; we have a wifi driver that sends channel utilisation statistics in RTM_NEWLINK
messages, one per second per radio and it gets expensive - yes, it'd be nice to fix the driver but they're qualcomm and we're not).
from pyroute2.
Having spent today thinking about it, I've decided to drop IPDB and roll my own netlink socket code to listen for interface up/down events and parse out the bare minimum I need to identify the interface and its state. I don't think our use case is general enough to put a requirement on something like pyroute2.
from pyroute2.
Hello,
get_addr
will run a python filtering on all configured addresses, so it can be slow. Especially if you have a lot of addresses on the system.
If you want to use kernel side filtering, you can do something like this :
In [2]: ipr = pyroute2.IPRoute()
In [3]: ipr.get_addr(index=1)[0].get_attr("IFA_ADDRESS")
Out[3]: '127.0.0.1'
In [4]: %timeit ipr.get_addr(index=1)[0].get_attr("IFA_ADDRESS")
276 ms ± 11.5 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
In [15]: ipr = pyroute2.IPRoute(strict_check=True)
In [16]: %timeit ipr.addr("dump", index=1, family=socket.AF_INET)
977 µs ± 76.6 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
In [17]: ipr.addr("dump", index=1, family=socket.AF_INET)[0].get_attr("IFA_ADDRESS")
Out[17]: '127.0.0.1'
WARNING: strict_check is mandatory here. Without it, the kernel will ignore given filters.
from pyroute2.
Additionally, the latest versions support custom parsers, so you can avoid using the generic stuff that recursively decode NLA messages, and write a simple routine that takes only the data you need, as it's done here:
pyroute2/pyroute2/iproute/linux.py
Lines 755 to 760 in e96f3ad
pyroute2/pyroute2/iproute/parsers.py
Lines 34 to 46 in e96f3ad
from pyroute2.
Thanks for the testcase, @tomkcook ! Maybe I can run performance measurements on a platform like Raspberry PI, I believe it should be comparable.
But I have to tell that the library was designed to be a universal netlink solution, thus there are not so many RTNL-specific optimizations :) yet, so little by little we fix that.
Regarding IPDB -- could you pls tell me the reasons why NDB is not an option? Whatever NDB is missing, I think that adding this functionality will be for me a much more easy task than fixing IPDB, which is broken by design. IPDB is excluded now from the CI pipeline, and I planned to drop it one day.
from pyroute2.
I don't recall the thinking that went into using IPDB here, though it was admittedly probably mostly me that wrote the code. We're working on a fairly constrained embedded platform and I think NDB looked too big and complex. We only use IPDB to listen for events, to detect when interfaces go up and down.
And just to give you a warning, I left a system running with the above version of get_addr()
over the weekend and this morning it was locked hard and had to be power cycled. I haven't looked into it, but it seems likely it's leaking memory somewhere.
from pyroute2.
Thanks for the info!
from pyroute2.
@tomkcook could I please ask you to spend a time and write down the requirements to a possible IPDB replacement in your case? Like what are you using right now, multi/single threaded, constraints etc?
If a simple drop-in replacement could work for you, I would rather provide it, so an occasional drop of IPDB will not affect your project.
PS: regarding memory leaks -- tracing the code.
from pyroute2.
Related Issues (20)
- add_port, Operation not permitted HOT 1
- vxlan interface attrs missing unicast or multicast flag
- Unable to read the percentage options in Netem qdisc
- Socket is not closed in case of PermissionError
- DNS resolution inside network namespace HOT 1
- what about python 3.12 HOT 1
- Getting 'RecordSet' object has no attribute 'select_records' when a filter is applied on a dump of ndb interfaces view HOT 3
- Add rules to nftables
- Ethtool.get_strings_set() API inconsitent between Ioctl and Netlink variants HOT 1
- ethtool ioctl crashes if number of strings exceeds hardcoded value HOT 5
- NDB MPLS Not working HOT 2
- RFE: please restart making github releases HOT 3
- Using nftables in a netns HOT 1
- Can't change interface for the route using NDB
- NetNS does not handle fd parameters HOT 3
- Weird behaviour when calling del_ip with no arguments twice in a row
- Full route on with wireguard wont work
- Unable to use ndb when only installing that package on Ubuntu 22.04
- wireguard: missing native netns support (?)
- compat - object is not callable with ipdb_interfaces_view() HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pyroute2.