openli-nz / openli Goto Github PK

Allow users to query the current intercept config by sending GET requests to the update socket. This will be important for people to be able to properly integrate OpenLI into their web management systems.

Make sure it follows standard RESTful API principles.

Our current packages depend on rsyslog to provide logging capabilities, but some users prefer to use syslog-ng on their systems instead. Forcing rsyslog in those cases is not very pleasant for them.

Ideally, we want some sort of "either / or" dependency, as both logging systems should be more or less equivalent for what we use them for.

Alternatively, just not specify a direct dependency at all and just log based on whatever system is available -- not sure if we can do this though?

As seen with EUBA connections, the RADIUS packets can contain both an IPv4 and IPv6 address assignment so we effectively have a single session that should be generating two IRIs per RADIUS action (one for the IPv4 stream, one for the IPv6).

This will probably mean that we need to allow multiple "sessionips" per access session, and we will need to ensure that OpenLI generates appropriate IRIs for all sessionips when a suitable RADIUS state change occurs.

Another related problem is that IPv6 delegations are not individual addresses; they are prefixes, so our current ipv6_comm_contents() method probably won't that well. Static IPv6 prefixes should still work fine, but a dynamic one from RADIUS is going to also require a patricia tree lookup rather than a straight hash lookup like we do at the moment.

Under certain circumstances, the sync thread may need to generate an IRI based on some external change in an intercept's properties (i.e. not in response to a RADIUS, GTP or some other session protocol packet).

The obvious example would be needing to generate an "end of intercept while session active" message when the end time for an intercept has been reached. This works fine in the general case, because all of the information needed to generate the IRI is stored with the sync thread's local structure describing the intercept. This includes the CIN, which is derived from fields in the session management protocol packets.

The problem case is vendor mirror intercepts, which can take place without any session management traffic being sent back to the sync thread. However, we need the CIN to be able to generate IRIs for those intercepts and the CIN is only present in the packets that are intercepted by the collector threads.

Therefore we need a message that the collector threads can pass up to the sync thread to state the known CIN for a vendmirror intercept. The sync thread can then use that CIN (if available) to populate an IRI if needed. If no CIN has been received from a collector thread, then we can assume that no packets have been intercepted and an IRI is probably not needed (?).

Hello,

Issue:

We encountered a bug with UDP segment (mostly INVITE message where we 've seen one message of 1800bytes, fragmented in to two IP datagram). This UDP segment isn't intercepted and isn't sent to mediator

Context

• Everything is in UDP (SIP, RTP)
• Some SIP client will send a big INVITE message and so it is broken mostly into two IP datagrams
• This message isn't intercepted

Analyze

After digging into the code we found out that the following code has some problems.

Firslty, the SIP message is correctly concatednated from multiple IP datagram into a buffer sip_message
And then the following code will call a function to try to parse the message:
https://github.com/wanduow/openli/blob/fb3f13937c7914dbb092bb63cf09cbcbb20f404c/src/collector/collector_sync_voip.c#L1575

If we look several lines above this one, we find that:
https://github.com/wanduow/openli/blob/fb3f13937c7914dbb092bb63cf09cbcbb20f404c/src/collector/collector_sync_voip.c#L1548-L1551
pktref = NULL

so in the parse_next_sip_message:
https://github.com/wanduow/openli/blob/0a7e614562c0096fa04a49fc50cbd75a327c7e6d/src/collector/sipparsing.c#L128-L142
The code to reassemble the tcp stream is called, though we don't need it because it it udp and it seems to me that the message is already concatenated into buffer p->sipmesage before hand.
Anyway, the function get_next_tcp_reassembled will return 0 because of the check at the beginning of this function.
So parse_next_sip_message will return 0 instead of parsing the concatenated fragment.

And the INVITE message is discarded just after that.

"Dirty" fix

If we disable the line for tcp reassembled:
https://github.com/wanduow/openli/blob/fb3f13937c7914dbb092bb63cf09cbcbb20f404c/src/collector/collector_sync_voip.c#L1548-L1551
It will work as attended for UDP stream, no more drops, but it isn't an elegant way and it doesn't support TCP then.

My initial plan was to disable this to avoid duplicating code relating to checking if various bits of configuration have changed, but there may be users relying on that feature for automating their intercept provisioning. It would be a bit rude therefore for me to just casually take that away from them because the new REST API is "better" -- I instead now think we should have a transition period where both ways of modifying running intercept config work, even if it is a pain to be looking after two code paths to do the same thing.

Make sure this is done before 1.0.4 is released!

LEAs that only respond to TCP connections and not ETSI keepalive messages, such that the provisoner configuration is:
keepalivefreq; 0
keepalivewait: 0
Lets call these one-way receiver agencies (1way)

A single warning is produced if the HI2 and HI3 are on the same host
OpenLI: warning, keep alive timer has been disabled for agency [IPADDRESS]:[PORT] this displays the HI2 port only - possibly drop the port, or produce one message per handover port

The Mediation functions correctly, however there is no logging of the state of the TCP connection beyond an inital state.

If a inital TCP connection is established with the 1way when the Mediator is started, this is logged:
OpenLI: mediator has connected to agency [AGENCY_NAME] on HI[2-3] [IPADDRESS]:[PORT].
Should this then disconnect this is also logged.
OpenLI: mediator is disconnecting from handover [IPADDRESS]:[PORT] HI[2-3]
Beyond this should the TCP connection be reestablished, no log messages are created.

If an inital TCP connection can not be established with the 1way when the mediator is started the following messages is generated:
OpenLI: Failed to connect to [IPADDRESS]:[PORT] -- Connection refused.
OpenLI: Will retry connection periodically.
No other messages are generated.
There is no on-going visibility as to the state of the connection with the 1way.

It would be beneficial to log TCP connect and disconnect messages for all handovers as the
OpenLI: reconnected to handover [IPADDRESS]:[PORT] HI[2-3] successfully message is not generated until the Keepalive message is sent to the LEA which waits keepalivefreq seconds

Hi,
We are checking how we can use this great software for both data and voip taps.
Is there support for receiving the traffic from a Nokia 7750 SR (former Alcatel Lucent SROS) ?
We can sent the traffic from a IP tap with the added header: layer-3-encap header-type ip-udp-shim
config below would use a source/dest ip and source/dest UDP port eg 601 in the config below: /configure mirror mirror-dest "601" admin-state enable
/configure mirror mirror-dest "601" encap { }
/configure mirror mirror-dest "601" encap { layer-3-encap }
/configure mirror mirror-dest "601" encap { layer-3-encap header-type ip-udp-shim }
/configure mirror mirror-dest "601" encap { layer-3-encap direction-bit true }
/configure mirror mirror-dest "601" encap { layer-3-encap router-instance "Base" }
/configure mirror mirror-dest "601" encap { layer-3-encap gateway }
/configure mirror mirror-dest "601" encap { layer-3-encap gateway ip-address }
/configure mirror mirror-dest "601" encap { layer-3-encap gateway ip-address source 10.10.10.1 }
/configure mirror mirror-dest "601" encap { layer-3-encap gateway ip-address destination 192.168.168.254 }
/configure mirror mirror-dest "601" encap { layer-3-encap gateway udp-port }
/configure mirror mirror-dest "601" encap { layer-3-encap gateway udp-port source 601 }
/configure mirror mirror-dest "601" encap { layer-3-encap gateway udp-port destination 601 }

The intercept itself will send also the additional details like intercept-id and the session-id:
/li li-source "601" admin-state enable
/li li-source "601" sap 1/1/c3/1:0 { }
/li li-source "601" sap 1/1/c3/1:0 { intercept-id 20210319 }
/li li-source "601" sap 1/1/c3/1:0 { session-id 1 }
/li li-source "601" sap 1/1/c3/1:0 { ingress true }
/li li-source "601" sap 1/1/c3/1:0 { egress true }

Thanx in advance for the help and support !
Niek van der Ven

An empty file will be treated as there being no intercepts configured, but will allow users to add intercepts etc via the REST API.

A missing file will likely be treated as an error and cause the provisioner to halt. To avoid confusion for new and upgraded installs, we should add an empty file to the default intercept config file location on package install (if no file exists).

Also as part of this, think about the best default location for the config file -- code currently has it set to a file in /var/lib/openli/, but all of our other config goes in /etc/openli/. We should probably try to be consistent with where we put config (?)

We currently re-parse the identity, authorization and URI fields in a SIP packet once for each active VOIP intercept. This is a bit unnecessary, as the values of these fields are always going to be constant within each packet.

Instead, we should store the value for each identity "type" (identity, authorization, to URI) for the lifetime of the SIP packet and only free them once we're done with parsing the packet. The first VOIP intercept that we try to match against will set and store the values, then each subsequent one will simply used the cached identity.

Not a big deal, but saves us a few CPU cycles in the VOIP sync thread once the number of active intercepts starts to go up.

Pretty self-explanatory. Init scripts will make starting, stopping and reloading config for the OpenLI components a lot easier, especially when they are being run as a daemon.

Software is complicated enough now that we really need a proper set of tests that we can run regularly to make sure that a change in one area hasn't broken anything that we already have working.

Because OpenLI is a multi-component system, this is not as simple as just writing a few quick unit tests -- we also need to test that the whole end-to-end process works as intended (e.g. intercepts specified by the provisioner do not over or under collect at the collector, all intercepted records reach the mediator, components respond properly to restarts and config changes etc).

It's a big job and probably not something we can expect to do without some of that extra "polish" funding or some enthusiastic volunteers, but it will definitely make life easier going forward (especially if once we do a public release and start getting contributions from people other than myself).

Keep alives are currently appended to the end of the buffer of data to send to the handover.

In the situation where we have a agency that has been absent for some time and therefore is having records queued up for it on the mediator, this leads to some keep alive failures.

If the keep alive response timeout is too low, then the keep alive may not even be sent to the agency before the timer expires, causing the handover to be dropped while clearing the backlog. The handover will likely reconnect and carry on, but it's not ideal and will likely cause concern for both users and agencies to see these connections continuing to drop.

Instead, we should avoid queuing a keep alive as long as there is data in the buffer to send via that handover, since keep alives should only be sent when the outgoing data connection is idle (as opposed to there being no incoming data for that handover coming from the collectors). We can instead restart the keep alive timer once the buffer is empty again.

When two SIP targets belonging to two different LIIDs call each other, RTP packets are missing. This behaviour appeared many times with different scenarios where both LIIDs have a call. The library is working correctly and all packets are present for one LIID being intercepted with a non targeted SIP user.
Is this related to the collector creating unique interception sessions or the provioner's config ?
Thank you for your support and for this awsome Library.

Realistically, we don't want to be editing and reloading config files every time we get a new intercept (or have to stop an ongoing one). A better solution would be a listening socket on the provisioner that allows an external process to "push" commands to the provisioner to add, modify or halt an intercept. This could also be used to change LEA details, locations of SIP servers, etc -- basically any major config that could change while OpenLI is running.

Obviously, the user would have to make sure that access to this listening socket is very restricted.

Eventually, this socket would be used by the web configuration interface to configure the provisioner in real-time.

The IRIs that we generate for ongoing RADIUS sessions whenever an intercept is started or stopped are missing some of the fields that we would normally include in an IPIRI (e.g. NAS port, NAS identifier, etc.).

This is because these properties are not stored with the session itself -- we read them directly out of the attribute list derived from a RADIUS packet. But when the generation of certain IRIs is triggered (i.e. a user starts an intercept for an active session), we don't have a specific packet to grab this data from; we only have a reference to the session.

Therefore, to fix this, we're going to need to start saving more information with each RADIUS session so that we can generate full IRIs from the session pointer alone.

Each input gets a set of "collocals" allocated for it within a global array, where the local variables for the libtrace processing threads for the inputs are stored.

As stated, collocals are currently stored as an array. The array is extended whenever we add new inputs or change existing inputs such that they need to be stopped and restarted. collocals for inputs that have been stopped are not freed until the collector process itself ends, which is a bit wasteful.

In theory, this means that if there is a lot of input churn for any reason then we will slowly be wasting an increasing amount of memory as this array grows and grows. Ideally, we wouldn't do that and should consider a better way of storing and managing our collocals.

Reasons why this is not a high priority issue:

• problem only occurs when reloading collector config AND inputs are being changed -- this will be very rare in most sane usage scenarios.
• amount of memory wasted is low, mainly due to the expected rarity of the situation where the array would be extended.
• restarting the collector will resolve the issue (but don't do this while you've got intercepts active!)

See reload_inputs() function in src/collector/collector.c (in develop branch for now, but will also be in master from 1.0.10 onwards) for more in-code doc notes on the issue.

This is especially true for the provisioner and mediator -- the collector may need root to be able to listen on the capture interface (but even then, we may only need those permissions temporarily).

Ideally, our packaged installs would create an "openli" user, install everything (binaries, config files, etc.) as belonging that user and have the systemd scripts run the components as "openli".

Unfortunately, because we didn't do this earlier, we'll also need some sort of script that will detect existing config etc. that is owned by root and make sure that ownership is changed to the "openli" user when the package is updated.

Currently, the socket just does plain HTTP which means all intercept updates and query results are easily visible on the wire.

Instead, we should enable HTTPS via libmicrohttpd (probably using the provisioner TLS cert?) and force all requests to come over HTTPS instead.

Note: this may end up forcing all of our users to start using TLS for everything -- that's not necessarily a bad thing, though :)

Situation:

• collector has been running for some time, user A is not an intercept target.
• RADIUS session for A begins after collector is started.
• Auth for RADIUS session for A is successful and A is assigned an address.
• Operator receives warrant for intercept of A's traffic, updates provisioner to include A as a target.
• Collector correctly starts intercepting traffic for A's IP.
• But the IRI-Begin for an already active session is not emitted.

We do have code in the RADIUS plugin to detect cases where a session was already active before the collector started -- this should be working fine, but it doesn't cover the above scenario.

Instead, when the sync thread is pushing existing user sessions down to the collector threads, we should also emit a similar IRI (as long as there is at least one existing user session to be pushed).

Currently, anyone can push intercept config updates or query the running intercept config, just by pushing the right looking requests to the update service socket.

While we're strongly encouraging operators to lock down access to the update service using firewall rules, etc., we should also provide standard authentication (e.g. user + password) methods to add additional security.

The current implementation of emitting new config after an update from the REST API will immediately start trying to write over the top of the existing config file.

This means that if something goes wrong during the emission process, we end up with an empty config file. All of the previous config is lost forever.

What we should do instead is write the new config to a temporary file. Once the new config is completely written successfully, then we can overwrite the old config file.

BER is a less strict method than DER, which lets us make certain optimizations w.r.t. pre-encoding portions of the ETSI output.

Especially, BER allows us to encode lengths using any number of bytes rather than being forced to use the most efficient encoding -- we can always use 4 bytes to encode a sequence number, instead of having to encode seqno < 128 using 1 byte, etc. This means we can pre-construct the entire PS header with fixed-size spaces available for seqno, timestamp that get filled in at encoding time. At encode time, we can just copy the entire pre-encoded header into a buffer to send, then fill in the correct values for the missing fields, which should be way faster than the inside-out encoding that we do right now for DER.

Things to consider:

• Can agencies handle BER-encoded records or do we need to still have DER available as an encoding option? Definitely check with NZP at least before we put too much time into this.
• If DER is still required, how do we design an API that supports both encoding methods?
• CC Payload sequences are a bit trickier -- every sequence within the Payload object will need its length set to account for the size of the saved payload. On the plus side, there is no variable-length intercept-specific content in the payload sequences, so the offsets for each of the length fields are likely to be in the same place.
• PS Headers can be pre-encoded in the seqtracker thread and pushed to the encoder threads to be filled in.
• IRI Payload probably won't have too much worth pre-encoding but hopefully we aren't generating these records at high rates anyway.

At present, all OpenLI messages have no encryption or validation that they are coming from a trusted source. This means that a) anyone who can get between two OpenLI components can easily see what OpenLI is up to and b) anyone who can figure out the (publicly defined) OpenLI protocol and talk to an OpenLI component can issue intercept instructions, including cancelling active intercepts).

What we really want is something similar to the AMP SSL cert + keys implementation, whereby each deployment has its own certs and keys which are used to encrypt all communications. External parties don't have the right keys, so can't talk to any of the genuine components and can't figure out what they are saying either.

Urgency will depend on what NCSC thinks of our current communication model -- in theory, operators should be able to isolate OpenLI such that nobody external should be able to see or interfere with the internal comms, but I can understand why someone might object to our present behaviour.

Hello,

Firstly, We 'd like to thank you for this awesome project.
We had encountered a weird issue with SIP client behind NAT.

Issue:

We found that there is no RTP stream captured and sent to mediator

Context

• All SIP clients (Linphone application on iOS/Android) is behind home router, work office or mobile operator's gateway and has private IP address.
• The SIP server is in public, no nat. And the OpenLI is setup to listen on a dummy interface which has mirrored traffic from SIP servers.

Analyze

After digging into the code we found out that the matching of RTP stream with IP source/destination has some problems.

Firstly, the RTP stream is associated with one or two address IP (targetaddr and otheraddr in code), corresponding to two endpoints/peers of RTP stream.

The address is taken from get_sip_media_ipaddr, e.g:
https://github.com/wanduow/openli/blob/fb3f13937c7914dbb092bb63cf09cbcbb20f404c/src/collector/collector_sync_voip.c#L602-L607

When I read the code of get_sip_media_ipaddr, the ip string is taken from "c" field in SIP/SDP protocol, so it could be a private address and then be used to associated to thisrtp stream.

And then later, when trying to find if a particular sock belong to the that particular rtp stream:
https://github.com/wanduow/openli/blob/8dc9dcb76a2cc09b1682dff4bb8e89ddd8ad915e/src/collector/ipmmcc.c#L92-L119

The ipa and ipb is taken directly from IP header of RTP packet, so they are public addresses
So there isn't any match because RTP stream 's target addr or other addr is private address. So the RTP packet is thrown away.

A "dirty" fix

If we tried to patch the line with get_sip_media_ipaddr which will return public IP, the problem is solved and we could see the RTP stream after mediator but we are not sure if this is the elegant way of doing so?

NZP are running into cases where people are testing OpenLI with them but not using the latest version of the code -- bugs and problems that we've already fixed are rearing their head again as a result.

To help with recognising these situations quickly, Neil suggested encoding the code version in the 'bogus' LIID that we write into the keepalive messages. At the moment, this LIID is "openlikeepalive", but we should change it to be "openliVXXXXXX", where XXXXXX is either the stable release version (easy) or the most recent git commit (harder?). Ideally, we'd fit both in there somehow but there probably aren't enough characters available.

Right now there's just Debian stuff, would be great to have RPM bits.

I intend to provide a PR with specs and gitlab-ci things to build them.

When we start provisioner, we get error as screenshot in below. We wrote collector address to clientaddr and port field. What we should write to updateaddr and port ?

provisioner-config-yaml file

clientaddr: 192.168.80.144
clientport: 9001

mediationaddr: 192.168.80.146
mediationport: 12001

updateaddr: 192.168.80.145
updateport: 8992

Where inactive == no accounting updates for 15 minutes (by default).

If our RADIUS capture interface experiences loss then we run the risk of missing the status updates that we would normally use to recognise the session as over. This means the session ends up sticking around forever, slowly gobbling up more and more memory.

This has been observed in production where the RADIUS interface is also used by other chatty services -- BPF filters on the interface can help reduce the impact, but I think we still need this as a backstop just in case.

Note: we need to be reasonably smart about how we check for session expiry -- we don't want to be inefficient and waste time looking at a bunch of unexpired sessions, as this is more likely to cause further packet loss because we're doing that instead of reading packets.

The REST API is going to be pretty useless if we don't provide documentation on how to use it.

I've started a wiki page: https://github.com/wanduow/openli/wiki/Intercept-Configuration-REST-API

There's no reason not to have it, and it can be handy for users who, for example, just want to use an interface for RADIUS input but happen to have other traffic on there as well that they'd like to automatically ignore.

If a mediator->agency session remains down for a very long time, it is possible that the mediator will use all of the host's memory to buffer intercept records. Running out of memory is bad -- the host (and therefore the mediator process) will struggle to recover, even if the session comes back. Also, we won't be able to buffer any more records in the mean time so we're still going to drop stuff anyway.

Ideally, we should try and write some of the buffered records to disk if they start using a lot of memory. This will keep the mediator from wrecking the host and will reduce the likelihood of dropped records (or at least buy some more time before we do have to drop something).

Jmirror is a special encapsulation format used by Juniper E and MX devices for mirroring traffic based on a particular set of criteria (such as a RADIUS username). This is an easy method for our Juniper users to get their devices to mirror an intercept target's traffic into an OpenLI collector, but we'll need to strip the jmirror headers and replace them with a proper ETSI encapsulation.

This will be almost exactly the same as what we already support for ALU mirroring, just with a slightly different header format.

Relevant docs:

• https://wiki.wireshark.org/jmirror
• https://www.juniper.net/documentation/en_US/junos/topics/concept/subscriber-management-subscriber-secure-policy-radius-header.html

Make sure we acquire some useful testing examples and include a test in our test suite for this format.

Traditional hashing methods don't work well when distributing RADIUS traffic, as they tend to spread RADIUS state for the same users across multiple threads.

For OpenLI, this can be a problem as we can have RADIUS updates for the same user that are close together in time reach the sync thread in a non-deterministic order (i.e. Access-Request being seen by sync after an immediately subsequent Account-Request-Start, because the Account-Request-Start 5-tuple is different to the Access-Request one). While this doesn't really affect our ability to track user->IP mappings, it does mean that our IRIs are not strictly correct given the packets that we receive.

The current workaround is to just ensure that users only configure any stand-alone RADIUS input interfaces to use one processing thread only, but this means that a) we have a pretty hard limit on our RADIUS processing performance and b) this only really works if the RADIUS is going to be delivered via its own distinct interface.

Not quite sure how to go about resolving this (probably some sort of RADIUS-friendly software hashing method needs to be added to libtrace?), but I am noting all of this down here so that we don't forget that this problem exists and wonder why our RADIUS IRIs are inconsistently ordered compared with the input traffic.

Pretty self-explanatory -- there's a TODO in the code that even explicitly mentions that this capability is missing and should be implemented.

The provisioner should recognise that an intercept will need to be started or stopped at some future time and create a suitable timer event to trigger the appropriate action at the right time.

For now, the start and end time should be assumed to be present in the config file. Default start time will be now and a default end time will be "until further notice".

Running our test suite on Centos 8 seems to generate failures with RabbitMQ -- SSL primarily, although I'm not convinced plain is entirely reliable either right now.

Hard to debug from packages, so will need to throw the openli source onto a centos8 testing container and see if we can figure out exactly what the problem is.

Tests passed fine on a Debian buster container, so may be RHEL specific (?)

A basic example would be the national-HI1-ASN1Parameters in HI1 operations messages.

Each country is going to have different possible structures that may need to be included in here and we cannot realistically hope to support everyone right away. Of course, nobody appears to publicly document what their specific country's agencies expect to be in here, so that makes it even harder.

Ideally, we would implement this as some sort of runtime loadable plugin system, but we cannot necessarily guarantee that all of the information required will be available when constructing a message. Some fields may be static, which is easier, or they may be derived from the properties of an intercept, agency, etc.

All in all, this could be a very tricky problem and is going to require a lot of thought.

While I'm developing the software, it's quite handy to have log messages that contain the name of the user, SIP URI, IP address etc of the targets just for debugging purposes.

Once we deploy though, we can't be recording any info about who or what is being intercepted -- that's going to get us in some amount of trouble.

Therefore, once I'm pretty confident the provisioner is doing its job and the other components are acting on its commands correctly, I'll have to go back and re-evaluate every log message to make sure we're only reporting non-identifiable information (LIIDs are probably OK?).

Running the provisioner, collector and mediator using valgrind --leak-check=full should reveal any memory leaks that we currently have.

I did do this with the mediator relatively recently, but this should be done regularly as code changes could introduce new memory leaks.

I have a question regarding the target detection of the caller. As far as I understand, openli does not use the "From:" header, because it can be spoofed, but instead uses the (Proxy-)Authorization header. However, this header does not necessarily contain the phone number. But the LEAs supply the targets as phone numbers, not as sip users (here in germany).

Therefore it would be very nice if not just the (Proxy-)Authorization header were used to detect target callers, but also the headers "P-Asserted-Identity" and "Remote-Party-ID", which usually contain the phone number. Would it be possible for you to implement this?

Currently, all intercept->agency mappings are pushed to each mediator. However, in a multi-mediator deployment, each intercept will likely only be pushed to one of the mediators. In that case, we should probably avoid pushing intercept->agency mappings to mediators that are never going to see records for those intercepts.

Still having problems with HI1 Operations messages that are not able to be properly parsed by recipients.

Let's try to make sure our message matches the example given below as much as possible, and try to push ahead with some testing with an agency as soon as we can.

An example expected message from NZP:

PS-PDU SEQUENCE
{
  pSHeader PSHeader
  {
    li-psDomainId OBJECT IDENTIFIER = {itu-t(0) identified-organization(4) 0 2 2 5 1 14}
    lawfulInterceptionIdentifier LawfulInterceptionIdentifier = 31.33.34.33.30.39.31.30.30.34 [1343091004]
    authorizationCountryCode PrintableString = "NZ"
    communicationIdentifier CommunicationIdentifier
    {
      networkIdentifier NetworkIdentifier
      {
        operatorIdentifier OCTET STRING = 31.32.39.37.37 [12977]
      }
      deliveryCountryCode PrintableString = "NZ"
    }
    sequenceNumber INTEGER = 0
    timeStamp GeneralizedTime = "20200924011407.086Z"
    timeStampQualifier TimeStampQualifier = timeOfMediation
  }  
  payload Payload  {    
    hI1-Operation HI1-Operation    
    {      
      liActivated Notification      
      {        
        domainID OBJECT IDENTIFIER = {itu-t(0) identified-organization(4) 0 2 2 0 1 6}        
        lawfulInterceptionIdentifier LawfulInterceptionIdentifier = 31.33.34.33.30.39.31.30.30.34 [1343091004]        
        communicationIdentifier CommunicationIdentifier        
        {          
          network-Identifier Network-Identifier          
          {            
            operator-Identifier OCTET STRING = 31.32.39.37.37 [12977]          
          }
        }
        timeStamp TimeStamp
        {
          localTime LocalTimeStamp
          {
            generalizedTime GeneralizedTime = "20200924011407.086Z"
            winterSummerIndication ENUMERATED = notProvided
          }
        }
        national-HI1-ASN1parameters National-HI1-ASN1parameters
        {
          domainID OBJECT IDENTIFIER = {itu-t(0) identified-organization(4) 0 2 2 0 1 6}
          countryCode PrintableString = "NZ"
        }
      }
    }
  }
}

For some ISPs, RADIUS username is not a useful identifier. Instead, the customer CPE just uses a default set of credentials and the real auth happens based on the Calling Station ID that has been injected by the fibre provider.

Therefore we need to allow CSID to be used as an alternative method of inferring target identity from RADIUS packets -- this would likely have to be mutually exclusive from username, and we'll need some high quality documentation to explain which options should be used under various circumstances.

Hello @salcock ,

We had encountered an issue with support for multiple media stream in OpenLI

Issue:

We found that if one call has multiple media streams (audio/video) there is only one of the two stream which will be captured

Context

• In the case of video call, if we observed the SIP/SDP requests, we will normally find two media section in session section of INVITE Request

• The SIP/SDP RFC said that we can have zero or more media section.

• In the current state of the art, only the first media is captured (normally the audio stream)

Analyze

After digging into the code we found out that the code has supported only one stream at the moment.

Traceback:

• every call to get_sip_media_port in
  https://github.com/wanduow/openli/blob/fe28a82c12297b97ea1e935581c6e7ac31cfff95/src/collector/collector_sync_voip.c#L571

• This function has the definition below
  https://github.com/wanduow/openli/blob/fe28a82c12297b97ea1e935581c6e7ac31cfff95/src/collector/sipparsing.c#L741-L751

• This issue came from this line in the above code:
  https://github.com/wanduow/openli/blob/fe28a82c12297b97ea1e935581c6e7ac31cfff95/src/collector/sipparsing.c#L749

• It seems to me that the second parameter indicates the media number, based on

char* sdp_message_m_port_get( sdp_message_t * sdp, int pos_media )

Get the port number ('m' field) of a SDP packet.

Parameters:
sdp The element to work on.
pos_media The line number.

And as we take only the zero-th index, so it is normal that only the first RTP stream which will be captured.

A possible fix

We are currently trying to see if we could make a loop around get_sip_media_port with correct media number, and if there are any regression bug

Btw, I see that the SIP function sdp_message_c_addr_get called by get_sip_media_ipaddr also has media position as parameter, so I'm not sure if there are another corner cases where we can have multiple IP addr and multiple media stream for each?

The forwarding thread is now our biggest bottleneck, so we need to make sure that the user can gain some benefit by adding more of them.

Currently, we have a config option but it doesn't do anything useful -- every encoded record goes to the first thread ;)

Had a few requests now for something like this.

The main use case seems to be to allow RADIUS collection to occur on a separate collector to the IP collection -- any sessions detected by the sync thread would also be pushed back to the collector for re-distribution to any other collectors (along with the appropriate CIN etc), so they can intercept any relevant IP traffic.

This will give ISPs a bit more flexibility in how they deploy OpenLI, i.e. not having to ship RADIUS traffic to every collector.

Some code assumes that an IP intercept will be specified with either a RADIUS username or a vendor mirror ID, but not both. This is incorrect -- ISPs using a vendor mirror will still want to use RADIUS to derive the IRIs for HI2 where possible.

This is likely to be a particular problem when detecting config changes after a reload, but there are almost certainly other places where I made this flawed assumption which need to be cleared up as well.

Pretty self-explanatory.

Problem is probably in how we are configuring rsyslog -- what works for Debian probably doesn't work for the RHEL-based systems...

Users have asked for more visibility into whether the collector is receiving, intercepting and/or dropping packets.

A simple solution would be to add a config option allowing users to specify a frequency at which these sorts of stats are logged (defaulting to 0, which would NOT log at all). Probably want to have this configured in terms of minutes, so that users don't accidentally fill their disk with stat log messages by setting this to be too frequent.

Possibly useful stats:

• packets captured
• packets intercepted
• packets forwarded to sync threads
• IP CCs and IRIs created
• IPMM CCs and IRIs created
• packets dropped by capture process
• packets discarded due to errors, parsing failures?