Missing Flow Duration or Flow End Time about goflow

[debugloop]

I can see why some cleanup is desirable, and in the end it does not make to much of a difference whether it is some or most fields in the protobuf. I figure I'll have to drain all topics (enriched or not) and upgrade all tooling in the maximum retention period interval (only 30mins right now) to avoid any Consumers misinterpreting conflicting fields. However, I'll wait with adopting until I have your ok to go ahead, I'd like to avoid doing it twice :)

My issue with the Direction is that it is part of my enrichment process, as it was part of the lookup of static Flow Exporter/Peer interface information base. These days however, I've integrated SNMP and BPG, and the only real use for this lookup is the Direction, which could be obtained freely using Netflow itself. So yes, I'd appreciate having it in.

We're using either Cisco ASR9k or NCS540 routers, all running some version of IOS-XR (mostly 6.4 or 6.5). The Direction field template says it is 1 Byte long. I think with that goflow will support any meaningful field exported by IOS-XR, see the template straight from Wireshark:

User Datagram Protocol, Src Port: 10090, Dst Port: 2055
Cisco NetFlow/IPFIX
    Version: 9
    Count: 1
    SysUptime: 2542165.834000000 seconds
    Timestamp: Apr 24, 2019 14:00:56.000000000 CEST
        CurrentSecs: 1556107256
    FlowSequence: 1924740437
    SourceId: 2177
    FlowSet 1 [id=0] (Data Template): 260
        FlowSet Id: Data Template (V9) (0)
        FlowSet Length: 92
        Template (Id = 260, Count = 21)
            Template Id: 260
            Field Count: 21
            Field (1/21): PKTS
                Type: PKTS (2)
                Length: 4
            Field (2/21): BYTES
                Type: BYTES (1)
                Length: 4
            Field (3/21): IP_SRC_ADDR
                Type: IP_SRC_ADDR (8)
                Length: 4
            Field (4/21): IP_DST_ADDR
                Type: IP_DST_ADDR (12)
                Length: 4
            Field (5/21): INPUT_SNMP
                Type: INPUT_SNMP (10)
                Length: 4
            Field (6/21): OUTPUT_SNMP
                Type: OUTPUT_SNMP (14)
                Length: 4
            Field (7/21): LAST_SWITCHED
                Type: LAST_SWITCHED (21)
                Length: 4
            Field (8/21): FIRST_SWITCHED
                Type: FIRST_SWITCHED (22)
                Length: 4
            Field (9/21): L4_SRC_PORT
                Type: L4_SRC_PORT (7)
                Length: 2
            Field (10/21): L4_DST_PORT
                Type: L4_DST_PORT (11)
                Length: 2
            Field (11/21): SRC_AS
                Type: SRC_AS (16)
                Length: 4
            Field (12/21): DST_AS
                Type: DST_AS (17)
                Length: 4
            Field (13/21): BGP_NEXT_HOP
                Type: BGP_NEXT_HOP (18)
                Length: 4
            Field (14/21): SRC_MASK
                Type: SRC_MASK (9)
                Length: 1
            Field (15/21): DST_MASK
                Type: DST_MASK (13)
                Length: 1
            Field (16/21): PROTOCOL
                Type: PROTOCOL (4)
                Length: 1
            Field (17/21): TCP_FLAGS
                Type: TCP_FLAGS (6)
                Length: 1
            Field (18/21): IP_TOS
                Type: IP_TOS (5)
                Length: 1
            Field (19/21): DIRECTION
                Type: DIRECTION (61)
                Length: 1
            Field (20/21): FORWARDING_STATUS
                Type: FORWARDING_STATUS (89)
                Length: 1
            Field (21/21): FLOW_SAMPLER_ID
                Type: FLOW_SAMPLER_ID (48)
                Length: 2

[lspgn]

I think this was to minimize the delay with the flow is accounted originally but we never used more than a single time (and sFlow does not have this information).

I'd rather add TimeFlowStart and TimeFlowEnd for compatibility reasons and eventually removing TimeFlow.

[debugloop]

I probably won't use both times either, but I want to give customers the option of reproducing plain Netflow from their Kafka topics I aggregate for them by their IP prefixes, so they can continue to use their old tools. Thus, I am in need for some value to put into the first switched field.

Thanks for your input, I'll tackle this tomorrow then.

[lspgn]

@debugloop Could you test the new v3? I integrated your changes.
https://github.com/cloudflare/goflow/releases/tag/v3.0.0.0

[debugloop]

@lspgn I can not easily test this right now, as you changed the protobuf definition (link to diff) heavily, and the rest of my tooling uses it too. Specifically, I've taken the old format and added some fields in the back I use for enrichment (geolocations, AS Paths, Interface/Peer names, ...).

I can see why you're changing some of the fields, but some strike me as unnecessary (switching id 3 and 4, ...) and I do not understand the large id skips in between (it is just colliding with my custom fields, I've started at 90) . I'd appreciate the introduction of a "reserved" range for custom fields, just a note in the docs or a comment maybe. At first I thought that you switched over to using the officiall IANA IPFIX ids, but that is not the case.

If you confirm that this'll be the version making it to the final, stable release I'll get started on figuring out how to introduce this with the least impact on my experimental setup and my colleagues (which are the only "users", so it is not too much trouble at this stage).

[debugloop]

Something else I've just noticed: I'd appreciate if you added the NFv9 Direction. The reason I'm writing this here instead of just doing it in another PR is that it might fit here 569608b#diff-9258e25e261124ae551eeb1b6b5d1d55R278 (as each message contains whether its IPFIX or v9, so one can discern)

I am not clear on the sematics of biFlowDirection though, so this might be a bad idea after all.

[lspgn]

Thank you for coming back to me!

You are raising good points.
I mostly wanted a cleanup (rename, order) as also our internal tools will also be colliding with new fields anyway.
The idea behind the skips of IDs was to group fields and allow new ones to be added below (eg: layer 4, hardware...).
I would not switch to IANA fields as it is IPFIX specific.

I will rebuild a protobuf with the fields in the correct order and will indicate the expected reserved field. It makes more sense. But that will still require re-ordering on most of people's implementation.

Regarding your last comment: biFlowDirection was another request. I can add flowDirection (id 61) which is present in both NetFlow and IPFIX.
I have never seen it used though: do you know how many bytes is this field in NetFlow? Which router/switch are you using?

[lspgn]

Update of the protobuf:
40bba56
Flow direction:
22783e3

[lspgn]

Just merged into v2 / master. Closing this ticket.