qmsk / snmpbot Goto Github PK

View Code? Open in Web Editor NEW

52.0 52.0 13.0 438 KB

Golang SNMP library + SNMP REST API

License: Mozilla Public License 2.0

Go 92.42% Python 7.09% Dockerfile 0.40% Shell 0.10%

golang json rest-api snmp

snmpbot's People

Contributors

Stargazers

Watchers

Forkers

fuyuande arolypoly nh-live pierresantana annttu ya-jeks doytsujin danlan027 arturbrasil ashmanpan sam80180 ztaozhang iq-scm

snmpbot's Issues

Polling broken when SNMP Request-ID reaches INT_MAX

It seems upon reaching INT_MAX, queries begin returning Error 500.

Log message:

Feb 18 02:14:33 monni snmpbot[28606]: WARN client: Engine<[::]:43176>: Unknown request [email protected]:161[-2147421757] recv

Referring to earlier IRC conversation, possible fix would be to apply modulo INT_MAX at

snmpbot/client/engine.go

Line 58 in 8ac389a

return engine.requestIDPool.atomicNext()

SNMP probing is broken for MIBs with objects under different top-level OIDs

Some common MIBs like SNMPv2-MIB and IF-MIB contain objects in multiple different OID trees. While the top-level OID for the SNMPv2-MIB itself is .1.3.6.1.6.3.1, most of the interesting objects are actually under the legacy .1.3.6.1.2.1.1 system group, and others under the 1.3.6.1.2.1.11 snmp groups: http://www.oidview.com/mibs/0/SNMPv2-MIB.html

Probing for the .1.3.6.1.6.3.1 OID will fail if the system only supports the older system objects, and does not support any of the newer objects defined under the MIB OID.

This affects Linux snmpd, and breaks API queries like GET /api/hosts/test/objects/?object=SNMPv2-MIB::* - although note that GET /api/hosts/test/objects/SNMPv2-MIB::sysDescr still works fine, since that isn't limited to probed MIBs.

Flaky tests: Log in goroutine after ... has completed

Seems like some version of go added panics for t.Logf(...) calls after test completion. This happens for our withTestClient -> go engine.Run() -> engine.teardown as triggerd by the defer engine.Close() not waiting for the engine to actually close.

panic: Log in goroutine after TestWalkBulk has completed

goroutine 120 [running]:
testing.(*common).logDepth(0xc00042c300, 0xc00009f020, 0x23, 0x3)
	/home/terom/opt/go-1.15/src/testing/testing.go:738 +0x59f
testing.(*common).log(...)
	/home/terom/opt/go-1.15/src/testing/testing.go:720
testing.(*common).Logf(0xc00042c300, 0xc00009eff0, 0x23, 0x0, 0x0, 0x0)
	/home/terom/opt/go-1.15/src/testing/testing.go:766 +0x7e
github.com/qmsk/go-logging.testLogger.Printf(...)
	/home/terom/go/pkg/mod/github.com/qmsk/[email protected]/testing.go:13
github.com/qmsk/go-logging.Logging.Debugf(...)
	/home/terom/go/pkg/mod/github.com/qmsk/[email protected]/logging.go:13
github.com/qmsk/go-logging.PrefixLogging.Debugf(...)
	/home/terom/go/pkg/mod/github.com/qmsk/[email protected]/prefix_logging.go:17
github.com/qmsk/snmpbot/client.(*Engine).teardown(0xc0004463c0)
	/home/terom/go/src/github.com/qmsk/snmpbot/client/engine.go:68 +0x21c
github.com/qmsk/snmpbot/client.(*Engine).run(0xc0004463c0, 0x69bc40, 0xc000098030)
	/home/terom/go/src/github.com/qmsk/snmpbot/client/engine.go:192 +0x2a9
github.com/qmsk/snmpbot/client.(*Engine).Run(0xc0004463c0, 0xc0002b22a0, 0xc0004463c0)
	/home/terom/go/src/github.com/qmsk/snmpbot/client/engine.go:212 +0x128
created by github.com/qmsk/snmpbot/client.withTestClient
	/home/terom/go/src/github.com/qmsk/snmpbot/client/client_test.go:37 +0x8e
FAIL	github.com/qmsk/snmpbot/client	0.168s

Table walking doesn't deal with sparse tables

The table walking ATM assumes that each GETNEXT request returns a complete row, i.e. each entry object in the response has the same index suffix. This assumption isn't necessarily valid, there can be "holes in tables" where some column objects are missing entries:

http://net-snmp.sourceforge.net/wiki/index.php/TUT:snmpgetnext#Multiple_Values

One thing to be aware of is that there may potentially be "holes" in the table, where a particular column does not have a value in every row. The GETNEXT request will step over these missing values, and return the value of that column from the next row where it does exist. The next iteration would need to adjust to this if the walk is to keep stepping through a row at a time:

http://net-snmp.sourceforge.net/wiki/index.php/TUT:snmptable#Holes_in_Tables

The snmpgetnext tutorial includes a discussion of the idea of "holes" in a table, illustrated by an (artificially) missing value for sysORDescr.4. The snmptable command will handle such holes automatically, filling in any such missing values:

Per #17 these currently result in Mismatching VarBind[%v] OID for %v: index %v != expected %v errors for the sparse columns, and the sparse column values after the first hole will be missing (null in the API JSON).

It seems like the table-walking logic needs to pick the minimum index from the response, and also use that index for the other objects in the following GETNEXT request?

TODO: Use Go 1.10 encoding/asn1:MarshalWithParams

New in Go 1.10: https://beta.golang.org/pkg/encoding/asn1/#MarshalWithParams

Should simplify snmp.pack

TODO: Automatic import of MIBs

Currently MIBs have to be writte out by hand as importable Go packages:

https://github.com/qmsk/snmpbot/blob/master/mibs/bridge_mib/bridge_mib.go

This is painful, but so is complete SNMP/SMI/ASN.1 MBI parsing...

Find/implement an ASN.1 parser in Go for direct runtime loading of MIBs
Use some existing (Python?) ASN.1 parser to transpile MIBs to either:
- compiled/statically-linked Go packages
- runtime-importable JSON files

Server busyloop on query for empty objects set

For GET /api/hosts/test/objects/?object=x (or something equivalent like an object in a MIB that failed to probe):

INFO server: Query objects {} @ {test}
SIGABRT: abort

...

goroutine 9 [chan receive]:
github.com/qmsk/snmpbot/server.(*objectsHandler).query(0xc420332780, 0xc420330380, 0x1, 0x1, 0xc42032e270, 0xc42006e960)
        /home/terom/go/src/github.com/qmsk/snmpbot/server/object.go:262 +0x589
github.com/qmsk/snmpbot/server.(*objectsHandler).GetREST(0xc420332780, 0x71b140, 0xc420332780, 0x7f924cae65a8, 0xc420332780)
        /home/terom/go/src/github.com/qmsk/snmpbot/server/object.go:293 +0x11d
github.com/qmsk/go-web.API.handle(0x726560, 0xc4201ec300, 0x8fee00, 0xc420110a80, 0xc4200a0b00, 0xc420327bf0, 0x411bb7)
        /home/terom/go/src/github.com/qmsk/go-web/rest.go:232 +0x736
github.com/qmsk/go-web.API.ServeHTTP(0x726560, 0xc4201ec300, 0x8fee00, 0xc420110a80, 0xc4200a0b00)
        /home/terom/go/src/github.com/qmsk/go-web/rest.go:315 +0x74
github.com/qmsk/go-web.(*API).ServeHTTP(0xc4200441c0, 0x8fee00, 0xc420110a80, 0xc4200a0b00)
        <autogenerated>:1 +0x63
net/http.StripPrefix.func1(0x8fee00, 0xc420110a80, 0xc4200a0a00)
        /home/terom/opt/go-1.9.2/src/net/http/server.go:1957 +0x1a5
net/http.HandlerFunc.ServeHTTP(0xc42026cb70, 0x8fee00, 0xc420110a80, 0xc4200a0a00)
        /home/terom/opt/go-1.9.2/src/net/http/server.go:1918 +0x44
net/http.(*ServeMux).ServeHTTP(0xc42026cba0, 0x8fee00, 0xc420110a80, 0xc4200a0a00)
        /home/terom/opt/go-1.9.2/src/net/http/server.go:2254 +0x130
net/http.serverHandler.ServeHTTP(0xc4203cc680, 0x8fee00, 0xc420110a80, 0xc4200a0a00)
        /home/terom/opt/go-1.9.2/src/net/http/server.go:2619 +0xb4
net/http.(*conn).serve(0xc4204be000, 0x8ff440, 0xc4202fc000)
        /home/terom/opt/go-1.9.2/src/net/http/server.go:1801 +0x71d
created by net/http.(*Server).Serve
        /home/terom/opt/go-1.9.2/src/net/http/server.go:2720 +0x288

Per some stack traces, the query goroutine is stuck in client:Client.walkGetBulk:

goroutine 36 [runnable]:
github.com/qmsk/snmpbot/client.walkScalarVars(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /home/terom/go/src/github.com/qmsk/snmpbot/client/walk.go:7 +0x3a7
github.com/qmsk/snmpbot/client.(*Client).walkGetBulk(0xc420096370, 0x0, 0x0, 0x0, 0x952d50, 0x0, 0x0, 0xc420041e80, 0x0, 0x0)
        /home/terom/go/src/github.com/qmsk/snmpbot/client/walk.go:117 +0x383
github.com/qmsk/snmpbot/client.(*Client).WalkWithScalars(0xc420096370, 0x0, 0x0, 0x0, 0x952d50, 0x0, 0x0, 0xc42002ae80, 0x0, 0x730de0)
        /home/terom/go/src/github.com/qmsk/snmpbot/client/walk.go:62 +0x108
github.com/qmsk/snmpbot/client.(*Client).Walk(0xc420096370, 0x952d50, 0x0, 0x0, 0xc42002af00, 0x0, 0x0)
        /home/terom/go/src/github.com/qmsk/snmpbot/client/walk.go:135 +0x91
github.com/qmsk/snmpbot/mibs.Client.WalkObjects(0xc420096370, 0x952d50, 0x0, 0x0, 0xc42002af78, 0xc420312180, 0x933800)
        /home/terom/go/src/github.com/qmsk/snmpbot/mibs/client.go:43 +0x19e
github.com/qmsk/snmpbot/server.(*ObjectQuery).queryHost(0xc42032e2a0, 0xc4200962c0, 0xc42032e2b8, 0x775001)
        /home/terom/go/src/github.com/qmsk/snmpbot/server/query.go:42 +0x9a
github.com/qmsk/snmpbot/server.(*ObjectQuery).query.func1(0xc42032e2a0, 0xc4200962c0)
        /home/terom/go/src/github.com/qmsk/snmpbot/server/query.go:66 +0x65
created by github.com/qmsk/snmpbot/server.(*ObjectQuery).query
        /home/terom/go/src/github.com/qmsk/snmpbot/server/query.go:63 +0x105

goroutine 20 [runnable]:
github.com/qmsk/snmpbot/client.makeBulkVars(0x0, 0x0, 0x0, 0x952d50, 0x0, 0x0, 0x952d50, 0x0, 0x0)
        /home/terom/go/src/github.com/qmsk/snmpbot/client/client.go:182 +0x60
github.com/qmsk/snmpbot/client.(*Client).GetBulk(0xc4200a4370, 0x0, 0x0, 0x0, 0x952d50, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /home/terom/go/src/github.com/qmsk/snmpbot/client/client.go:221 +0xb6
github.com/qmsk/snmpbot/client.(*Client).walkGetBulk(0xc4200a4370, 0x0, 0x0, 0x0, 0x952d50, 0x0, 0x0, 0xc420041e80, 0x0, 0x0)
        /home/terom/go/src/github.com/qmsk/snmpbot/client/walk.go:112 +0x2f9
github.com/qmsk/snmpbot/client.(*Client).WalkWithScalars(0xc4200a4370, 0x0, 0x0, 0x0, 0x952d50, 0x0, 0x0, 0xc42002be80, 0x0, 0x730de0)
        /home/terom/go/src/github.com/qmsk/snmpbot/client/walk.go:62 +0x108
github.com/qmsk/snmpbot/client.(*Client).Walk(0xc4200a4370, 0x952d50, 0x0, 0x0, 0xc42002bf00, 0x0, 0x0)
        /home/terom/go/src/github.com/qmsk/snmpbot/client/walk.go:135 +0x91
github.com/qmsk/snmpbot/mibs.Client.WalkObjects(0xc4200a4370, 0x952d50, 0x0, 0x0, 0xc42002bf78, 0xc4201325c0, 0x0)
        /home/terom/go/src/github.com/qmsk/snmpbot/mibs/client.go:43 +0x19e
github.com/qmsk/snmpbot/server.(*ObjectQuery).queryHost(0xc4203d2bd0, 0xc4200a42c0, 0xc4203d2be8, 0x0)
        /home/terom/go/src/github.com/qmsk/snmpbot/server/query.go:42 +0x9a
github.com/qmsk/snmpbot/server.(*ObjectQuery).query.func1(0xc4203d2bd0, 0xc4200a42c0)
        /home/terom/go/src/github.com/qmsk/snmpbot/server/query.go:66 +0x65
created by github.com/qmsk/snmpbot/server.(*ObjectQuery).query
        /home/terom/go/src/github.com/qmsk/snmpbot/server/query.go:63 +0x105

server: hosts are only probed once at start

The will only probe each host once when loading a new host. The probing determines the supported host MIBs and the Online state, so if the probe fails, then the offline host will not be included in any /api/objects/... queries, and the empty mibs will result in an empty /api/hosts/.../objects/ query set. Conversely, if the probing succeeds but the host later goes offline, then any /api/objects/... queries will get delayed waiting for that host to timeout.

The host online states should be polled and updated at runtime to avoid those problems.

Workaround: Use PUT /api/hosts/... to re-load the host.

Going a bit further for the long term, the host mibs might also need to get re-probed if the device is upgraded... The hostname might also need to get re-resolved if the device moves? Alternatively, just require the server to be restarted in those cases...

table walk fails if any entry object is not accessible

If mibs:Client.WalkTable() => client:Client.Walk() returns snmp.EndOfMibView for any table entry object, then the entire table walk fails. It should probably return some invalid value for that column instead.

$ ~/go/bin/snmptable public@edgeswitch-098730 IF-MIB::ifTable
2018/03/04 19:18:15 Invalid VarBind[.1.3.6.1.2.1.2.2.1.15] Value for IF-MIB::ifInUnknownProtos: SNMP VarBind Error: EndOfMibView

$ ~/go/bin/snmpwalk -debug public@edgeswitch-098730 IF-MIB::ifInUnknownProtos
DEBUG client: Engine<[::]:51693>: Run...
DEBUG client: Engine<[::]:51693>: Send: client.IO{Addr:(*net.UDPAddr)(0xc420396180), Packet:snmp.Packet{Version:1, Community:[]uint8{0x70, 0x75, 0x62, 0x6c, 0x69, 0x63}, RawPDU:asn1.RawValue{Class:0, Tag:0, IsCompound:false, Bytes:[]uint8(nil), FullBytes:[]uint8(nil)}}, PDUType:1, PDU:snmp.GenericPDU{RequestID:0, ErrorStatus:0, ErrorIndex:0, VarBinds:[]snmp.VarBind{snmp.VarBind{Name:asn1.ObjectIdentifier{1, 3, 6, 1, 2, 1, 2, 2, 1, 15}, RawValue:asn1.RawValue{Class:0, Tag:5, IsCompound:false, Bytes:[]uint8(nil), FullBytes:[]uint8(nil)}}}}}
DEBUG client: Engine<[::]:51693>: Start request [email protected]:161[0]: GetNextRequest<1.3.6.1.2.1.2.2.1.15>@172.28.2.2:161[1]
DEBUG client: Engine<[::]:51693>: Recv: client.IO{Addr:(*net.UDPAddr)(0xc420198000), Packet:snmp.Packet{Version:1, Community:[]uint8{0x70, 0x75, 0x62, 0x6c, 0x69, 0x63}, RawPDU:asn1.RawValue{Class:2, Tag:2, IsCompound:true, Bytes:[]uint8{0x2, 0x1, 0x0, 0x2, 0x1, 0x0, 0x2, 0x1, 0x0, 0x30, 0x15, 0x30, 0x13, 0x6, 0xa, 0x2b, 0x6, 0x1, 0x2, 0x1, 0x2, 0x2, 0x1, 0x10, 0x1, 0x41, 0x5, 0x0, 0xb5, 0x98, 0xe, 0x22}, FullBytes:[]uint8{0xa2, 0x20, 0x2, 0x1, 0x0, 0x2, 0x1, 0x0, 0x2, 0x1, 0x0, 0x30, 0x15, 0x30, 0x13, 0x6, 0xa, 0x2b, 0x6, 0x1, 0x2, 0x1, 0x2, 0x2, 0x1, 0x10, 0x1, 0x41, 0x5, 0x0, 0xb5, 0x98, 0xe, 0x22}}}, PDUType:2, PDU:snmp.GenericPDU{RequestID:0, ErrorStatus:0, ErrorIndex:0, VarBinds:[]snmp.VarBind{snmp.VarBind{Name:asn1.ObjectIdentifier{1, 3, 6, 1, 2, 1, 2, 2, 1, 16, 1}, RawValue:asn1.RawValue{Class:1, Tag:1, IsCompound:false, Bytes:[]uint8{0x0, 0xb5, 0x98, 0xe, 0x22}, FullBytes:[]uint8{0x41, 0x5, 0x0, 0xb5, 0x98, 0xe, 0x22}}}}}}
DEBUG client: Engine<[::]:51693>: Request [email protected]:161[0] done: GetNextRequest<1.3.6.1.2.1.2.2.1.15>@172.28.2.2:161[1]
INFO client: Client<[email protected]:161>: Request GetNextRequest<1.3.6.1.2.1.2.2.1.15>@172.28.2.2:161[1] => 1.3.6.1.2.1.2.2.1.16.1=3046641186
DEBUG client: Engine<[::]:51693>: Close...

GET .../tables/?table=... does not return per-entry errors

The GET .../tables/$TABLE API returns Entries with partial data + Errors if some table entry objects are invalid:

{
  "ID": "IF-MIB::ifTable",
  ...
  "Entries": [
    ...
    {
      "HostID": "public@localhost",
      "Index": {
        "IF-MIB::ifIndex": 5
      },
      "Objects": {
        "IF-MIB::ifAdminStatus": "up",
        "IF-MIB::ifDescr": "docker0",
        ...
        "IF-MIB::ifInNUcastPkts": null,
       ...
      },
      "Errors": [
        "Invalid VarBind[.1.3.6.1.2.1.2.2.1.12.5] index for IF-MIB::ifInNUcastPkts: index [5] starts with 5"
      ]
    },
    ...
  ]
}


The `GET .../tables/?table=$TABLE` API does not, if any entry object results in errors, then it omits the entire row from `Entires` and returns a top-level `Errors` instead:

```json
[
  {
    "ID": "IF-MIB::ifTable",
    ...
    "Entries": [
      ...
    ],
    "Errors": [
      {
        "HostID": "public@localhost",
        "Error": "Invalid VarBind[.1.3.6.1.2.1.2.2.1.12.5] index for IF-MIB::ifInNUcastPkts: index [5] starts with 5"
      }
    ]
  }
]

This is a bug in #17

TODO: Rate-limit parallel queries

There is currently no rate-limit on the number of parallel queries for a server.Host... I'm afraid some cheaply implemented network devices will choke if they get too many concurrent requests, and the client.Client starts retrying?

client should support GetBulk

See RFC1905 #4.2.3, avoids the high query latency associated with traversing large tables one row a time with the associated round-trip delay.

The client:Client.Walk function should know how to do either a GetNext walk or a GetBulk walk, possibly based on some client:Options.WalkBulk flag?

While we're at it, the client:Client.getRequest request splitting using MaxVars should probably also be moved into the higher-level Walk interface (options.WalkMaxVars?), keeping the low-level Get, GetNext and GetBulk requests simple.