Failure testing about vector

[michaelfairley]

I pored over the code base and roughly cataloged the main things that can go wrong:

• Splunk TCP source

  • Listener can't listen (e.g. socket already in use)
  • Client sends a line that's bigger than our buffer
  • Misc connection errors. (See TcpListener#incoming Errors docs for more info)

• Splunk HEC sink (error code docs)

  • 4xx that's probably the user's fault (e.g. unauthorized because of wrong credentials)
  • 4xx that's probably router's fault (e.g. bad formatting of message)
  • 5xx
  • Can't connect to server
  • Misc connection error

• Splunk TCP sink

  • Can't connect
  • Connection broken

• S3 sink (error object docs)

  • Config error (e.g. bucket doesn't exist, bad credentials)
  • Validation error (probably out fault)
  • 5xx
  • Connection error
  • Misc other error

It seems like most of these are going to be easier to test by running fake servers (rather than using fail).

Something that we don't have but that we probably want: for sinks, we could run some sort of health check at boot time (e.g. hitting HEC's healthcheck route) to make sure we can alert about configuration or connection issues as soon as possible, instead of waiting on the first log messages to come in before they're noticed.