firehose-nozzle-v2

Connecting

There are two ways to connect get data from the v2 API.

Each is described below. Using the Gateway is the easier path to building a nozzle.

Building a Nozzle Using RLP Gateway (Recommended Path)

The RLP Gateway adds:

The ability to deploy applications as a nozzle
Eliminating the need for mTLS
Does not require any Loggregator specific libraries to process data

The gateway was released in PCF version 2.4

The code in this repo is for bootstraping, but the client here: https://github.com/cloudfoundry/go-loggregator/blob/master/rlp_gateway_client.go should be the basis for a production nozzle.

Authentication & Testing

To create a UAA user that can access the data, use the UAA CLI.

Create the user:

uaac target https://uaa.sys.<pcf system domain> --skip-ssl-validation
uaac token client get admin -s <admin client secret>
uaac client add my-v2-nozzle \
  --name my-v2-nozzle \
  --secret <my-v2-nozzle client secret> \
  --authorized_grant_types client_credentials,refresh_token \
  --authorities logs.admin

To manually get a token:

uaac token client get v2-nozzle-test -s <my-v2-nozzle client secret>
uaac context

The RLP Gateway data can be tested with just curl. To view the data (newline delimited JSON payloads), copy the token and run:

export token=<my-v2-nozzle token from context>
curl -k -H "Authorization: $token" 'https://log-stream.sys.<pcf system domain>/v2/read?counter&gauge'

Building a Nozzle Directly Connecting to RLP

Communication is done directly to the RLP over HTTP/2. This endpoint is discoverable via its BOSH LINK

The link is shared in a PCF deployment:

...
provides: |
  reverse_log_proxy: {as: reverse_log_proxy, shared: true}
...

In a full runtime deployment, this component is on the loggregator_trafficcontroller vm and listens on :8082
In the small footprint PAS, it is deployed on the control vm and listens on :8086

Authentication is done via mTLS. The mTLS connection is authenticated by connecting with a certificate signed by the Ops Manager Root CA.

For development, the nozzle author can manually generate a certificate signed by OpsMan's root CA using its certificate API and the om tool:

om -t https://pcf.example.com -k -u ${user} -p ${pwd} \
    curl -x POST -p /api/v0/certificates/generate -d '{ "domains": ["*.example.com", "*.sub.example.com"] }'

The domain used in the certificate does not matter.

To get the root certificate from Ops Manager, download in advanced settings: Admin -> Settings -> Advanced -> Download Root CA Cert

As mentioned above, the communication uses http/2. In a PCF environment (rather than cfdev or some other tooling where the component would be directly accessible), one way to develop locally is to setup a ssh tunnel through OpsMan:

ssh -i [path to ssh private key] \
  -L 9000:[IP of loggregator_trafficcontroller or control VM]:[8082 or 8086] \
  [email protected]

src/local_dev.template.sh is a sample script that will run the nozzle, once the certificates are generated and put on disk.

Tile

The tile directory packages the gateway version of the nozzle as an app deployed on the platform.

Build is done via PCF Tile Generator

References

v1 -> v2 mapping: https://github.com/cloudfoundry/loggregator-api/blob/master/README.md#v2---v1-mapping
v2 reference example https://github.com/cloudfoundry-incubator/refnozzle
Envelope proto buff def https://github.com/cloudfoundry/loggregator-api/blob/master/v2/envelope.proto
Example: https://github.com/cloudfoundry/go-loggregator/blob/master/examples/envelope_stream_connector/main.go
CLI plugin to stream v2 data https://github.com/cloudfoundry/log-cache-cli
https://github.com/cloudfoundry/cf-drain-cli

petereltgroth / firehose-nozzle-v2 Goto Github PK