azure / oms-log-analytics-firehose-nozzle

Microsoft Azure Log Analytics Nozzle

Microsoft Azure Log Analytics Cloud Foundry Firehose Nozzle

Summary

Microsoft Operations Management Suite (OMS) is Microsoft's cloud-based IT management solution that helps you manage and protect your on-premises and cloud infrastructure.

Azure Log Analytics is a service in OMS that helps you collect and analyze data generated by resources in your cloud and on-premises environments. In the following document, it will be referred to as OMS Log Analytics.

The Microsoft Azure Log Analytics Nozzle is a Cloud Foundry (CF) component which forwards metrics from the Loggregator Firehose to OMS Log Analytics. In the following document, it will be referred to as Log Analytics Nozzle or nozzle for short.

Warning: This product was originally written against and published to the Pivotal Network against the Firehose V1 API (So no gurantee that it will work with current version of Cloud Foundry). In November 2019 the Cloud Foundry project indicated that this API would be deprecated. In preparation for this the OMS Nozzle has been removed from general support and is no longer supported under the standard Pivotal Lifecycle Policy. For future support, please raise a ticket with Pivotal, indicating the existence of this ticket with Microsoft.

Prerequisites

1. Deploy a CF or PCF environment in Azure

• Deploy Cloud Foundry on Azure
• Deploy Pivotal Cloud Foundry on Azure

2. Install CLIs on your dev box

• Install Cloud Foundry CLI
• Install Cloud Foundry UAA Command Line Client

3. Create an OMS Workspace in Azure

• Get started with Log Analytics
• OMS Cloud Foundry Monitoring Solution

Deploy - Push the Nozzle as an App to Cloud Foundry

1. Utilize the CF CLI to authenticate with your CF instance

cf login -a https://api.${ENDPOINT} -u ${CF_USER} --skip-ssl-validation

2. Create a CF user and grant required privileges

The Log Analytics Nozzle requires a CF user who is authorized to access the loggregator firehose.

uaac target https://uaa.${ENDPOINT} --skip-ssl-validation
uaac token client get admin
cf create-user ${FIREHOSE_USER} ${FIREHOSE_USER_PASSWORD}
uaac member add cloud_controller.admin ${FIREHOSE_USER}
uaac member add doppler.firehose ${FIREHOSE_USER}

3. Download the latest code

git clone https://github.com/Azure/oms-log-analytics-firehose-nozzle.git
cd oms-log-analytics-firehose-nozzle

4. Set environment variables in manifest.yml

OMS_WORKSPACE             : OMS workspace ID
OMS_KEY                   : OMS key
OMS_POST_TIMEOUT          : HTTP post timeout for sending events to OMS Log Analytics
OMS_BATCH_TIME            : Interval for posting a batch to OMS Log Analytics
OMS_MAX_MSG_NUM_PER_BATCH : The max number of messages in a batch to OMS Log Analytics
API_ADDR                  : The API address of the CF environment. If set empty or absent, nozzle will use API address for current CF environment
DOPPLER_ADDR              : Loggregator's traffic controller URL. If set empty or absent, nozzle will generate it from API address
FIREHOSE_USER             : CF user who has admin and firehose access
FIREHOSE_USER_PASSWORD    : Password of the CF user
EVENT_FILTER              : Event types to be filtered out. The format is a comma separated list, valid event types are METRIC,LOG,HTTP
SPACE_WHITELIST           : Comma separated space white list, logs from apps within these spaces will be forwarded. If not set, all apps will be monitored. Format should be ORG_NAME.SPACE_NAME or ORG_NAME.*
SKIP_SSL_VALIDATION       : If true, allows insecure connections to the UAA and the Trafficcontroller
CF_ENVIRONMENT            : Set to any string value for identifying logs and metrics from different CF environments
IDLE_TIMEOUT              : Keep Alive duration for the firehose consumer
LOG_LEVEL                 : Logging level of the nozzle, valid levels: DEBUG, INFO, ERROR
LOG_EVENT_COUNT           : If true, the total count of events that the nozzle has received and sent will be logged to OMS Log Analytics as CounterEvents
LOG_EVENT_COUNT_INTERVAL  : The time interval of logging event count to OMS Log Analytics

5. Push the app

cf push

Additional logging

For the most part, the Log Analytics Nozzle forwards metrics from the Loggregator Firehose to OMS Log Analytics without too much processing. In a few cases the nozzle might push some additional metrics to OMS Log Analytics.

1. eventsReceived, eventsSent and eventsLost

If LOG_EVENT_COUNT is set to true, the nozzle will periodically send to OMS Log Analytics the count of received events, sent events and lost events, at intervals of LOG_EVENT_COUNT_INTERVAL.

The statistic count is sent as a CounterEvent, with CounterKey of one of nozzle.stats.eventsReceived, nozzle.stats.eventsSent and nozzle.stats.eventsLost. Each CounterEvent contains the value of delta count during the interval, and the total count from the beginning. eventsReceived counts all the events that the nozzle received from firehose, eventsSent counts all the events that the nozzle sent to OMS Log Analytics successfully, eventsLost counts all the events that the nozzle tried to send but failed after 4 attempts.

These CounterEvents themselves are not counted in the received, sent or lost count.

In normal cases, the total count of eventsSent plus eventsLost is less than total eventsReceived at the same time, as the nozzle buffers some messages and then post them in a batch to OMS Log Analytics. Operator can adjust the buffer size by changing the configurations OMS_BATCH_TIME and OMS_MAX_MSG_NUM_PER_BATCH.

2. slowConsumerAlert

When the nozzle receives slow consumer alert from loggregator in three ways:

1. the nozzle receives a WebSocket close error with error code ClosePolicyViolation (1008)
2. the nozzle receives a CounterEvent with the name TruncatingBuffer.DroppedMessages
3. the nozzle receives a CounterEvent with the name doppler_proxy.slow_consumer

the nozzle will send a slowConsumerAlert as a ValueMetric to OMS Log Analytics, with MetricKey nozzle.alert.slowConsumerAlert and value 1.

This ValueMetric is not counted in the above statistic received, sent or lost count.

Scaling guidance

1. Scaling Nozzle

Operators should run at least two instances of the nozzle to reduce message loss. The Firehose will evenly distribute events across all instances of the nozzle.

When the nozzle couldn't keep up with processing the logs from firehose, Loggregator alerts the nozzle and then the nozzle logs slowConsumerAlert message to OMS Log Analytics. Operator can create Alert rule for this slowConsumerAlert message in OMS Log Analytics, and when the alert is triggered, the operator should scale up the nozzle to minimize the loss of data.

We did some workload test against the nozzle and got a few data for operaters' reference:

• In our test, the size of each log and metric sent to OMS Log Analytics is around 550 bytes, suggest each nozzle instance should handle no more than 300000 such messages per minute. Under such workload, the CPU usage of each instance is around 40%, and the memory usage of each instance is around 80M.

2. Scaling Loggregator

Loggregator emits LGR log message to indicate problems with the logging process. When operaters see this message in OMS Log Analytics, they might need to scale Loggregator.

View in OMS Portal

1. Import OMS Default Solution from Azure Marketplace

With our OMS Cloud Foundry Monitoring Solution, you can import our 20+ default views, 130+ alerts and 90+ saved searches covering all KPI of Cloud Foundry you might care about, with only simple configurations.

Please check here for detailed document and underlying templates.

You can find a complete list of KPI here.

2. Customizing your OMS Workplace

2.1. Import Individual OMS View

From the main OMS Overview page, go to View Designer -> Import -> Browse, select one of the omsview files, e.g. Cloud Foundry.omsview, and save the view. Now a Tile will be displayed on the main OMS Overview page. Click the Tile, it shows visualized metrics.

Operators could customize these views or create new views through View Designer.

To create your own view, please refer to the document here.

Please feel free to send your suggestions and feedback for the full view by creating Github issues.

3.1. Create Alert Rules Manually

This section describes some sample alert rules that operators may want to create for identifying important information in their Cloud Foundry deployments.

For the process of creating alert rules in OMS Log Analytics, please refer to this article.

Operators could customize the queries and threshold values as needed.

Search query	Generate alert based on	Description
Type=CF_ValueMetric_CL Origin_s=bbs Name_s="Domain.cf-apps"	Number of results < 1	bbs.Domain.cf-apps indicates if the cf-apps Domain is up-to-date, meaning that CF App requests from Cloud Controller are synchronized to bbs.LRPsDesired (Diego-desired AIs) for execution. No data received means cf-apps Domain is not up-to-date in the given time window.
Type=CF_ValueMetric_CL Origin_s=rep Name_s=UnhealthyCell Value_d>1	Number of results > 0	For Diego cells, 0 means healthy, and 1 means unhealthy. Set the alert if multiple unhealthy Diego cells are detected in the given time window.
Type=CF_ValueMetric_CL Origin_s="bosh-hm-forwarder" Name_s="system.healthy" Value_d=0	Number of results > 0	1 means the system is healthy, and 0 means the system is not healthy.
Type=CF_ValueMetric_CL Origin_s=route_emitter Name_s=ConsulDownMode Value_d>0	Number of results > 0	Consul emits its health status periodically. 0 means the system is healthy, and 1 means that route emitter detects that Consul is down.
Type=CF_CounterEvent_CL Origin_s=DopplerServer (Name_s="TruncatingBuffer.DroppedMessages" or Name_s="doppler.shedEnvelopes") Delta_d>0	Number of results > 0	The delta number of messages intentionally dropped by Doppler due to back pressure.
Type=CF_LogMessage_CL SourceType_s=LGR MessageType_s=ERR	Number of results > 0	Loggregator emits LGR to indicate problems with the logging process, e.g. when log message output is too high.
Type=CF_ValueMetric_CL Name_s=slowConsumerAlert	Number of results > 0	When the nozzle receives slow consumer alert from Loggregator, it sends slowConsumerAlert ValueMetric to OMS.
Type=CF_CounterEvent_CL Job_s=nozzle Name_s=eventsLost Delta_d>0	Number of results > 0	If the delta number of lost events reaches a threshold, it means the nozzle might have some problem running.

Access OMS Everywhere

OMS also provides a mobile app for users to view OMS views, receiving alerts and searching for logs from your mobile devices.

Simply download App from your app store and login with your account, you can have experience just the same as on your workplace everywhere.

OMS Apps now available on Windows (Mobile devices), Android and iOS devices.

Test

You need ginkgo to run the test. Run the following command to execute test:

ginkgo -r

Additional Reference

To collect syslogs and performance metrics of VMs in CloudFoundry deployment, a system metric provider is required.

For PCF 2.0+, 'Bosh System Metrics Forwarder` has been included in your deployment, VM metrics will be forwarded to loggregator automatically.

At the same time, you can also install BOSH Health Metric Forwarder manually if you're using older version. Please refer to its document for instructions. Be advanced, we have noticed that it might occasionally conflict with Azure OMS Agent.

For Syslog, we recommend you to use Azure OMS Agent Bosh release. We have planned to provide other approaches for Syslog in the future.

oms-log-analytics-firehose-nozzle's Issues

Offline environment will not be able to use Pivotal tile

Offline environment will not be able to download buildpack

App log Filtering by Orgs or/and Spaces

We have a CF installation that is used by many teams, we would only want to ship the logs for the apps we care about into OMS. Is it possible to do some filtering by space or app?

column explanation document

Is there any document to explain means of column for Developers ?
For example, TimeGenerated and EventTime are confusing for Developers without reading source code.

I want to some kind of list to explain columns

Unmarshal error

There is an error with the nozzle when trying to unmarshal Doppler data.

Retrieving logs for app �[36;1mazure-log-analytics-nozzle-1.1.0�[0m in org �[36;1msystem�[0m / space �[36;1mazure-log-analytics-nozzle-space�[0m as �[36;1madmin�[0m...

   2017-08-23T09:25:39.41-0600 [CELL/1] OUT Destroying container
   2017-08-23T09:25:39.45-0600 [CELL/1] OUT Creating container
   2017-08-23T09:25:39.85-0600 [CELL/1] OUT Successfully destroyed container
   �[31m2017-08-23T09:25:40.31-0600 [APP/PROC/WEB/1] ERR oms-log-analytics-firehose-nozzle: error: time: missing unit in duration 30, try --help�[0m
   2017-08-23T09:25:40.44-0600 [CELL/0] OUT Creating container
   2017-08-23T09:25:41.60-0600 [CELL/0] OUT Successfully destroyed container
   2017-08-23T09:26:25.31-0600 [API/0] OUT Process has crashed with type: "web"
   2017-08-23T09:27:55.19-0600 [CELL/1] OUT Successfully created container
   2017-08-23T09:27:55.54-0600 [CELL/1] OUT Destroying container
   2017-08-23T09:27:55.68-0600 [API/0] OUT App instance exited with guid a1e48e70-b22d-4bc1-b1d1-d1451823782a payload: {"instance"=>"3de8840e-fcac-4549-42e6-5c56", "index"=>0, "reason"=>"CRASHED", "exit_description"=>"2 error(s) occurred:\n\n* Exited with status 1\n* cancelled", "crash_count"=>5, "crash_timestamp"=>1503502075642244799, "version"=>"46904aef-03bb-4cf1-8136-d856c77ccf0f"}
   2017-08-23T09:27:56.05-0600 [CELL/1] OUT Successfully destroyed container
   2017-08-23T09:30:24.86-0600 [CELL/0] OUT Creating container
   2017-08-23T09:30:25.54-0600 [CELL/0] OUT Exit status 0
   2017-08-23T09:30:25.55-0600 [CELL/0] OUT Destroying container
   2017-08-23T09:30:26.02-0600 [CELL/0] OUT Successfully destroyed container
   2017-08-23T09:34:55.52-0600 [CELL/1] OUT Successfully created container
   2017-08-23T09:34:55.84-0600 [CELL/1] OUT Exit status 0
   2017-08-23T09:43:26.17-0600 [CELL/0] OUT Successfully created container
   2017-08-23T09:43:26.59-0600 [CELL/0] OUT Exit status 0
   2017-08-23T09:43:26.63-0600 [API/0] OUT App instance exited with guid a1e48e70-b22d-4bc1-b1d1-d1451823782a payload: {"instance"=>"1114cf50-3c01-422a-76b8-0a12", "index"=>0, "reason"=>"CRASHED", "exit_description"=>"2 error(s) occurred:\n\n* Exited with status 1\n* cancelled", "crash_count"=>8, "crash_timestamp"=>1503503006591899799, "version"=>"46904aef-03bb-4cf1-8136-d856c77ccf0f"}
   2017-08-23T09:59:27.54-0600 [API/0] OUT Process has crashed with type: "web"
   2017-08-23T10:15:28.89-0600 [CELL/1] OUT Exit status 0
   2017-08-23T10:15:28.89-0600 [CELL/1] OUT Destroying container
   2017-08-23T10:15:29.42-0600 [CELL/1] OUT Successfully destroyed container
   2017-08-23T10:31:29.37-0600 [CELL/1] OUT Successfully created container
   2017-08-23T10:31:29.75-0600 [API/0] OUT App instance exited with guid a1e48e70-b22d-4bc1-b1d1-d1451823782a payload: {"instance"=>"f45ff5c0-94df-440f-63df-564d", "index"=>1, "reason"=>"CRASHED", "exit_description"=>"2 error(s) occurred:\n\n* Exited with status 1\n* cancelled", "crash_count"=>11, "crash_timestamp"=>1503505889712173999, "version"=>"46904aef-03bb-4cf1-8136-d856c77ccf0f"}
   2017-08-23T10:31:30.22-0600 [CELL/1] OUT Successfully destroyed container
   2017-08-23T10:47:29.88-0600 [CELL/1] OUT Creating container
   �[31m2017-08-23T10:47:32.35-0600 [APP/PROC/WEB/1] ERR oms-log-analytics-firehose-nozzle: error: time: missing unit in duration 30, try --help�[0m
   2017-08-23T10:47:32.35-0600 [APP/PROC/WEB/1] OUT Exit status 1
   2017-08-23T10:47:32.36-0600 [CELL/1] OUT Exit status 0
   2017-08-23T10:47:32.38-0600 [API/0] OUT Process has crashed with type: "web"
   2017-08-23T10:47:32.85-0600 [CELL/1] OUT Successfully destroyed container
   2017-08-23T11:03:31.15-0600 [CELL/0] OUT Creating container
   2017-08-23T11:19:32.75-0600 [API/0] OUT Process has crashed with type: "web"
   2017-08-23T11:35:33.45-0600 [CELL/0] OUT Successfully created container
   2017-08-23T11:35:33.73-0600 [APP/PROC/WEB/0] OUT Exit status 1
   2017-08-23T11:35:33.80-0600 [API/0] OUT App instance exited with guid a1e48e70-b22d-4bc1-b1d1-d1451823782a payload: {"instance"=>"39765669-d44e-4ac4-65ce-96dc", "index"=>0, "reason"=>"CRASHED", "exit_description"=>"2 error(s) occurred:\n\n* Exited with status 1\n* cancelled", "crash_count"=>15, "crash_timestamp"=>1503509733759388299, "version"=>"46904aef-03bb-4cf1-8136-d856c77ccf0f"}
   2017-08-23T11:36:20.49-0600 [API/0] OUT Updated app with guid a1e48e70-b22d-4bc1-b1d1-d1451823782a ({"name"=>"azure-log-analytics-nozzle-1.1.0", "instances"=>2, "memory"=>512, "buildpack"=>"https://github.com/cloudfoundry/go-buildpack.git", "environment_json"=>"PRIVATE DATA HIDDEN"})
   2017-08-23T11:36:28.22-0600 [API/0] OUT Updated app with guid a1e48e70-b22d-4bc1-b1d1-d1451823782a ({"environment_json"=>"PRIVATE DATA HIDDEN"})
   2017-08-23T11:36:31.63-0600 [API/0] OUT Updated app with guid a1e48e70-b22d-4bc1-b1d1-d1451823782a ({"environment_json"=>"PRIVATE DATA HIDDEN"})
   2017-08-23T11:36:31.91-0600 [API/0] OUT Updated app with guid a1e48e70-b22d-4bc1-b1d1-d1451823782a ({"environment_json"=>"PRIVATE DATA HIDDEN"})
   2017-08-23T11:36:32.69-0600 [API/0] OUT Updated app with guid a1e48e70-b22d-4bc1-b1d1-d1451823782a ({"environment_json"=>"PRIVATE DATA HIDDEN"})
   2017-08-23T11:36:33.22-0600 [API/0] OUT Updated app with guid a1e48e70-b22d-4bc1-b1d1-d1451823782a ({"environment_json"=>"PRIVATE DATA HIDDEN"})
   2017-08-23T11:36:33.99-0600 [STG/0] OUT Creating container
   2017-08-23T11:36:34.45-0600 [STG/0] OUT Downloading build artifacts cache...
   2017-08-23T11:36:45.81-0600 [STG/0] OUT -----> Go Buildpack version 1.8.6
   2017-08-23T11:36:45.81-0600 [STG/0] OUT -----> Installing godep v79
   2017-08-23T11:36:46.09-0600 [STG/0] OUT        �[31;1m**WARNING**�[0m A newer version of godep is available in this buildpack. Please adjust your app to use version 79.0.0 instead of version v79 as soon as possible. Old versions of godep are only provided to assist in migrating to newer versions.
   2017-08-23T11:36:46.73-0600 [STG/0] OUT -----> Installing go 1.7.6
   2017-08-23T11:36:53.59-0600 [STG/0] OUT -----> Running go build finalize
   2017-08-23T11:37:13.20-0600 [STG/0] OUT Exit status 0
   2017-08-23T11:37:13.25-0600 [STG/0] OUT Uploaded build artifacts cache (198B)
   2017-08-23T11:37:16.07-0600 [CELL/1] OUT Successfully created container
   2017-08-23T11:37:16.55-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503509836.553406239","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"OMS_BATCH_TIME":"10s"}}
   �[31m2017-08-23T11:37:16.62-0600 [APP/PROC/WEB/1] ERR goroutine 1 [running]:�[0m
   �[31m2017-08-23T11:37:16.62-0600 [APP/PROC/WEB/1] ERR github.com/Azure/oms-log-analytics-firehose-nozzle/vendor/code.cloudfoundry.org/lager.(*logger).Fatal(0xc4200186c0, 0x561eb5f7555d, 0x17, 0x561eb6463760, 0xc42011a150, 0x0, 0x0, 0x0)�[0m
   �[31m2017-08-23T11:37:16.62-0600 [APP/PROC/WEB/1] ERR 	/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/vendor/code.cloudfoundry.org/lager/logger.go:152 +0x41e�[0m
   �[31m2017-08-23T11:37:16.62-0600 [APP/PROC/WEB/1] ERR github.com/Azure/oms-log-analytics-firehose-nozzle/caching.(*Caching).Initialize(0xc420014bc0)�[0m
   �[31m2017-08-23T11:37:16.62-0600 [APP/PROC/WEB/1] ERR 	/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/caching/caching.go:39 +0x5ae�[0m
   2017-08-23T11:37:17.61-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503509837.610341072","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"IDLE_TIMEOUT":"30s"}}
   �[31m2017-08-23T11:37:17.68-0600 [APP/PROC/WEB/0] ERR panic: Could not get api /v2/info: json: cannot unmarshal number into Go value of type cfclient.Endpoint�[0m
   �[31m2017-08-23T11:37:17.68-0600 [APP/PROC/WEB/0] ERR 	/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/caching/caching.go:39 +0x5ae�[0m
   �[31m2017-08-23T11:37:17.68-0600 [APP/PROC/WEB/0] ERR 	/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/main.go:162 +0x1f13�[0m
   2017-08-23T11:37:17.71-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503509837.711792946","source":"oms-nozzle","message":"oms-nozzle.error creating cfclient","log_level":3,"data":{"error":"Could not get api /v2/info: json: cannot unmarshal number into Go value of type cfclient.Endpoint","trace":"goroutine 1 [running]:\ngithub.com/Azure/oms-log-analytics-firehose-nozzle/vendor/code.cloudfoundry.org/lager.(*logger).Fatal(0xc420018720, 0x5610c591155d, 0x17, 0x5610c5dff760, 0xc42010ada0, 0x0, 0x0, 0x0)\n\t/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/vendor/code.cloudfoundry.org/lager/logger.go:131 +0xc9\ngithub.com/Azure/oms-log-analytics-firehose-nozzle/caching.(*Caching).Initialize(0xc420014bc0)\n\t/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/caching/caching.go:39 +0x5ae\ngithub.com/Azure/oms-log-analytics-firehose-nozzle/omsnozzle.(*OmsNozzle).Start(0xc42008e750, 0xc42000de40, 0xc4200dce00)\n\t/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/omsnozzle/oms_nozzle.go:67 +0x3d\nmain.main()\n\t/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/main.go:162 +0x1f13\n"}}
   2017-08-23T11:37:17.71-0600 [CELL/0] OUT Exit status 0
   �[31m2017-08-23T11:37:17.71-0600 [APP/PROC/WEB/1] ERR panic: Could not get api /v2/info: json: cannot unmarshal number into Go value of type cfclient.Endpoint�[0m
   �[31m2017-08-23T11:37:17.71-0600 [APP/PROC/WEB/1] ERR goroutine 1 [running]:�[0m
   �[31m2017-08-23T11:37:17.71-0600 [APP/PROC/WEB/1] ERR panic(0x5610c5bdf0a0, 0xc42010ada0)�[0m
   �[31m2017-08-23T11:37:17.71-0600 [APP/PROC/WEB/1] ERR github.com/Azure/oms-log-analytics-firehose-nozzle/vendor/code.cloudfoundry.org/lager.(*logger).Fatal(0xc420018720, 0x5610c591155d, 0x17, 0x5610c5dff760, 0xc42010ada0, 0x0, 0x0, 0x0)�[0m
   �[31m2017-08-23T11:37:17.71-0600 [APP/PROC/WEB/1] ERR 	/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/vendor/code.cloudfoundry.org/lager/logger.go:152 +0x41e�[0m
   �[31m2017-08-23T11:37:17.71-0600 [APP/PROC/WEB/1] ERR github.com/Azure/oms-log-analytics-firehose-nozzle/caching.(*Caching).Initialize(0xc420014bc0)�[0m
   �[31m2017-08-23T11:37:17.71-0600 [APP/PROC/WEB/1] ERR 	/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/caching/caching.go:39 +0x5ae�[0m
   �[31m2017-08-23T11:37:17.71-0600 [APP/PROC/WEB/1] ERR 	/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/omsnozzle/oms_nozzle.go:67 +0x3d�[0m
   �[31m2017-08-23T11:37:17.71-0600 [APP/PROC/WEB/1] ERR main.main()�[0m
   �[31m2017-08-23T11:37:17.71-0600 [APP/PROC/WEB/1] ERR 	/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/main.go:162 +0x1f13�[0m
   2017-08-23T11:37:17.73-0600 [CELL/1] OUT Exit status 0
   2017-08-23T11:37:17.77-0600 [API/0] OUT Process has crashed with type: "web"
   2017-08-23T11:37:17.78-0600 [API/0] OUT App instance exited with guid a1e48e70-b22d-4bc1-b1d1-d1451823782a payload: {"instance"=>"3ba56c75-3583-4b1f-6e59-a5f3", "index"=>0, "reason"=>"CRASHED", "exit_description"=>"2 error(s) occurred:\n\n* Exited with status 2\n* cancelled", "crash_count"=>2, "crash_timestamp"=>1503509837721271199, "version"=>"6f568a4d-fce4-4879-a8e3-caeb22a0c267"}
   2017-08-23T11:37:17.80-0600 [CELL/0] OUT Creating container
   2017-08-23T11:37:17.80-0600 [CELL/1] OUT Creating container
   2017-08-23T11:37:18.17-0600 [CELL/0] OUT Successfully created container
   2017-08-23T11:37:18.18-0600 [CELL/1] OUT Successfully created container
   2017-08-23T11:37:18.45-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503509838.454396725","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"OMS_POST_TIMEOUT":"10s"}}
   2017-08-23T11:37:18.45-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503509838.454525709","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"SKIP_SSL_VALIDATION":true}}
   2017-08-23T11:37:18.45-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503509838.454556704","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"IDLE_TIMEOUT":"30s"}}
   2017-08-23T11:37:18.45-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503509838.454566240","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"OMS_BATCH_TIME":"10s"}}
   2017-08-23T11:37:18.45-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503509838.454634905","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"CF_ENVIRONMENT":"cf"}}
   2017-08-23T11:37:18.45-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503509838.454652786","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"OMS_MAX_MSG_NUM_PER_BATCH":1000}}
   2017-08-23T11:37:18.45-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503509838.454662800","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"LOG_EVENT_COUNT":true}}
   2017-08-23T11:37:18.45-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503509838.454745531","source":"oms-nozzle","message":"oms-nozzle.config EVENT_FILTER is nil. all events will be published","log_level":1,"data":{}}
   2017-08-23T11:37:18.48-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503509838.480115414","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"OMS_POST_TIMEOUT":"10s"}}
   2017-08-23T11:37:18.48-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503509838.480288029","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"IDLE_TIMEOUT":"30s"}}
   2017-08-23T11:37:18.48-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503509838.480304956","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"OMS_BATCH_TIME":"10s"}}
   2017-08-23T11:37:18.48-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503509838.480319500","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"CF_ENVIRONMENT":"cf"}}
   2017-08-23T11:37:18.48-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503509838.480335474","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"OMS_MAX_MSG_NUM_PER_BATCH":1000}}
   2017-08-23T11:37:18.48-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503509838.480373859","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"LOG_EVENT_COUNT":true}}
   2017-08-23T11:37:18.48-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503509838.480407238","source":"oms-nozzle","message":"oms-nozzle.config EVENT_FILTER is nil. all events will be published","log_level":1,"data":{}}
   2017-08-23T11:37:18.48-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503509838.480471373","source":"oms-nozzle","message":"oms-nozzle.getting nozzle instance name","log_level":1,"data":{"name":"pid-19@88370911-c601-4074-6439-4ce7"}}
   2017-08-23T11:37:18.49-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503509838.492529631","source":"oms-nozzle","message":"oms-nozzle.error creating cfclient","log_level":3,"data":{"error":"Could not get api /v2/info: json: cannot unmarshal number into Go value of type cfclient.Endpoint","trace":"goroutine 1 [running]:\ngithub.com/Azure/oms-log-analytics-firehose-nozzle/vendor/code.cloudfoundry.org/lager.(*logger).Fatal(0xc420018660, 0x55e65194e55d, 0x17, 0x55e651e3c760, 0xc4201221b0, 0x0, 0x0, 0x0)\n\t/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/vendor/code.cloudfoundry.org/lager/logger.go:131 +0xc9\ngithub.com/Azure/oms-log-analytics-firehose-nozzle/caching.(*Caching).Initialize(0xc420014bc0)\n\t/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/caching/caching.go:39 +0x5ae\ngithub.com/Azure/oms-log-analytics-firehose-nozzle/omsnozzle.(*OmsNozzle).Start(0xc420080750, 0xc42000de30, 0xc4200cee00)\n\t/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/omsnozzle/oms_nozzle.go:67 +0x3d\nmain.main()\n\t/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/main.go:162 +0x1f13\n"}}
   �[31m2017-08-23T11:37:18.49-0600 [APP/PROC/WEB/0] ERR panic: Could not get api /v2/info: json: cannot unmarshal number into Go value of type cfclient.Endpoint�[0m
   �[31m2017-08-23T11:37:18.49-0600 [APP/PROC/WEB/0] ERR 	/tmp/app/.cloudfoundry/0/go1.7.6/go/src/runtime/panic.go:500 +0x1a5�[0m
   �[31m2017-08-23T11:37:18.49-0600 [APP/PROC/WEB/0] ERR github.com/Azure/oms-log-analytics-firehose-nozzle/vendor/code.cloudfoundry.org/lager.(*logger).Fatal(0xc420018660, 0x55e65194e55d, 0x17, 0x55e651e3c760, 0xc4201221b0, 0x0, 0x0, 0x0)�[0m
   �[31m2017-08-23T11:37:18.49-0600 [APP/PROC/WEB/0] ERR 	/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/caching/caching.go:39 +0x5ae�[0m
   �[31m2017-08-23T11:37:18.49-0600 [APP/PROC/WEB/0] ERR 	/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/omsnozzle/oms_nozzle.go:67 +0x3d�[0m
   2017-08-23T11:37:18.49-0600 [APP/PROC/WEB/0] OUT Exit status 2
   2017-08-23T11:37:18.51-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503509838.514317036","source":"oms-nozzle","message":"oms-nozzle.error creating cfclient","log_level":3,"data":{"error":"Could not get api /v2/info: json: cannot unmarshal number into Go value of type cfclient.Endpoint","trace":"goroutine 1 [running]:\ngithub.com/Azure/oms-log-analytics-firehose-nozzle/vendor/code.cloudfoundry.org/lager.(*logger).Fatal(0xc420018660, 0x55e9f6bd955d, 0x17, 0x55e9f70c7760, 0xc42011e050, 0x0, 0x0, 0x0)\n\t/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/vendor/code.cloudfoundry.org/lager/logger.go:131 +0xc9\ngithub.com/Azure/oms-log-analytics-firehose-nozzle/caching.(*Caching).Initialize(0xc420014bc0)\n\t/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/caching/caching.go:39 +0x5ae\ngithub.com/Azure/oms-log-analytics-firehose-nozzle/omsnozzle.(*OmsNozzle).Start(0xc420080750, 0xc42000de30, 0xc4200cee00)\n\t/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/omsnozzle/oms_nozzle.go:67 +0x3d\nmain.main()\n\t/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/main.go:162 +0x1f13\n"}}
   2017-08-23T11:37:18.51-0600 [CELL/0] OUT Exit status 0
   �[31m2017-08-23T11:37:18.51-0600 [APP/PROC/WEB/1] ERR 	/tmp/app/.cloudfoundry/0/go1.7.6/go/src/runtime/panic.go:500 +0x1a5�[0m
   �[31m2017-08-23T11:37:18.51-0600 [APP/PROC/WEB/1] ERR 	/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/omsnozzle/oms_nozzle.go:67 +0x3d�[0m
   �[31m2017-08-23T11:37:18.51-0600 [APP/PROC/WEB/1] ERR 	/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/main.go:162 +0x1f13�[0m
   2017-08-23T11:37:18.51-0600 [APP/PROC/WEB/1] OUT Exit status 2
   2017-08-23T11:37:18.52-0600 [CELL/0] OUT Destroying container
   2017-08-23T11:37:18.53-0600 [CELL/1] OUT Exit status 0
   2017-08-23T11:37:18.53-0600 [API/1] OUT Process has crashed with type: "web"
   2017-08-23T11:37:18.54-0600 [API/1] OUT App instance exited with guid a1e48e70-b22d-4bc1-b1d1-d1451823782a payload: {"instance"=>"ba910590-0b7b-4262-5507-35f2", "index"=>0, "reason"=>"CRASHED", "exit_description"=>"2 error(s) occurred:\n\n* Exited with status 2\n* cancelled", "crash_count"=>3, "crash_timestamp"=>1503509838504482899, "version"=>"6f568a4d-fce4-4879-a8e3-caeb22a0c267"}
   2017-08-23T11:37:18.56-0600 [API/0] OUT Process has crashed with type: "web"
   2017-08-23T11:37:19.03-0600 [CELL/1] OUT Successfully destroyed container
   2017-08-23T11:38:03.26-0600 [CELL/0] OUT Creating container
   2017-08-23T11:38:03.65-0600 [CELL/1] OUT Successfully created container
   2017-08-23T11:38:03.92-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503509883.925692081","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"SKIP_SSL_VALIDATION":true}}
   2017-08-23T11:38:03.92-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503509883.925754786","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"IDLE_TIMEOUT":"30s"}}
   2017-08-23T11:38:03.92-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503509883.925806046","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"OMS_BATCH_TIME":"10s"}}
   2017-08-23T11:38:03.92-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503509883.926029205","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"CF_ENVIRONMENT":"cf"}}
   2017-08-23T11:38:03.92-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503509883.926046133","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"OMS_MAX_MSG_NUM_PER_BATCH":1000}}
   2017-08-23T11:38:03.92-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503509883.926115990","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"LOG_EVENT_COUNT":true}}
   2017-08-23T11:38:03.92-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503509883.926169157","source":"oms-nozzle","message":"oms-nozzle.config EVENT_FILTER is nil. all events will be published","log_level":1,"data":{}}
   2017-08-23T11:38:03.92-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503509883.926533461","source":"oms-nozzle","message":"oms-nozzle.getting nozzle instance name","log_level":1,"data":{"name":"pid-14@b3d1937c-daca-4481-49f9-3e62"}}
   2017-08-23T11:38:03.96-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503509883.963166237","source":"oms-nozzle","message":"oms-nozzle.error creating cfclient","log_level":3,"data":{"error":"Could not get api /v2/info: json: cannot unmarshal number into Go value of type cfclient.Endpoint","trace":"goroutine 1 [running]:\ngithub.com/Azure/oms-log-analytics-firehose-nozzle/vendor/code.cloudfoundry.org/lager.(*logger).Fatal(0xc420018660, 0x5561886ac55d, 0x17, 0x556188b9a760, 0xc4200fa170, 0x0, 0x0, 0x0)\n\t/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/vendor/code.cloudfoundry.org/lager/logger.go:131 +0xc9\ngithub.com/Azure/oms-log-analytics-firehose-nozzle/caching.(*Caching).Initialize(0xc420014bc0)\n\t/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/caching/caching.go:39 +0x5ae\ngithub.com/Azure/oms-log-analytics-firehose-nozzle/omsnozzle.(*OmsNozzle).Start(0xc42008e750, 0xc42000de30, 0xc4200dce00)\n\t/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/omsnozzle/oms_nozzle.go:67 +0x3d\nmain.main()\n\t/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/main.go:162 +0x1f13\n"}}
   �[31m2017-08-23T11:38:03.96-0600 [APP/PROC/WEB/1] ERR panic: Could not get api /v2/info: json: cannot unmarshal number into Go value of type cfclient.Endpoint�[0m
   �[31m2017-08-23T11:38:03.96-0600 [APP/PROC/WEB/1] ERR goroutine 1 [running]:�[0m
   �[31m2017-08-23T11:38:03.96-0600 [APP/PROC/WEB/1] ERR panic(0x55618897a0a0, 0xc4200fa170)�[0m
   �[31m2017-08-23T11:38:03.96-0600 [APP/PROC/WEB/1] ERR 	/tmp/app/.cloudfoundry/0/go1.7.6/go/src/runtime/panic.go:500 +0x1a5�[0m
   �[31m2017-08-23T11:38:03.96-0600 [APP/PROC/WEB/1] ERR github.com/Azure/oms-log-analytics-firehose-nozzle/vendor/code.cloudfoundry.org/lager.(*logger).Fatal(0xc420018660, 0x5561886ac55d, 0x17, 0x556188b9a760, 0xc4200fa170, 0x0, 0x0, 0x0)�[0m
   �[31m2017-08-23T11:38:03.96-0600 [APP/PROC/WEB/1] ERR 	/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/vendor/code.cloudfoundry.org/lager/logger.go:152 +0x41e�[0m
   �[31m2017-08-23T11:38:03.96-0600 [APP/PROC/WEB/1] ERR github.com/Azure/oms-log-analytics-firehose-nozzle/caching.(*Caching).Initialize(0xc420014bc0)�[0m
   �[31m2017-08-23T11:38:03.96-0600 [APP/PROC/WEB/1] ERR 	/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/caching/caching.go:39 +0x5ae�[0m
   �[31m2017-08-23T11:38:03.96-0600 [APP/PROC/WEB/1] ERR github.com/Azure/oms-log-analytics-firehose-nozzle/omsnozzle.(*OmsNozzle).Start(0xc42008e750, 0xc42000de30, 0xc4200dce00)�[0m
   �[31m2017-08-23T11:38:03.96-0600 [APP/PROC/WEB/1] ERR 	/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/omsnozzle/oms_nozzle.go:67 +0x3d�[0m
   �[31m2017-08-23T11:38:03.96-0600 [APP/PROC/WEB/1] ERR main.main()�[0m
   �[31m2017-08-23T11:38:03.96-0600 [APP/PROC/WEB/1] ERR 	/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/main.go:162 +0x1f13�[0m
   2017-08-23T11:38:03.96-0600 [APP/PROC/WEB/1] OUT Exit status 2
   2017-08-23T11:38:03.96-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503509883.968169212","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"OMS_POST_TIMEOUT":"10s"}}
   2017-08-23T11:38:03.96-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503509883.968282461","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"SKIP_SSL_VALIDATION":true}}
   2017-08-23T11:38:03.96-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503509883.968301535","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"IDLE_TIMEOUT":"30s"}}
   2017-08-23T11:38:03.96-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503509883.968318701","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"OMS_BATCH_TIME":"10s"}}
   2017-08-23T11:38:03.96-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503509883.968343258","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"CF_ENVIRONMENT":"cf"}}
   2017-08-23T11:38:03.96-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503509883.968365669","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"OMS_MAX_MSG_NUM_PER_BATCH":1000}}
   2017-08-23T11:38:03.96-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503509883.968382359","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"LOG_EVENT_COUNT":true}}
   2017-08-23T11:38:03.96-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503509883.968398094","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"LOG_EVENT_COUNT_INTERVAL":"1m0s"}}
   2017-08-23T11:38:03.96-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503509883.968406200","source":"oms-nozzle","message":"oms-nozzle.config EVENT_FILTER is nil. all events will be published","log_level":1,"data":{}}
   2017-08-23T11:38:03.96-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503509883.968469620","source":"oms-nozzle","message":"oms-nozzle.getting nozzle instance name","log_level":1,"data":{"name":"pid-20@5b89ddaf-10df-4e0b-6849-9cfa"}}
   2017-08-23T11:38:03.98-0600 [CELL/1] OUT Exit status 0
   2017-08-23T11:38:03.99-0600 [CELL/1] OUT Destroying container
   2017-08-23T11:38:04.00-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503509884.003629684","source":"oms-nozzle","message":"oms-nozzle.error creating cfclient","log_level":3,"data":{"error":"Could not get api /v2/info: json: cannot unmarshal number into Go value of type cfclient.Endpoint","trace":"goroutine 1 [running]:\ngithub.com/Azure/oms-log-analytics-firehose-nozzle/vendor/code.cloudfoundry.org/lager.(*logger).Fatal(0xc420018660, 0x55b75bca155d, 0x17, 0x55b75c18f760, 0xc4201041e0, 0x0, 0x0, 0x0)\n\t/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/vendor/code.cloudfoundry.org/lager/logger.go:131 +0xc9\ngithub.com/Azure/oms-log-analytics-firehose-nozzle/caching.(*Caching).Initialize(0xc420014bc0)\n\t/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/caching/caching.go:39 +0x5ae\ngithub.com/Azure/oms-log-analytics-firehose-nozzle/omsnozzle.(*OmsNozzle).Start(0xc420080750, 0xc42000de30, 0xc4200d0e00)\n\t/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/omsnozzle/oms_nozzle.go:67 +0x3d\nmain.main()\n\t/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/main.go:162 +0x1f13\n"}}
   �[31m2017-08-23T11:38:04.00-0600 [APP/PROC/WEB/0] ERR panic: Could not get api /v2/info: json: cannot unmarshal number into Go value of type cfclient.Endpoint�[0m
   �[31m2017-08-23T11:38:04.00-0600 [APP/PROC/WEB/0] ERR goroutine 1 [running]:�[0m
   �[31m2017-08-23T11:38:04.00-0600 [APP/PROC/WEB/0] ERR panic(0x55b75bf6f0a0, 0xc4201041e0)�[0m
   �[31m2017-08-23T11:38:04.00-0600 [APP/PROC/WEB/0] ERR 	/tmp/app/.cloudfoundry/0/go1.7.6/go/src/runtime/panic.go:500 +0x1a5�[0m
   �[31m2017-08-23T11:38:04.00-0600 [APP/PROC/WEB/0] ERR github.com/Azure/oms-log-analytics-firehose-nozzle/vendor/code.cloudfoundry.org/lager.(*logger).Fatal(0xc420018660, 0x55b75bca155d, 0x17, 0x55b75c18f760, 0xc4201041e0, 0x0, 0x0, 0x0)�[0m
   �[31m2017-08-23T11:38:04.00-0600 [APP/PROC/WEB/0] ERR 	/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/vendor/code.cloudfoundry.org/lager/logger.go:152 +0x41e�[0m
   �[31m2017-08-23T11:38:04.00-0600 [APP/PROC/WEB/0] ERR github.com/Azure/oms-log-analytics-firehose-nozzle/caching.(*Caching).Initialize(0xc420014bc0)�[0m
   �[31m2017-08-23T11:38:04.00-0600 [APP/PROC/WEB/0] ERR 	/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/caching/caching.go:39 +0x5ae�[0m
   �[31m2017-08-23T11:38:04.00-0600 [APP/PROC/WEB/0] ERR github.com/Azure/oms-log-analytics-firehose-nozzle/omsnozzle.(*OmsNozzle).Start(0xc420080750, 0xc42000de30, 0xc4200d0e00)�[0m
   �[31m2017-08-23T11:38:04.00-0600 [APP/PROC/WEB/0] ERR 	/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/omsnozzle/oms_nozzle.go:67 +0x3d�[0m
   �[31m2017-08-23T11:38:04.00-0600 [APP/PROC/WEB/0] ERR main.main()�[0m
   �[31m2017-08-23T11:38:04.00-0600 [APP/PROC/WEB/0] ERR 	/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/main.go:162 +0x1f13�[0m
   2017-08-23T11:38:04.00-0600 [APP/PROC/WEB/0] OUT Exit status 2
   2017-08-23T11:38:04.02-0600 [CELL/0] OUT Exit status 0
   2017-08-23T11:38:04.02-0600 [API/0] OUT Process has crashed with type: "web"
   2017-08-23T11:38:04.03-0600 [CELL/0] OUT Destroying container
   2017-08-23T11:38:04.03-0600 [API/0] OUT App instance exited with guid a1e48e70-b22d-4bc1-b1d1-d1451823782a payload: {"instance"=>"b3d1937c-daca-4481-49f9-3e62", "index"=>1, "reason"=>"CRASHED", "exit_description"=>"2 error(s) occurred:\n\n* Exited with status 2\n* cancelled", "crash_count"=>4, "crash_timestamp"=>1503509883998985699, "version"=>"6f568a4d-fce4-4879-a8e3-caeb22a0c267"}
   2017-08-23T11:38:04.06-0600 [API/0] OUT Process has crashed with type: "web"
   2017-08-23T11:38:04.07-0600 [API/0] OUT App instance exited with guid a1e48e70-b22d-4bc1-b1d1-d1451823782a payload: {"instance"=>"5b89ddaf-10df-4e0b-6849-9cfa", "index"=>0, "reason"=>"CRASHED", "exit_description"=>"2 error(s) occurred:\n\n* Exited with status 2\n* cancelled", "crash_count"=>4, "crash_timestamp"=>1503509884039440999, "version"=>"6f568a4d-fce4-4879-a8e3-caeb22a0c267"}
   2017-08-23T11:38:04.47-0600 [CELL/1] OUT Successfully destroyed container
   2017-08-23T11:38:04.51-0600 [CELL/0] OUT Successfully destroyed container
   2017-08-23T11:39:33.36-0600 [CELL/0] OUT Creating container
   2017-08-23T11:39:33.37-0600 [CELL/1] OUT Creating container
   2017-08-23T11:39:33.75-0600 [CELL/1] OUT Successfully created container
   2017-08-23T11:39:33.75-0600 [CELL/0] OUT Successfully created container
   2017-08-23T11:39:34.03-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503509974.038251162","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"OMS_POST_TIMEOUT":"10s"}}
   2017-08-23T11:39:34.03-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503509974.038713694","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"SKIP_SSL_VALIDATION":true}}
   2017-08-23T11:39:34.03-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503509974.038729906","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"IDLE_TIMEOUT":"30s"}}
   2017-08-23T11:39:34.03-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503509974.038788557","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"OMS_BATCH_TIME":"10s"}}
   2017-08-23T11:39:34.03-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503509974.039063692","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"CF_ENVIRONMENT":"cf"}}
   2017-08-23T11:39:34.03-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503509974.039081335","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"OMS_MAX_MSG_NUM_PER_BATCH":1000}}
   2017-08-23T11:39:34.03-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503509974.039090157","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"LOG_EVENT_COUNT":true}}
   2017-08-23T11:39:34.03-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503509974.039097309","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"LOG_EVENT_COUNT_INTERVAL":"1m0s"}}
   2017-08-23T11:39:34.03-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503509974.039103985","source":"oms-nozzle","message":"oms-nozzle.config EVENT_FILTER is nil. all events will be published","log_level":1,"data":{}}
   2017-08-23T11:39:34.03-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503509974.039147615","source":"oms-nozzle","message":"oms-nozzle.getting nozzle instance name","log_level":1,"data":{"name":"pid-14@d918cbcb-ce5b-4223-512b-eb07"}}
   2017-08-23T11:39:34.04-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503509974.047873259","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"OMS_POST_TIMEOUT":"10s"}}
   2017-08-23T11:39:34.04-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503509974.047947884","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"SKIP_SSL_VALIDATION":true}}
   2017-08-23T11:39:34.04-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503509974.047963858","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"IDLE_TIMEOUT":"30s"}}
   2017-08-23T11:39:34.04-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503509974.047972679","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"OMS_BATCH_TIME":"10s"}}
   2017-08-23T11:39:34.04-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503509974.048011541","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"CF_ENVIRONMENT":"cf"}}
   2017-08-23T11:39:34.04-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503509974.048020124","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"OMS_MAX_MSG_NUM_PER_BATCH":1000}}
   2017-08-23T11:39:34.04-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503509974.048029184","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"LOG_EVENT_COUNT":true}}
   2017-08-23T11:39:34.04-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503509974.048036814","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"LOG_EVENT_COUNT_INTERVAL":"1m0s"}}
   2017-08-23T11:39:34.04-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503509974.048044205","source":"oms-nozzle","message":"oms-nozzle.config EVENT_FILTER is nil. all events will be published","log_level":1,"data":{}}
   2017-08-23T11:39:34.04-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503509974.048091412","source":"oms-nozzle","message":"oms-nozzle.getting nozzle instance name","log_level":1,"data":{"name":"pid-19@04d45746-0029-4546-526e-59bd"}}
   2017-08-23T11:39:34.08-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503509974.082948685","source":"oms-nozzle","message":"oms-nozzle.error creating cfclient","log_level":3,"data":{"error":"Could not get api /v2/info: json: cannot unmarshal number into Go value of type cfclient.Endpoint","trace":"goroutine 1 [running]:\ngithub.com/Azure/oms-log-analytics-firehose-nozzle/vendor/code.cloudfoundry.org/lager.(*logger).Fatal(0xc4200186c0, 0x563812bcd55d, 0x17, 0x5638130bb760, 0xc4201f0050, 0x0, 0x0, 0x0)\n\t/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/vendor/code.cloudfoundry.org/lager/logger.go:131 +0xc9\ngithub.com/Azure/oms-log-analytics-firehose-nozzle/caching.(*Caching).Initialize(0xc420014bc0)\n\t/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/caching/caching.go:39 +0x5ae\ngithub.com/Azure/oms-log-analytics-firehose-nozzle/omsnozzle.(*OmsNozzle).Start(0xc42008e750, 0xc42000de40, 0xc4200dce00)\n\t/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/omsnozzle/oms_nozzle.go:67 +0x3d\nmain.main()\n\t/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/main.go:162 +0x1f13\n"}}
   �[31m2017-08-23T11:39:34.08-0600 [APP/PROC/WEB/1] ERR panic: Could not get api /v2/info: json: cannot unmarshal number into Go value of type cfclient.Endpoint�[0m
   �[31m2017-08-23T11:39:34.08-0600 [APP/PROC/WEB/1] ERR goroutine 1 [running]:�[0m
   �[31m2017-08-23T11:39:34.08-0600 [APP/PROC/WEB/1] ERR panic(0x563812e9b0a0, 0xc4201f0050)�[0m
   �[31m2017-08-23T11:39:34.08-0600 [APP/PROC/WEB/1] ERR 	/tmp/app/.cloudfoundry/0/go1.7.6/go/src/runtime/panic.go:500 +0x1a5�[0m
   �[31m2017-08-23T11:39:34.08-0600 [APP/PROC/WEB/1] ERR github.com/Azure/oms-log-analytics-firehose-nozzle/vendor/code.cloudfoundry.org/lager.(*logger).Fatal(0xc4200186c0, 0x563812bcd55d, 0x17, 0x5638130bb760, 0xc4201f0050, 0x0, 0x0, 0x0)�[0m
   �[31m2017-08-23T11:39:34.08-0600 [APP/PROC/WEB/1] ERR 	/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/vendor/code.cloudfoundry.org/lager/logger.go:152 +0x41e�[0m
   �[31m2017-08-23T11:39:34.08-0600 [APP/PROC/WEB/1] ERR github.com/Azure/oms-log-analytics-firehose-nozzle/caching.(*Caching).Initialize(0xc420014bc0)�[0m
   �[31m2017-08-23T11:39:34.08-0600 [APP/PROC/WEB/1] ERR 	/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/caching/caching.go:39 +0x5ae�[0m
   �[31m2017-08-23T11:39:34.08-0600 [APP/PROC/WEB/1] ERR github.com/Azure/oms-log-analytics-firehose-nozzle/omsnozzle.(*OmsNozzle).Start(0xc42008e750, 0xc42000de40, 0xc4200dce00)�[0m
   �[31m2017-08-23T11:39:34.08-0600 [APP/PROC/WEB/1] ERR 	/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/omsnozzle/oms_nozzle.go:67 +0x3d�[0m
   �[31m2017-08-23T11:39:34.08-0600 [APP/PROC/WEB/1] ERR main.main()�[0m
   �[31m2017-08-23T11:39:34.08-0600 [APP/PROC/WEB/1] ERR 	/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/main.go:162 +0x1f13�[0m
   2017-08-23T11:39:34.08-0600 [APP/PROC/WEB/1] OUT Exit status 2
   2017-08-23T11:39:34.09-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503509974.095694542","source":"oms-nozzle","message":"oms-nozzle.error creating cfclient","log_level":3,"data":{"error":"Could not get api /v2/info: json: cannot unmarshal number into Go value of type cfclient.Endpoint","trace":"goroutine 1 [running]:\ngithub.com/Azure/oms-log-analytics-firehose-nozzle/vendor/code.cloudfoundry.org/lager.(*logger).Fatal(0xc4200186c0, 0x558a1869255d, 0x17, 0x558a18b80760, 0xc4201c4050, 0x0, 0x0, 0x0)\n\t/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/vendor/code.cloudfoundry.org/lager/logger.go:131 +0xc9\ngithub.com/Azure/oms-log-analytics-firehose-nozzle/caching.(*Caching).Initialize(0xc420014bc0)\n\t/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/caching/caching.go:39 +0x5ae\ngithub.com/Azure/oms-log-analytics-firehose-nozzle/omsnozzle.(*OmsNozzle).Start(0xc42008e750, 0xc42000de40, 0xc4200dce00)\n\t/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/omsnozzle/oms_nozzle.go:67 +0x3d\nmain.main()\n\t/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/main.go:162 +0x1f13\n"}}
   �[31m2017-08-23T11:39:34.09-0600 [APP/PROC/WEB/0] ERR panic: Could not get api /v2/info: json: cannot unmarshal number into Go value of type cfclient.Endpoint�[0m
   �[31m2017-08-23T11:39:34.09-0600 [APP/PROC/WEB/0] ERR goroutine 1 [running]:�[0m
   �[31m2017-08-23T11:39:34.09-0600 [APP/PROC/WEB/0] ERR panic(0x558a189600a0, 0xc4201c4050)�[0m
   �[31m2017-08-23T11:39:34.09-0600 [APP/PROC/WEB/0] ERR 	/tmp/app/.cloudfoundry/0/go1.7.6/go/src/runtime/panic.go:500 +0x1a5�[0m
   �[31m2017-08-23T11:39:34.09-0600 [APP/PROC/WEB/0] ERR github.com/Azure/oms-log-analytics-firehose-nozzle/vendor/code.cloudfoundry.org/lager.(*logger).Fatal(0xc4200186c0, 0x558a1869255d, 0x17, 0x558a18b80760, 0xc4201c4050, 0x0, 0x0, 0x0)�[0m
   �[31m2017-08-23T11:39:34.09-0600 [APP/PROC/WEB/0] ERR 	/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/vendor/code.cloudfoundry.org/lager/logger.go:152 +0x41e�[0m
   �[31m2017-08-23T11:39:34.09-0600 [APP/PROC/WEB/0] ERR github.com/Azure/oms-log-analytics-firehose-nozzle/caching.(*Caching).Initialize(0xc420014bc0)�[0m
   �[31m2017-08-23T11:39:34.09-0600 [APP/PROC/WEB/0] ERR 	/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/caching/caching.go:39 +0x5ae�[0m
   �[31m2017-08-23T11:39:34.09-0600 [APP/PROC/WEB/0] ERR github.com/Azure/oms-log-analytics-firehose-nozzle/omsnozzle.(*OmsNozzle).Start(0xc42008e750, 0xc42000de40, 0xc4200dce00)�[0m
   �[31m2017-08-23T11:39:34.09-0600 [APP/PROC/WEB/0] ERR 	/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/omsnozzle/oms_nozzle.go:67 +0x3d�[0m
   �[31m2017-08-23T11:39:34.09-0600 [APP/PROC/WEB/0] ERR main.main()�[0m
   �[31m2017-08-23T11:39:34.09-0600 [APP/PROC/WEB/0] ERR 	/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/main.go:162 +0x1f13�[0m
   2017-08-23T11:39:34.10-0600 [APP/PROC/WEB/0] OUT Exit status 2
   2017-08-23T11:39:34.10-0600 [CELL/1] OUT Exit status 0
   2017-08-23T11:39:34.11-0600 [CELL/0] OUT Exit status 0
   2017-08-23T11:39:34.11-0600 [CELL/1] OUT Destroying container
   2017-08-23T11:39:34.12-0600 [CELL/0] OUT Destroying container
   2017-08-23T11:39:34.14-0600 [API/0] OUT Process has crashed with type: "web"
   2017-08-23T11:39:34.15-0600 [API/0] OUT App instance exited with guid a1e48e70-b22d-4bc1-b1d1-d1451823782a payload: {"instance"=>"04d45746-0029-4546-526e-59bd", "index"=>1, "reason"=>"CRASHED", "exit_description"=>"2 error(s) occurred:\n\n* Exited with status 2\n* cancelled", "crash_count"=>5, "crash_timestamp"=>1503509974120302399, "version"=>"6f568a4d-fce4-4879-a8e3-caeb22a0c267"}
   2017-08-23T11:39:34.17-0600 [API/1] OUT Process has crashed with type: "web"
   2017-08-23T11:39:34.18-0600 [API/1] OUT App instance exited with guid a1e48e70-b22d-4bc1-b1d1-d1451823782a payload: {"instance"=>"d918cbcb-ce5b-4223-512b-eb07", "index"=>0, "reason"=>"CRASHED", "exit_description"=>"2 error(s) occurred:\n\n* Exited with status 2\n* cancelled", "crash_count"=>5, "crash_timestamp"=>1503509974134852099, "version"=>"6f568a4d-fce4-4879-a8e3-caeb22a0c267"}
   2017-08-23T11:39:34.57-0600 [CELL/1] OUT Successfully destroyed container
   2017-08-23T11:39:34.62-0600 [CELL/0] OUT Successfully destroyed container
   2017-08-23T11:42:03.54-0600 [CELL/1] OUT Creating container
   2017-08-23T11:42:03.54-0600 [CELL/0] OUT Creating container
   2017-08-23T11:42:03.91-0600 [CELL/1] OUT Successfully created container
   2017-08-23T11:42:03.93-0600 [CELL/0] OUT Successfully created container
   2017-08-23T11:42:04.21-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503510124.211110353","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"OMS_POST_TIMEOUT":"10s"}}
   2017-08-23T11:42:04.21-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503510124.211193800","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"SKIP_SSL_VALIDATION":true}}
   2017-08-23T11:42:04.21-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503510124.211219311","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"IDLE_TIMEOUT":"30s"}}
   2017-08-23T11:42:04.21-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503510124.211343050","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"OMS_BATCH_TIME":"10s"}}
   2017-08-23T11:42:04.21-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503510124.211359501","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"CF_ENVIRONMENT":"cf"}}
   2017-08-23T11:42:04.21-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503510124.211366892","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"OMS_MAX_MSG_NUM_PER_BATCH":1000}}
   2017-08-23T11:42:04.21-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503510124.211381435","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"LOG_EVENT_COUNT":true}}
   2017-08-23T11:42:04.21-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503510124.211394787","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"LOG_EVENT_COUNT_INTERVAL":"1m0s"}}
   2017-08-23T11:42:04.21-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503510124.211401463","source":"oms-nozzle","message":"oms-nozzle.config EVENT_FILTER is nil. all events will be published","log_level":1,"data":{}}
   2017-08-23T11:42:04.21-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503510124.211454630","source":"oms-nozzle","message":"oms-nozzle.getting nozzle instance name","log_level":1,"data":{"name":"pid-19@c6ccb8f3-21fb-4baa-7780-f990"}}
   2017-08-23T11:42:04.24-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503510124.243658066","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"OMS_POST_TIMEOUT":"10s"}}
   2017-08-23T11:42:04.24-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503510124.243902445","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"SKIP_SSL_VALIDATION":true}}
   2017-08-23T11:42:04.24-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503510124.243919849","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"IDLE_TIMEOUT":"30s"}}
   2017-08-23T11:42:04.24-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503510124.243936300","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"OMS_BATCH_TIME":"10s"}}
   2017-08-23T11:42:04.24-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503510124.243957520","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"CF_ENVIRONMENT":"cf"}}
   2017-08-23T11:42:04.24-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503510124.243982077","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"OMS_MAX_MSG_NUM_PER_BATCH":1000}}
   2017-08-23T11:42:04.24-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503510124.244013309","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"LOG_EVENT_COUNT":true}}
   2017-08-23T11:42:04.24-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503510124.244047165","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"LOG_EVENT_COUNT_INTERVAL":"1m0s"}}
   2017-08-23T11:42:04.24-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503510124.244056225","source":"oms-nozzle","message":"oms-nozzle.config EVENT_FILTER is nil. all events will be published","log_level":1,"data":{}}
   2017-08-23T11:42:04.24-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503510124.244127035","source":"oms-nozzle","message":"oms-nozzle.getting nozzle instance name","log_level":1,"data":{"name":"pid-19@c1f6cb17-9ea9-4cc1-49b5-d501"}}
   2017-08-23T11:42:04.24-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503510124.248155594","source":"oms-nozzle","message":"oms-nozzle.error creating cfclient","log_level":3,"data":{"error":"Could not get api /v2/info: json: cannot unmarshal number into Go value of type cfclient.Endpoint","trace":"goroutine 1 [running]:\ngithub.com/Azure/oms-log-analytics-firehose-nozzle/vendor/code.cloudfoundry.org/lager.(*logger).Fatal(0xc420018660, 0x55ed1f1b355d, 0x17, 0x55ed1f6a1760, 0xc420156b00, 0x0, 0x0, 0x0)\n\t/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/vendor/code.cloudfoundry.org/lager/logger.go:131 +0xc9\ngithub.com/Azure/oms-log-analytics-firehose-nozzle/caching.(*Caching).Initialize(0xc420014bc0)\n\t/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/caching/caching.go:39 +0x5ae\ngithub.com/Azure/oms-log-analytics-firehose-nozzle/omsnozzle.(*OmsNozzle).Start(0xc42008e750, 0xc42000de30, 0xc4200dce00)\n\t/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/omsnozzle/oms_nozzle.go:67 +0x3d\nmain.main()\n\t/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/main.go:162 +0x1f13\n"}}
   �[31m2017-08-23T11:42:04.25-0600 [APP/PROC/WEB/1] ERR panic: Could not get api /v2/info: json: cannot unmarshal number into Go value of type cfclient.Endpoint�[0m
   �[31m2017-08-23T11:42:04.25-0600 [APP/PROC/WEB/1] ERR goroutine 1 [running]:�[0m
   �[31m2017-08-23T11:42:04.25-0600 [APP/PROC/WEB/1] ERR panic(0x55ed1f4810a0, 0xc420156b00)�[0m
   �[31m2017-08-23T11:42:04.25-0600 [APP/PROC/WEB/1] ERR 	/tmp/app/.cloudfoundry/0/go1.7.6/go/src/runtime/panic.go:500 +0x1a5�[0m
   �[31m2017-08-23T11:42:04.25-0600 [APP/PROC/WEB/1] ERR github.com/Azure/oms-log-analytics-firehose-nozzle/vendor/code.cloudfoundry.org/lager.(*logger).Fatal(0xc420018660, 0x55ed1f1b355d, 0x17, 0x55ed1f6a1760, 0xc420156b00, 0x0, 0x0, 0x0)�[0m
   �[31m2017-08-23T11:42:04.25-0600 [APP/PROC/WEB/1] ERR 	/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/vendor/code.cloudfoundry.org/lager/logger.go:152 +0x41e�[0m
   �[31m2017-08-23T11:42:04.25-0600 [APP/PROC/WEB/1] ERR github.com/Azure/oms-log-analytics-firehose-nozzle/caching.(*Caching).Initialize(0xc420014bc0)�[0m
   �[31m2017-08-23T11:42:04.25-0600 [APP/PROC/WEB/1] ERR 	/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/caching/caching.go:39 +0x5ae�[0m
   �[31m2017-08-23T11:42:04.25-0600 [APP/PROC/WEB/1] ERR github.com/Azure/oms-log-analytics-firehose-nozzle/omsnozzle.(*OmsNozzle).Start(0xc42008e750, 0xc42000de30, 0xc4200dce00)�[0m
   �[31m2017-08-23T11:42:04.25-0600 [APP/PROC/WEB/1] ERR 	/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/omsnozzle/oms_nozzle.go:67 +0x3d�[0m
   �[31m2017-08-23T11:42:04.25-0600 [APP/PROC/WEB/1] ERR main.main()�[0m
   �[31m2017-08-23T11:42:04.25-0600 [APP/PROC/WEB/1] ERR 	/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/main.go:162 +0x1f13�[0m
   2017-08-23T11:42:04.25-0600 [APP/PROC/WEB/1] OUT Exit status 2
   2017-08-23T11:42:04.26-0600 [CELL/1] OUT Exit status 0
   2017-08-23T11:42:04.27-0600 [CELL/1] OUT Destroying container
   2017-08-23T11:42:04.27-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503510124.275655746","source":"oms-nozzle","message":"oms-nozzle.error creating cfclient","log_level":3,"data":{"error":"Could not get api /v2/info: json: cannot unmarshal number into Go value of type cfclient.Endpoint","trace":"goroutine 1 [running]:\ngithub.com/Azure/oms-log-analytics-firehose-nozzle/vendor/code.cloudfoundry.org/lager.(*logger).Fatal(0xc420018660, 0x5643c60ea55d, 0x17, 0x5643c65d8760, 0xc42000da10, 0x0, 0x0, 0x0)\n\t/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/vendor/code.cloudfoundry.org/lager/logger.go:131 +0xc9\ngithub.com/Azure/oms-log-analytics-firehose-nozzle/caching.(*Caching).Initialize(0xc420110280)\n\t/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/caching/caching.go:39 +0x5ae\ngithub.com/Azure/oms-log-analytics-firehose-nozzle/omsnozzle.(*OmsNozzle).Start(0xc4201205a0, 0xc4200fc478, 0xc42011c380)\n\t/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/omsnozzle/oms_nozzle.go:67 +0x3d\nmain.main()\n\t/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/main.go:162 +0x1f13\n"}}
   �[31m2017-08-23T11:42:04.27-0600 [APP/PROC/WEB/0] ERR panic: Could not get api /v2/info: json: cannot unmarshal number into Go value of type cfclient.Endpoint�[0m
   �[31m2017-08-23T11:42:04.27-0600 [APP/PROC/WEB/0] ERR goroutine 1 [running]:�[0m
   �[31m2017-08-23T11:42:04.27-0600 [APP/PROC/WEB/0] ERR panic(0x5643c63b80a0, 0xc42000da10)�[0m
   �[31m2017-08-23T11:42:04.27-0600 [APP/PROC/WEB/0] ERR 	/tmp/app/.cloudfoundry/0/go1.7.6/go/src/runtime/panic.go:500 +0x1a5�[0m
   �[31m2017-08-23T11:42:04.27-0600 [APP/PROC/WEB/0] ERR github.com/Azure/oms-log-analytics-firehose-nozzle/vendor/code.cloudfoundry.org/lager.(*logger).Fatal(0xc420018660, 0x5643c60ea55d, 0x17, 0x5643c65d8760, 0xc42000da10, 0x0, 0x0, 0x0)�[0m
   �[31m2017-08-23T11:42:04.27-0600 [APP/PROC/WEB/0] ERR 	/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/vendor/code.cloudfoundry.org/lager/logger.go:152 +0x41e�[0m
   �[31m2017-08-23T11:42:04.27-0600 [APP/PROC/WEB/0] ERR github.com/Azure/oms-log-analytics-firehose-nozzle/caching.(*Caching).Initialize(0xc420110280)�[0m
   �[31m2017-08-23T11:42:04.27-0600 [APP/PROC/WEB/0] ERR 	/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/caching/caching.go:39 +0x5ae�[0m
   �[31m2017-08-23T11:42:04.27-0600 [APP/PROC/WEB/0] ERR github.com/Azure/oms-log-analytics-firehose-nozzle/omsnozzle.(*OmsNozzle).Start(0xc4201205a0, 0xc4200fc478, 0xc42011c380)�[0m
   �[31m2017-08-23T11:42:04.27-0600 [APP/PROC/WEB/0] ERR 	/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/omsnozzle/oms_nozzle.go:67 +0x3d�[0m
   �[31m2017-08-23T11:42:04.27-0600 [APP/PROC/WEB/0] ERR main.main()�[0m
   �[31m2017-08-23T11:42:04.27-0600 [APP/PROC/WEB/0] ERR 	/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/main.go:162 +0x1f13�[0m
   2017-08-23T11:42:04.27-0600 [APP/PROC/WEB/0] OUT Exit status 2
   2017-08-23T11:42:04.29-0600 [CELL/0] OUT Exit status 0
   2017-08-23T11:42:04.30-0600 [API/0] OUT Process has crashed with type: "web"
   2017-08-23T11:42:04.30-0600 [CELL/0] OUT Destroying container
   2017-08-23T11:42:04.31-0600 [API/0] OUT App instance exited with guid a1e48e70-b22d-4bc1-b1d1-d1451823782a payload: {"instance"=>"c6ccb8f3-21fb-4baa-7780-f990", "index"=>1, "reason"=>"CRASHED", "exit_description"=>"2 error(s) occurred:\n\n* Exited with status 2\n* cancelled", "crash_count"=>6, "crash_timestamp"=>1503510124283089599, "version"=>"6f568a4d-fce4-4879-a8e3-caeb22a0c267"}
   2017-08-23T11:42:04.34-0600 [API/1] OUT Process has crashed with type: "web"
   2017-08-23T11:42:04.35-0600 [API/1] OUT App instance exited with guid a1e48e70-b22d-4bc1-b1d1-d1451823782a payload: {"instance"=>"c1f6cb17-9ea9-4cc1-49b5-d501", "index"=>0, "reason"=>"CRASHED", "exit_description"=>"2 error(s) occurred:\n\n* Exited with status 2\n* cancelled", "crash_count"=>6, "crash_timestamp"=>1503510124310964299, "version"=>"6f568a4d-fce4-4879-a8e3-caeb22a0c267"}
   2017-08-23T11:42:04.73-0600 [CELL/1] OUT Successfully destroyed container
   2017-08-23T11:42:04.78-0600 [CELL/0] OUT Successfully destroyed container

I used the latest commit from master.

Failed to compile due to the removal of the support for Go 1.7.x in go_buildpack

       FAILED  

          2018-10-15T05:04:53.99+0000 [STG/0] OUT -----> Go Buildpack version 1.8.28  
          2018-10-15T05:04:53.99+0000 [STG/0] OUT        [31;1m**WARNING**[0m buildpack version changed from 1.8.27 to 1.8.28  
          2018-10-15T05:04:54.66+0000 [STG/0] OUT        [31;1m**ERROR**[0m Unable to determine Go version to install: no match found for 1.7.x in [1.8.6 1.8.7 1.9.6 1.9.7 1.10.3 1.10.4 1.11 1.11.1]  
          2018-10-15T05:04:54.67+0000 [STG/0] ERR Failed to compile droplet: Failed to run all supply scripts: exit status 16  

This repo is missing important files

There are important files that Microsoft projects should all have that are not present in this repository. A pull request has been opened to add the missing file(s). When the pr is merged this issue will be closed automatically.

Microsoft teams can learn more about this effort and share feedback within the open source guidance available internally.

Merge this pull request

Re-generate nozzle tile to use xenial stemcell

Nozzle app is crashing with DNS lookup issues

2020-01-20T15:50:59.71-0800 [APP/PROC/WEB/4] ERR panic: Could not get api /v2/info: Get https://api.system.prod.**.**.com/v2/info: dial tcp: lookup api.system.prod.stratus.albertsons.com on 169.254.0.2:53: read udp 10.255.79.151:42306->169.254.0.2:53: i/o timeout
2020-01-20T15:50:59.71-0800 [APP/PROC/WEB/4] ERR goroutine 1 [running]:
2020-01-20T15:50:59.71-0800 [APP/PROC/WEB/4] ERR github.com/Azure/oms-log-analytics-firehose-nozzle/vendor/code.cloudfoundry.org/lager.(*logger).Fatal(0xc4201204e0, 0x5595c357001b, 0x17, 0x5595c38f8940, 0xc420160850, 0x0, 0x0, 0x0)
2020-01-20T15:50:59.71-0800 [APP/PROC/WEB/4] ERR /tmp/gobuildpack.gopath476918737/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/vendor/code.cloudfoundry.org/lager/logger.go:152 +0x4d4
2020-01-20T15:50:59.71-0800 [APP/PROC/WEB/4] ERR github.com/Azure/oms-log-analytics-firehose-nozzle/caching.(*Caching).Initialize(0xc420172540, 0x5595c31f4101)
2020-01-20T15:50:59.71-0800 [APP/PROC/WEB/4] ERR /tmp/gobuildpack.gopath476918737/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/caching/caching.go:99 +0x39f
2020-01-20T15:50:59.71-0800 [APP/PROC/WEB/4] ERR github.com/Azure/oms-log-analytics-firehose-nozzle/omsnozzle.(*OmsNozzle).Start(0xc420186000, 0xc420156320, 0xc420184000)
2020-01-20T15:50:59.71-0800 [APP/PROC/WEB/4] ERR /tmp/gobuildpack.gopath476918737/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/omsnozzle/oms_nozzle.go:69 +0x3f
2020-01-20T15:50:59.71-0800 [APP/PROC/WEB/4] ERR main.main()
2020-01-20T15:50:59.71-0800 [APP/PROC/WEB/4] ERR /tmp/gobuildpack.gopath476918737/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/main.go:187 +0x1af2

Nozzle App is crashing constantly

Nozzle App is crashing constantly we deployed the newer version of PCF tile 1.4.2. App is not stable and crashing often.

ERROR:
2020-11-24T11:57:58.386+05:30 [APP/PROC/WEB/2] [OUT] {'timestamp':'1606199278.385876656','source':'oms-nozzle','message':'oms-nozzle.Error while reading from the firehose','log_level':2,'data':{'error':'websocket: close 1006 (abnormal closure): unexpected EOF'}}

2020-11-30T14:52:42.622+05:30 [APP/PROC/WEB/12] [OUT] {"timestamp":"1606728162.622236252","source":"oms-nozzle","message":"oms-nozzle.App info not found for GUID","log_level":1,"data":{"guid":"5ce0c225-ccf4-4389-97a8-6a43c1ec7849"}}
2020-11-30T14:52:42.626+05:30 [APP/PROC/WEB/0] [OUT] {"timestamp":"1606728162.626188993","source":"oms-nozzle","message":"oms-nozzle.adding to app info cache","log_level":0,"data":{"guid":"","info":{"name":"","org":"","orgId":"","space":"","spaceId":"","monitored":true}}}
2020-11-30T14:52:42.626+05:30 [APP/PROC/WEB/0] [OUT] {"timestamp":"1606728162.626352787","source":"oms-nozzle","message":"oms-nozzle.App info not found for GUID","log_level":1,"data":{"guid":"5ce0c225-ccf4-4389-97a8-6a43c1ec7849"}}
2020-11-30T14:52:42.642+05:30 [APP/PROC/WEB/3] [OUT] {"timestamp":"1606728162.641838789","source":"oms-nozzle","message":"oms-nozzle.adding to app info cache","log_level":0,"data":{"guid":"","info":{"name":"","org":"","orgId":"","space":"","spaceId":"","monitored":true}}}
2020-11-30T14:52:42.642+05:30 [APP/PROC/WEB/3] [OUT] {"timestamp":"1606728162.641998053","source":"oms-nozzle","message":"oms-nozzle.App info not found for GUID","log_level":1,"data":{"guid":"5ce0c225-ccf4-4389-97a8-6a43c1ec7849"}}
2020-11-30T14:52:42.783+05:30 [APP/PROC/WEB/10] [OUT] {"timestamp":"1606728162.783232689","source":"oms-nozzle","message":"oms-nozzle.adding to app info cache","log_level":0,"data":{"guid":"","info":{"name":"","org":"","orgId":"","space":"","spaceId":"","monitored":true}}}
2020-11-30T14:52:42.783+05:30 [APP/PROC/WEB/10] [OUT] {"timestamp":"1606728162.783447504","source":"oms-nozzle","message":"oms-nozzle.App info not found for GUID","log_level":1,"data":{"guid":"8b84e70c-62f8-4dab-9326-f4e6e23a3eda"}}

Support app rename

Hello,

The Nozzle does not seem to support the renaming of a PCF application as its name is cached once on the start of the [Nozzle] instance or when the first message goes through it and is never invalidated. This behavior causes a mismatch in the ApplicationName field of indexed logs when you do blue-green deployments on PCF.

Steps to reproduce

1. Create and start an application named appA that emits some logs
2. Renaming this app into appB
3. Check logs in the OMS Workspace, they are still incorrectly indexed with appA as the ApplicationName

How to confirm if there's no problem on Nozzle in case of log message lost

I'm facing an issue that some of log messages generated by "cf push"-ed application might be lost when I try to confirm them from Azure OMS by using KUSTO query.

Here's a detail regarding my facing issue:
1, An application, hereafter calling it as A, generates some logs and these are forwarded to OMS via Nozzle.
2, I assume that all generated logs from A are preserved at least when they pass through Nozzle, as I confirmed them with "cf app-nozzle A" plugin.
3, However, when I try to find them through Azure OMS with KUSTO query, some of them are lost. (=wasn't displayed as the result of query.)

As of now, what I've done is:
1, Ask Microsoft to investigate it from OMS side, and I shared syslog of VMs which Nozzle is deployed on because of their request, the results of "cf app-nozzle A" and that of "cf logs A".
They can't specify the root cause so far.
2, They recommended me to raise a case here in GitHub, and I'm just issuing now.

My questions are as follows:
1, I expect that I can affirm there's no problem in or inside Nozzle as all logs are preserved when they're passing through Nozzle. However, I should confirm if there's other way to confirm there's no problem as well.
Do you have any log that would help my investigation?
2, Do you have any idea which logs I should check regarding this case?

Thank you in advance.

supporting PCF 2.0

will this nozzle be "officially" compatible for PCF 2.0 (meaning be downloadable as "depends on PCF 2.0.x" any time soon?

Log Analytics Nozzle should support Azure GovCloud

I have a customer looking to use the nozzle on Azure GovCloud. Currently the Nozzle does not support Azure GovCloud endpoints for the Log Analytics Workspace.

@nsagoo-pivotal for additional details

Add more app data in messages to Log Analytics

Currently in LogMessage and ContainerMetric, we add the additional information of application name.
We can also consider to add other app data, e.g. space name/id and organization name/id, .etc

Any plan to support pcf 1.12?

Add an exclusion list property for excluding apps belonging to a set of orgs and spaces

We have a use-case for separating logs from different spaces of a CF foundation.

For this, we use two instances of the Nozzle (replicating the Azure Log Analytics Nozzle tile), and we apply two different (but complementary) configurations to each one.

For example, let's say we have a Foundation with an ACME org, and spaces like development, staging, preprod and prod. When we need to isolate the logs for prod space, we setup this space_whitelist for the second Nozzle:

space_whitelist:
  ACME.prod

But for the main Nozzle, we must built a kind of “catch-all” space_whitelist to capture everything else.

space_whitelist:
  system.*,
  eventhub-firehose.*,
  credhub-service-broker-org.*,
  ACME.development,
  ACME.staging,
  ACME.preprod

The drawback is that any time anyone (or anything) adds a new org or a new space to the ACME org, then we must re-configure the main Nozzle to include it in the “catch-all” space_whitelist of the main Nozzle. This breaks the self-service nature of Cloud Foundry, that lets developers create orgs and spaces without any extra toil for platform engineers.

The missing feature that we need here would be a space_exclusion_list that would do the opposite of space_whitelist. Then, the config of the main Nozzle would be super easy.

space_exclusion_list:
  ACME.prod

And we wouldn't have any new toil anytime a new org or a new space is created in the ACME org.

Don't hesitate to ask for any further information in order to fully understand this feature request.

Best,
Benjamin

Filtering specific metric keys

Regarding to the filtering,

EVENT_FILTER : Event types to be filtered out. The format is a comma separated list, valid event types are METRIC,LOG,HTTP

This setting will stop the whole inject to each table (e.g. CF_ValueMetric_CL ) in Log analytics.
I have many, however not all, metric keys that are not used in ValueMetric table.
In order to reduce the azure usage cost from the log injection, we would like to filter some specific metric keys.

Is there any way to do that?

Slow consumer alert from traffic controller might has been changed

Previous alerts in two ways:
the nozzle receives a WebSocket close error with error code ClosePolicyViolation (1008)
the nozzle receives a CounterEvent with the name TruncatingBuffer.DroppedMessages

The latest document has changed to:
If a nozzle falls behind, Loggregator alerts the nozzle in two ways:
TruncatingBuffer alerts: If the nozzle consumes messages more slowly than they are produced, the Loggregator system may drop messages. In this case, Loggregator sends the log message, TB: Output channel too full. Dropped N messages, where N is the number of dropped messages. Loggregator also emits a CounterEvent with the name doppler_proxy.slow_consumer. The nozzle receives both messages from the Firehose, alerting the operator to the performance issue.

Failed to compile due to no support for Go 1.10.0 in the go_buildpack

Hey all it looks like the go version needs to be updated as the build pack no longer supports go 1.10.0.

I am installing the nozzle as a tile in ops-manager on Pivotal Cloud Foundry. Here are the relevant logs.

             2019-08-20T05:10:20.86+0000 [STG/0] OUT -----> Installing dep 0.5.4  
             2019-08-20T05:10:20.86+0000 [STG/0] OUT        Copy [/tmp/buildpacks/52a307ca94edfbb724794d48ed1a7c48/dependencies/f1900fcb2de60a12ea6743ecf05e14d2/dep-v0.5.4-linux-x64-cflinuxfs3-79b3ab9e.tgz]  
             2019-08-20T05:10:21.10+0000 [STG/0] OUT        [31;1m**ERROR**[0m Unable to determine Go version to install: no match found for 1.10.x in [1.11.11 1.11.12 1.12.6 1.12.7]  
             2019-08-20T05:10:21.11+0000 [STG/0] ERR Failed to compile droplet: Failed to run all supply scripts: exit status 16  

incompatible type (incompatible type for method 'Debug' (different parameter types)), in main.go

Hi.
I was not able run the main executable, on Ubuntu 20.04 x86_64:

$ go run main.go

command-line-arguments

./main.go:169:39: error: argument 1 has incompatible type
./main.go:169:71: error: argument 3 has incompatible type (incompatible type for method 'Debug' (different parameter types))
./main.go:171:76: error: argument 4 has incompatible type (incompatible type for method 'Debug' (different parameter types))
./main.go:184:38: error: argument 1 has incompatible type
./main.go:184:54: error: argument 2 has incompatible type (incompatible type for method 'Debug' (different parameter types))
./main.go:185:35: error: argument 1 has incompatible type (incompatible type for method 'Debug' (different parameter types))

$ go version
go version go1.15.2 gollvm LLVM 12.0.0git linux/amd64

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/oceanfish81/.cache/go-build"
GOENV="/home/oceanfish81/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/oceanfish81/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/oceanfish81/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/oceanfish81/gollvm_dist"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/oceanfish81/gollvm_dist/tools"
GCCGO="/home/oceanfish81/gollvm_dist/bin/llvm-goc"
AR="ar"
CC="/usr/bin/clang"
CXX="/usr/bin/clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build809405926=/tmp/go-build -gno-record-gcc-switches -funwind-tables"