azure / oms-log-analytics-firehose-nozzle Goto Github PK

View Code? Open in Web Editor NEW

11.0 11.0 14.0 1.56 MB

Microsoft Azure Log Analytics Nozzle

License: Apache License 2.0

Go 99.21% Shell 0.79%

oms-log-analytics-firehose-nozzle's Issues

Failed to compile due to no support for Go 1.10.0 in the go_buildpack

Hey all it looks like the go version needs to be updated as the build pack no longer supports go 1.10.0.

I am installing the nozzle as a tile in ops-manager on Pivotal Cloud Foundry. Here are the relevant logs.

             2019-08-20T05:10:20.86+0000 [STG/0] OUT -----&gt; Installing dep 0.5.4  
             2019-08-20T05:10:20.86+0000 [STG/0] OUT        Copy [/tmp/buildpacks/52a307ca94edfbb724794d48ed1a7c48/dependencies/f1900fcb2de60a12ea6743ecf05e14d2/dep-v0.5.4-linux-x64-cflinuxfs3-79b3ab9e.tgz]  
             2019-08-20T05:10:21.10+0000 [STG/0] OUT        [31;1m**ERROR**[0m Unable to determine Go version to install: no match found for 1.10.x in [1.11.11 1.11.12 1.12.6 1.12.7]  
             2019-08-20T05:10:21.11+0000 [STG/0] ERR Failed to compile droplet: Failed to run all supply scripts: exit status 16

App log Filtering by Orgs or/and Spaces

We have a CF installation that is used by many teams, we would only want to ship the logs for the apps we care about into OMS. Is it possible to do some filtering by space or app?

Unmarshal error

There is an error with the nozzle when trying to unmarshal Doppler data.

Retrieving logs for app �[36;1mazure-log-analytics-nozzle-1.1.0�[0m in org �[36;1msystem�[0m / space �[36;1mazure-log-analytics-nozzle-space�[0m as �[36;1madmin�[0m...

   2017-08-23T09:25:39.41-0600 [CELL/1] OUT Destroying container
   2017-08-23T09:25:39.45-0600 [CELL/1] OUT Creating container
   2017-08-23T09:25:39.85-0600 [CELL/1] OUT Successfully destroyed container
   �[31m2017-08-23T09:25:40.31-0600 [APP/PROC/WEB/1] ERR oms-log-analytics-firehose-nozzle: error: time: missing unit in duration 30, try --help�[0m
   2017-08-23T09:25:40.44-0600 [CELL/0] OUT Creating container
   2017-08-23T09:25:41.60-0600 [CELL/0] OUT Successfully destroyed container
   2017-08-23T09:26:25.31-0600 [API/0] OUT Process has crashed with type: "web"
   2017-08-23T09:27:55.19-0600 [CELL/1] OUT Successfully created container
   2017-08-23T09:27:55.54-0600 [CELL/1] OUT Destroying container
   2017-08-23T09:27:55.68-0600 [API/0] OUT App instance exited with guid a1e48e70-b22d-4bc1-b1d1-d1451823782a payload: {"instance"=>"3de8840e-fcac-4549-42e6-5c56", "index"=>0, "reason"=>"CRASHED", "exit_description"=>"2 error(s) occurred:\n\n* Exited with status 1\n* cancelled", "crash_count"=>5, "crash_timestamp"=>1503502075642244799, "version"=>"46904aef-03bb-4cf1-8136-d856c77ccf0f"}
   2017-08-23T09:27:56.05-0600 [CELL/1] OUT Successfully destroyed container
   2017-08-23T09:30:24.86-0600 [CELL/0] OUT Creating container
   2017-08-23T09:30:25.54-0600 [CELL/0] OUT Exit status 0
   2017-08-23T09:30:25.55-0600 [CELL/0] OUT Destroying container
   2017-08-23T09:30:26.02-0600 [CELL/0] OUT Successfully destroyed container
   2017-08-23T09:34:55.52-0600 [CELL/1] OUT Successfully created container
   2017-08-23T09:34:55.84-0600 [CELL/1] OUT Exit status 0
   2017-08-23T09:43:26.17-0600 [CELL/0] OUT Successfully created container
   2017-08-23T09:43:26.59-0600 [CELL/0] OUT Exit status 0
   2017-08-23T09:43:26.63-0600 [API/0] OUT App instance exited with guid a1e48e70-b22d-4bc1-b1d1-d1451823782a payload: {"instance"=>"1114cf50-3c01-422a-76b8-0a12", "index"=>0, "reason"=>"CRASHED", "exit_description"=>"2 error(s) occurred:\n\n* Exited with status 1\n* cancelled", "crash_count"=>8, "crash_timestamp"=>1503503006591899799, "version"=>"46904aef-03bb-4cf1-8136-d856c77ccf0f"}
   2017-08-23T09:59:27.54-0600 [API/0] OUT Process has crashed with type: "web"
   2017-08-23T10:15:28.89-0600 [CELL/1] OUT Exit status 0
   2017-08-23T10:15:28.89-0600 [CELL/1] OUT Destroying container
   2017-08-23T10:15:29.42-0600 [CELL/1] OUT Successfully destroyed container
   2017-08-23T10:31:29.37-0600 [CELL/1] OUT Successfully created container
   2017-08-23T10:31:29.75-0600 [API/0] OUT App instance exited with guid a1e48e70-b22d-4bc1-b1d1-d1451823782a payload: {"instance"=>"f45ff5c0-94df-440f-63df-564d", "index"=>1, "reason"=>"CRASHED", "exit_description"=>"2 error(s) occurred:\n\n* Exited with status 1\n* cancelled", "crash_count"=>11, "crash_timestamp"=>1503505889712173999, "version"=>"46904aef-03bb-4cf1-8136-d856c77ccf0f"}
   2017-08-23T10:31:30.22-0600 [CELL/1] OUT Successfully destroyed container
   2017-08-23T10:47:29.88-0600 [CELL/1] OUT Creating container
   �[31m2017-08-23T10:47:32.35-0600 [APP/PROC/WEB/1] ERR oms-log-analytics-firehose-nozzle: error: time: missing unit in duration 30, try --help�[0m
   2017-08-23T10:47:32.35-0600 [APP/PROC/WEB/1] OUT Exit status 1
   2017-08-23T10:47:32.36-0600 [CELL/1] OUT Exit status 0
   2017-08-23T10:47:32.38-0600 [API/0] OUT Process has crashed with type: "web"
   2017-08-23T10:47:32.85-0600 [CELL/1] OUT Successfully destroyed container
   2017-08-23T11:03:31.15-0600 [CELL/0] OUT Creating container
   2017-08-23T11:19:32.75-0600 [API/0] OUT Process has crashed with type: "web"
   2017-08-23T11:35:33.45-0600 [CELL/0] OUT Successfully created container
   2017-08-23T11:35:33.73-0600 [APP/PROC/WEB/0] OUT Exit status 1
   2017-08-23T11:35:33.80-0600 [API/0] OUT App instance exited with guid a1e48e70-b22d-4bc1-b1d1-d1451823782a payload: {"instance"=>"39765669-d44e-4ac4-65ce-96dc", "index"=>0, "reason"=>"CRASHED", "exit_description"=>"2 error(s) occurred:\n\n* Exited with status 1\n* cancelled", "crash_count"=>15, "crash_timestamp"=>1503509733759388299, "version"=>"46904aef-03bb-4cf1-8136-d856c77ccf0f"}
   2017-08-23T11:36:20.49-0600 [API/0] OUT Updated app with guid a1e48e70-b22d-4bc1-b1d1-d1451823782a ({"name"=>"azure-log-analytics-nozzle-1.1.0", "instances"=>2, "memory"=>512, "buildpack"=>"https://github.com/cloudfoundry/go-buildpack.git", "environment_json"=>"PRIVATE DATA HIDDEN"})
   2017-08-23T11:36:28.22-0600 [API/0] OUT Updated app with guid a1e48e70-b22d-4bc1-b1d1-d1451823782a ({"environment_json"=>"PRIVATE DATA HIDDEN"})
   2017-08-23T11:36:31.63-0600 [API/0] OUT Updated app with guid a1e48e70-b22d-4bc1-b1d1-d1451823782a ({"environment_json"=>"PRIVATE DATA HIDDEN"})
   2017-08-23T11:36:31.91-0600 [API/0] OUT Updated app with guid a1e48e70-b22d-4bc1-b1d1-d1451823782a ({"environment_json"=>"PRIVATE DATA HIDDEN"})
   2017-08-23T11:36:32.69-0600 [API/0] OUT Updated app with guid a1e48e70-b22d-4bc1-b1d1-d1451823782a ({"environment_json"=>"PRIVATE DATA HIDDEN"})
   2017-08-23T11:36:33.22-0600 [API/0] OUT Updated app with guid a1e48e70-b22d-4bc1-b1d1-d1451823782a ({"environment_json"=>"PRIVATE DATA HIDDEN"})
   2017-08-23T11:36:33.99-0600 [STG/0] OUT Creating container
   2017-08-23T11:36:34.45-0600 [STG/0] OUT Downloading build artifacts cache...
   2017-08-23T11:36:45.81-0600 [STG/0] OUT -----> Go Buildpack version 1.8.6
   2017-08-23T11:36:45.81-0600 [STG/0] OUT -----> Installing godep v79
   2017-08-23T11:36:46.09-0600 [STG/0] OUT        �[31;1m**WARNING**�[0m A newer version of godep is available in this buildpack. Please adjust your app to use version 79.0.0 instead of version v79 as soon as possible. Old versions of godep are only provided to assist in migrating to newer versions.
   2017-08-23T11:36:46.73-0600 [STG/0] OUT -----> Installing go 1.7.6
   2017-08-23T11:36:53.59-0600 [STG/0] OUT -----> Running go build finalize
   2017-08-23T11:37:13.20-0600 [STG/0] OUT Exit status 0
   2017-08-23T11:37:13.25-0600 [STG/0] OUT Uploaded build artifacts cache (198B)
   2017-08-23T11:37:16.07-0600 [CELL/1] OUT Successfully created container
   2017-08-23T11:37:16.55-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503509836.553406239","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"OMS_BATCH_TIME":"10s"}}
   �[31m2017-08-23T11:37:16.62-0600 [APP/PROC/WEB/1] ERR goroutine 1 [running]:�[0m
   �[31m2017-08-23T11:37:16.62-0600 [APP/PROC/WEB/1] ERR github.com/Azure/oms-log-analytics-firehose-nozzle/vendor/code.cloudfoundry.org/lager.(*logger).Fatal(0xc4200186c0, 0x561eb5f7555d, 0x17, 0x561eb6463760, 0xc42011a150, 0x0, 0x0, 0x0)�[0m
   �[31m2017-08-23T11:37:16.62-0600 [APP/PROC/WEB/1] ERR 	/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/vendor/code.cloudfoundry.org/lager/logger.go:152 +0x41e�[0m
   �[31m2017-08-23T11:37:16.62-0600 [APP/PROC/WEB/1] ERR github.com/Azure/oms-log-analytics-firehose-nozzle/caching.(*Caching).Initialize(0xc420014bc0)�[0m
   �[31m2017-08-23T11:37:16.62-0600 [APP/PROC/WEB/1] ERR 	/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/caching/caching.go:39 +0x5ae�[0m
   2017-08-23T11:37:17.61-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503509837.610341072","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"IDLE_TIMEOUT":"30s"}}
   �[31m2017-08-23T11:37:17.68-0600 [APP/PROC/WEB/0] ERR panic: Could not get api /v2/info: json: cannot unmarshal number into Go value of type cfclient.Endpoint�[0m
   �[31m2017-08-23T11:37:17.68-0600 [APP/PROC/WEB/0] ERR 	/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/caching/caching.go:39 +0x5ae�[0m
   �[31m2017-08-23T11:37:17.68-0600 [APP/PROC/WEB/0] ERR 	/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/main.go:162 +0x1f13�[0m
   2017-08-23T11:37:17.71-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503509837.711792946","source":"oms-nozzle","message":"oms-nozzle.error creating cfclient","log_level":3,"data":{"error":"Could not get api /v2/info: json: cannot unmarshal number into Go value of type cfclient.Endpoint","trace":"goroutine 1 [running]:\ngithub.com/Azure/oms-log-analytics-firehose-nozzle/vendor/code.cloudfoundry.org/lager.(*logger).Fatal(0xc420018720, 0x5610c591155d, 0x17, 0x5610c5dff760, 0xc42010ada0, 0x0, 0x0, 0x0)\n\t/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/vendor/code.cloudfoundry.org/lager/logger.go:131 +0xc9\ngithub.com/Azure/oms-log-analytics-firehose-nozzle/caching.(*Caching).Initialize(0xc420014bc0)\n\t/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/caching/caching.go:39 +0x5ae\ngithub.com/Azure/oms-log-analytics-firehose-nozzle/omsnozzle.(*OmsNozzle).Start(0xc42008e750, 0xc42000de40, 0xc4200dce00)\n\t/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/omsnozzle/oms_nozzle.go:67 +0x3d\nmain.main()\n\t/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/main.go:162 +0x1f13\n"}}
   2017-08-23T11:37:17.71-0600 [CELL/0] OUT Exit status 0
   �[31m2017-08-23T11:37:17.71-0600 [APP/PROC/WEB/1] ERR panic: Could not get api /v2/info: json: cannot unmarshal number into Go value of type cfclient.Endpoint�[0m
   �[31m2017-08-23T11:37:17.71-0600 [APP/PROC/WEB/1] ERR goroutine 1 [running]:�[0m
   �[31m2017-08-23T11:37:17.71-0600 [APP/PROC/WEB/1] ERR panic(0x5610c5bdf0a0, 0xc42010ada0)�[0m
   �[31m2017-08-23T11:37:17.71-0600 [APP/PROC/WEB/1] ERR github.com/Azure/oms-log-analytics-firehose-nozzle/vendor/code.cloudfoundry.org/lager.(*logger).Fatal(0xc420018720, 0x5610c591155d, 0x17, 0x5610c5dff760, 0xc42010ada0, 0x0, 0x0, 0x0)�[0m
   �[31m2017-08-23T11:37:17.71-0600 [APP/PROC/WEB/1] ERR 	/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/vendor/code.cloudfoundry.org/lager/logger.go:152 +0x41e�[0m
   �[31m2017-08-23T11:37:17.71-0600 [APP/PROC/WEB/1] ERR github.com/Azure/oms-log-analytics-firehose-nozzle/caching.(*Caching).Initialize(0xc420014bc0)�[0m
   �[31m2017-08-23T11:37:17.71-0600 [APP/PROC/WEB/1] ERR 	/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/caching/caching.go:39 +0x5ae�[0m
   �[31m2017-08-23T11:37:17.71-0600 [APP/PROC/WEB/1] ERR 	/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/omsnozzle/oms_nozzle.go:67 +0x3d�[0m
   �[31m2017-08-23T11:37:17.71-0600 [APP/PROC/WEB/1] ERR main.main()�[0m
   �[31m2017-08-23T11:37:17.71-0600 [APP/PROC/WEB/1] ERR 	/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/main.go:162 +0x1f13�[0m
   2017-08-23T11:37:17.73-0600 [CELL/1] OUT Exit status 0
   2017-08-23T11:37:17.77-0600 [API/0] OUT Process has crashed with type: "web"
   2017-08-23T11:37:17.78-0600 [API/0] OUT App instance exited with guid a1e48e70-b22d-4bc1-b1d1-d1451823782a payload: {"instance"=>"3ba56c75-3583-4b1f-6e59-a5f3", "index"=>0, "reason"=>"CRASHED", "exit_description"=>"2 error(s) occurred:\n\n* Exited with status 2\n* cancelled", "crash_count"=>2, "crash_timestamp"=>1503509837721271199, "version"=>"6f568a4d-fce4-4879-a8e3-caeb22a0c267"}
   2017-08-23T11:37:17.80-0600 [CELL/0] OUT Creating container
   2017-08-23T11:37:17.80-0600 [CELL/1] OUT Creating container
   2017-08-23T11:37:18.17-0600 [CELL/0] OUT Successfully created container
   2017-08-23T11:37:18.18-0600 [CELL/1] OUT Successfully created container
   2017-08-23T11:37:18.45-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503509838.454396725","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"OMS_POST_TIMEOUT":"10s"}}
   2017-08-23T11:37:18.45-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503509838.454525709","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"SKIP_SSL_VALIDATION":true}}
   2017-08-23T11:37:18.45-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503509838.454556704","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"IDLE_TIMEOUT":"30s"}}
   2017-08-23T11:37:18.45-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503509838.454566240","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"OMS_BATCH_TIME":"10s"}}
   2017-08-23T11:37:18.45-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503509838.454634905","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"CF_ENVIRONMENT":"cf"}}
   2017-08-23T11:37:18.45-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503509838.454652786","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"OMS_MAX_MSG_NUM_PER_BATCH":1000}}
   2017-08-23T11:37:18.45-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503509838.454662800","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"LOG_EVENT_COUNT":true}}
   2017-08-23T11:37:18.45-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503509838.454745531","source":"oms-nozzle","message":"oms-nozzle.config EVENT_FILTER is nil. all events will be published","log_level":1,"data":{}}
   2017-08-23T11:37:18.48-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503509838.480115414","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"OMS_POST_TIMEOUT":"10s"}}
   2017-08-23T11:37:18.48-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503509838.480288029","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"IDLE_TIMEOUT":"30s"}}
   2017-08-23T11:37:18.48-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503509838.480304956","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"OMS_BATCH_TIME":"10s"}}
   2017-08-23T11:37:18.48-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503509838.480319500","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"CF_ENVIRONMENT":"cf"}}
   2017-08-23T11:37:18.48-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503509838.480335474","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"OMS_MAX_MSG_NUM_PER_BATCH":1000}}
   2017-08-23T11:37:18.48-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503509838.480373859","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"LOG_EVENT_COUNT":true}}
   2017-08-23T11:37:18.48-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503509838.480407238","source":"oms-nozzle","message":"oms-nozzle.config EVENT_FILTER is nil. all events will be published","log_level":1,"data":{}}
   2017-08-23T11:37:18.48-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503509838.480471373","source":"oms-nozzle","message":"oms-nozzle.getting nozzle instance name","log_level":1,"data":{"name":"pid-19@88370911-c601-4074-6439-4ce7"}}
   2017-08-23T11:37:18.49-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503509838.492529631","source":"oms-nozzle","message":"oms-nozzle.error creating cfclient","log_level":3,"data":{"error":"Could not get api /v2/info: json: cannot unmarshal number into Go value of type cfclient.Endpoint","trace":"goroutine 1 [running]:\ngithub.com/Azure/oms-log-analytics-firehose-nozzle/vendor/code.cloudfoundry.org/lager.(*logger).Fatal(0xc420018660, 0x55e65194e55d, 0x17, 0x55e651e3c760, 0xc4201221b0, 0x0, 0x0, 0x0)\n\t/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/vendor/code.cloudfoundry.org/lager/logger.go:131 +0xc9\ngithub.com/Azure/oms-log-analytics-firehose-nozzle/caching.(*Caching).Initialize(0xc420014bc0)\n\t/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/caching/caching.go:39 +0x5ae\ngithub.com/Azure/oms-log-analytics-firehose-nozzle/omsnozzle.(*OmsNozzle).Start(0xc420080750, 0xc42000de30, 0xc4200cee00)\n\t/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/omsnozzle/oms_nozzle.go:67 +0x3d\nmain.main()\n\t/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/main.go:162 +0x1f13\n"}}
   �[31m2017-08-23T11:37:18.49-0600 [APP/PROC/WEB/0] ERR panic: Could not get api /v2/info: json: cannot unmarshal number into Go value of type cfclient.Endpoint�[0m
   �[31m2017-08-23T11:37:18.49-0600 [APP/PROC/WEB/0] ERR 	/tmp/app/.cloudfoundry/0/go1.7.6/go/src/runtime/panic.go:500 +0x1a5�[0m
   �[31m2017-08-23T11:37:18.49-0600 [APP/PROC/WEB/0] ERR github.com/Azure/oms-log-analytics-firehose-nozzle/vendor/code.cloudfoundry.org/lager.(*logger).Fatal(0xc420018660, 0x55e65194e55d, 0x17, 0x55e651e3c760, 0xc4201221b0, 0x0, 0x0, 0x0)�[0m
   �[31m2017-08-23T11:37:18.49-0600 [APP/PROC/WEB/0] ERR 	/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/caching/caching.go:39 +0x5ae�[0m
   �[31m2017-08-23T11:37:18.49-0600 [APP/PROC/WEB/0] ERR 	/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/omsnozzle/oms_nozzle.go:67 +0x3d�[0m
   2017-08-23T11:37:18.49-0600 [APP/PROC/WEB/0] OUT Exit status 2
   2017-08-23T11:37:18.51-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503509838.514317036","source":"oms-nozzle","message":"oms-nozzle.error creating cfclient","log_level":3,"data":{"error":"Could not get api /v2/info: json: cannot unmarshal number into Go value of type cfclient.Endpoint","trace":"goroutine 1 [running]:\ngithub.com/Azure/oms-log-analytics-firehose-nozzle/vendor/code.cloudfoundry.org/lager.(*logger).Fatal(0xc420018660, 0x55e9f6bd955d, 0x17, 0x55e9f70c7760, 0xc42011e050, 0x0, 0x0, 0x0)\n\t/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/vendor/code.cloudfoundry.org/lager/logger.go:131 +0xc9\ngithub.com/Azure/oms-log-analytics-firehose-nozzle/caching.(*Caching).Initialize(0xc420014bc0)\n\t/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/caching/caching.go:39 +0x5ae\ngithub.com/Azure/oms-log-analytics-firehose-nozzle/omsnozzle.(*OmsNozzle).Start(0xc420080750, 0xc42000de30, 0xc4200cee00)\n\t/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/omsnozzle/oms_nozzle.go:67 +0x3d\nmain.main()\n\t/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/main.go:162 +0x1f13\n"}}
   2017-08-23T11:37:18.51-0600 [CELL/0] OUT Exit status 0
   �[31m2017-08-23T11:37:18.51-0600 [APP/PROC/WEB/1] ERR 	/tmp/app/.cloudfoundry/0/go1.7.6/go/src/runtime/panic.go:500 +0x1a5�[0m
   �[31m2017-08-23T11:37:18.51-0600 [APP/PROC/WEB/1] ERR 	/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/omsnozzle/oms_nozzle.go:67 +0x3d�[0m
   �[31m2017-08-23T11:37:18.51-0600 [APP/PROC/WEB/1] ERR 	/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/main.go:162 +0x1f13�[0m
   2017-08-23T11:37:18.51-0600 [APP/PROC/WEB/1] OUT Exit status 2
   2017-08-23T11:37:18.52-0600 [CELL/0] OUT Destroying container
   2017-08-23T11:37:18.53-0600 [CELL/1] OUT Exit status 0
   2017-08-23T11:37:18.53-0600 [API/1] OUT Process has crashed with type: "web"
   2017-08-23T11:37:18.54-0600 [API/1] OUT App instance exited with guid a1e48e70-b22d-4bc1-b1d1-d1451823782a payload: {"instance"=>"ba910590-0b7b-4262-5507-35f2", "index"=>0, "reason"=>"CRASHED", "exit_description"=>"2 error(s) occurred:\n\n* Exited with status 2\n* cancelled", "crash_count"=>3, "crash_timestamp"=>1503509838504482899, "version"=>"6f568a4d-fce4-4879-a8e3-caeb22a0c267"}
   2017-08-23T11:37:18.56-0600 [API/0] OUT Process has crashed with type: "web"
   2017-08-23T11:37:19.03-0600 [CELL/1] OUT Successfully destroyed container
   2017-08-23T11:38:03.26-0600 [CELL/0] OUT Creating container
   2017-08-23T11:38:03.65-0600 [CELL/1] OUT Successfully created container
   2017-08-23T11:38:03.92-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503509883.925692081","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"SKIP_SSL_VALIDATION":true}}
   2017-08-23T11:38:03.92-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503509883.925754786","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"IDLE_TIMEOUT":"30s"}}
   2017-08-23T11:38:03.92-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503509883.925806046","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"OMS_BATCH_TIME":"10s"}}
   2017-08-23T11:38:03.92-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503509883.926029205","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"CF_ENVIRONMENT":"cf"}}
   2017-08-23T11:38:03.92-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503509883.926046133","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"OMS_MAX_MSG_NUM_PER_BATCH":1000}}
   2017-08-23T11:38:03.92-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503509883.926115990","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"LOG_EVENT_COUNT":true}}
   2017-08-23T11:38:03.92-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503509883.926169157","source":"oms-nozzle","message":"oms-nozzle.config EVENT_FILTER is nil. all events will be published","log_level":1,"data":{}}
   2017-08-23T11:38:03.92-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503509883.926533461","source":"oms-nozzle","message":"oms-nozzle.getting nozzle instance name","log_level":1,"data":{"name":"pid-14@b3d1937c-daca-4481-49f9-3e62"}}
   2017-08-23T11:38:03.96-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503509883.963166237","source":"oms-nozzle","message":"oms-nozzle.error creating cfclient","log_level":3,"data":{"error":"Could not get api /v2/info: json: cannot unmarshal number into Go value of type cfclient.Endpoint","trace":"goroutine 1 [running]:\ngithub.com/Azure/oms-log-analytics-firehose-nozzle/vendor/code.cloudfoundry.org/lager.(*logger).Fatal(0xc420018660, 0x5561886ac55d, 0x17, 0x556188b9a760, 0xc4200fa170, 0x0, 0x0, 0x0)\n\t/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/vendor/code.cloudfoundry.org/lager/logger.go:131 +0xc9\ngithub.com/Azure/oms-log-analytics-firehose-nozzle/caching.(*Caching).Initialize(0xc420014bc0)\n\t/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/caching/caching.go:39 +0x5ae\ngithub.com/Azure/oms-log-analytics-firehose-nozzle/omsnozzle.(*OmsNozzle).Start(0xc42008e750, 0xc42000de30, 0xc4200dce00)\n\t/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/omsnozzle/oms_nozzle.go:67 +0x3d\nmain.main()\n\t/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/main.go:162 +0x1f13\n"}}
   �[31m2017-08-23T11:38:03.96-0600 [APP/PROC/WEB/1] ERR panic: Could not get api /v2/info: json: cannot unmarshal number into Go value of type cfclient.Endpoint�[0m
   �[31m2017-08-23T11:38:03.96-0600 [APP/PROC/WEB/1] ERR goroutine 1 [running]:�[0m
   �[31m2017-08-23T11:38:03.96-0600 [APP/PROC/WEB/1] ERR panic(0x55618897a0a0, 0xc4200fa170)�[0m
   �[31m2017-08-23T11:38:03.96-0600 [APP/PROC/WEB/1] ERR 	/tmp/app/.cloudfoundry/0/go1.7.6/go/src/runtime/panic.go:500 +0x1a5�[0m
   �[31m2017-08-23T11:38:03.96-0600 [APP/PROC/WEB/1] ERR github.com/Azure/oms-log-analytics-firehose-nozzle/vendor/code.cloudfoundry.org/lager.(*logger).Fatal(0xc420018660, 0x5561886ac55d, 0x17, 0x556188b9a760, 0xc4200fa170, 0x0, 0x0, 0x0)�[0m
   �[31m2017-08-23T11:38:03.96-0600 [APP/PROC/WEB/1] ERR 	/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/vendor/code.cloudfoundry.org/lager/logger.go:152 +0x41e�[0m
   �[31m2017-08-23T11:38:03.96-0600 [APP/PROC/WEB/1] ERR github.com/Azure/oms-log-analytics-firehose-nozzle/caching.(*Caching).Initialize(0xc420014bc0)�[0m
   �[31m2017-08-23T11:38:03.96-0600 [APP/PROC/WEB/1] ERR 	/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/caching/caching.go:39 +0x5ae�[0m
   �[31m2017-08-23T11:38:03.96-0600 [APP/PROC/WEB/1] ERR github.com/Azure/oms-log-analytics-firehose-nozzle/omsnozzle.(*OmsNozzle).Start(0xc42008e750, 0xc42000de30, 0xc4200dce00)�[0m
   �[31m2017-08-23T11:38:03.96-0600 [APP/PROC/WEB/1] ERR 	/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/omsnozzle/oms_nozzle.go:67 +0x3d�[0m
   �[31m2017-08-23T11:38:03.96-0600 [APP/PROC/WEB/1] ERR main.main()�[0m
   �[31m2017-08-23T11:38:03.96-0600 [APP/PROC/WEB/1] ERR 	/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/main.go:162 +0x1f13�[0m
   2017-08-23T11:38:03.96-0600 [APP/PROC/WEB/1] OUT Exit status 2
   2017-08-23T11:38:03.96-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503509883.968169212","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"OMS_POST_TIMEOUT":"10s"}}
   2017-08-23T11:38:03.96-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503509883.968282461","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"SKIP_SSL_VALIDATION":true}}
   2017-08-23T11:38:03.96-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503509883.968301535","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"IDLE_TIMEOUT":"30s"}}
   2017-08-23T11:38:03.96-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503509883.968318701","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"OMS_BATCH_TIME":"10s"}}
   2017-08-23T11:38:03.96-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503509883.968343258","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"CF_ENVIRONMENT":"cf"}}
   2017-08-23T11:38:03.96-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503509883.968365669","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"OMS_MAX_MSG_NUM_PER_BATCH":1000}}
   2017-08-23T11:38:03.96-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503509883.968382359","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"LOG_EVENT_COUNT":true}}
   2017-08-23T11:38:03.96-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503509883.968398094","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"LOG_EVENT_COUNT_INTERVAL":"1m0s"}}
   2017-08-23T11:38:03.96-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503509883.968406200","source":"oms-nozzle","message":"oms-nozzle.config EVENT_FILTER is nil. all events will be published","log_level":1,"data":{}}
   2017-08-23T11:38:03.96-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503509883.968469620","source":"oms-nozzle","message":"oms-nozzle.getting nozzle instance name","log_level":1,"data":{"name":"pid-20@5b89ddaf-10df-4e0b-6849-9cfa"}}
   2017-08-23T11:38:03.98-0600 [CELL/1] OUT Exit status 0
   2017-08-23T11:38:03.99-0600 [CELL/1] OUT Destroying container
   2017-08-23T11:38:04.00-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503509884.003629684","source":"oms-nozzle","message":"oms-nozzle.error creating cfclient","log_level":3,"data":{"error":"Could not get api /v2/info: json: cannot unmarshal number into Go value of type cfclient.Endpoint","trace":"goroutine 1 [running]:\ngithub.com/Azure/oms-log-analytics-firehose-nozzle/vendor/code.cloudfoundry.org/lager.(*logger).Fatal(0xc420018660, 0x55b75bca155d, 0x17, 0x55b75c18f760, 0xc4201041e0, 0x0, 0x0, 0x0)\n\t/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/vendor/code.cloudfoundry.org/lager/logger.go:131 +0xc9\ngithub.com/Azure/oms-log-analytics-firehose-nozzle/caching.(*Caching).Initialize(0xc420014bc0)\n\t/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/caching/caching.go:39 +0x5ae\ngithub.com/Azure/oms-log-analytics-firehose-nozzle/omsnozzle.(*OmsNozzle).Start(0xc420080750, 0xc42000de30, 0xc4200d0e00)\n\t/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/omsnozzle/oms_nozzle.go:67 +0x3d\nmain.main()\n\t/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/main.go:162 +0x1f13\n"}}
   �[31m2017-08-23T11:38:04.00-0600 [APP/PROC/WEB/0] ERR panic: Could not get api /v2/info: json: cannot unmarshal number into Go value of type cfclient.Endpoint�[0m
   �[31m2017-08-23T11:38:04.00-0600 [APP/PROC/WEB/0] ERR goroutine 1 [running]:�[0m
   �[31m2017-08-23T11:38:04.00-0600 [APP/PROC/WEB/0] ERR panic(0x55b75bf6f0a0, 0xc4201041e0)�[0m
   �[31m2017-08-23T11:38:04.00-0600 [APP/PROC/WEB/0] ERR 	/tmp/app/.cloudfoundry/0/go1.7.6/go/src/runtime/panic.go:500 +0x1a5�[0m
   �[31m2017-08-23T11:38:04.00-0600 [APP/PROC/WEB/0] ERR github.com/Azure/oms-log-analytics-firehose-nozzle/vendor/code.cloudfoundry.org/lager.(*logger).Fatal(0xc420018660, 0x55b75bca155d, 0x17, 0x55b75c18f760, 0xc4201041e0, 0x0, 0x0, 0x0)�[0m
   �[31m2017-08-23T11:38:04.00-0600 [APP/PROC/WEB/0] ERR 	/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/vendor/code.cloudfoundry.org/lager/logger.go:152 +0x41e�[0m
   �[31m2017-08-23T11:38:04.00-0600 [APP/PROC/WEB/0] ERR github.com/Azure/oms-log-analytics-firehose-nozzle/caching.(*Caching).Initialize(0xc420014bc0)�[0m
   �[31m2017-08-23T11:38:04.00-0600 [APP/PROC/WEB/0] ERR 	/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/caching/caching.go:39 +0x5ae�[0m
   �[31m2017-08-23T11:38:04.00-0600 [APP/PROC/WEB/0] ERR github.com/Azure/oms-log-analytics-firehose-nozzle/omsnozzle.(*OmsNozzle).Start(0xc420080750, 0xc42000de30, 0xc4200d0e00)�[0m
   �[31m2017-08-23T11:38:04.00-0600 [APP/PROC/WEB/0] ERR 	/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/omsnozzle/oms_nozzle.go:67 +0x3d�[0m
   �[31m2017-08-23T11:38:04.00-0600 [APP/PROC/WEB/0] ERR main.main()�[0m
   �[31m2017-08-23T11:38:04.00-0600 [APP/PROC/WEB/0] ERR 	/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/main.go:162 +0x1f13�[0m
   2017-08-23T11:38:04.00-0600 [APP/PROC/WEB/0] OUT Exit status 2
   2017-08-23T11:38:04.02-0600 [CELL/0] OUT Exit status 0
   2017-08-23T11:38:04.02-0600 [API/0] OUT Process has crashed with type: "web"
   2017-08-23T11:38:04.03-0600 [CELL/0] OUT Destroying container
   2017-08-23T11:38:04.03-0600 [API/0] OUT App instance exited with guid a1e48e70-b22d-4bc1-b1d1-d1451823782a payload: {"instance"=>"b3d1937c-daca-4481-49f9-3e62", "index"=>1, "reason"=>"CRASHED", "exit_description"=>"2 error(s) occurred:\n\n* Exited with status 2\n* cancelled", "crash_count"=>4, "crash_timestamp"=>1503509883998985699, "version"=>"6f568a4d-fce4-4879-a8e3-caeb22a0c267"}
   2017-08-23T11:38:04.06-0600 [API/0] OUT Process has crashed with type: "web"
   2017-08-23T11:38:04.07-0600 [API/0] OUT App instance exited with guid a1e48e70-b22d-4bc1-b1d1-d1451823782a payload: {"instance"=>"5b89ddaf-10df-4e0b-6849-9cfa", "index"=>0, "reason"=>"CRASHED", "exit_description"=>"2 error(s) occurred:\n\n* Exited with status 2\n* cancelled", "crash_count"=>4, "crash_timestamp"=>1503509884039440999, "version"=>"6f568a4d-fce4-4879-a8e3-caeb22a0c267"}
   2017-08-23T11:38:04.47-0600 [CELL/1] OUT Successfully destroyed container
   2017-08-23T11:38:04.51-0600 [CELL/0] OUT Successfully destroyed container
   2017-08-23T11:39:33.36-0600 [CELL/0] OUT Creating container
   2017-08-23T11:39:33.37-0600 [CELL/1] OUT Creating container
   2017-08-23T11:39:33.75-0600 [CELL/1] OUT Successfully created container
   2017-08-23T11:39:33.75-0600 [CELL/0] OUT Successfully created container
   2017-08-23T11:39:34.03-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503509974.038251162","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"OMS_POST_TIMEOUT":"10s"}}
   2017-08-23T11:39:34.03-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503509974.038713694","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"SKIP_SSL_VALIDATION":true}}
   2017-08-23T11:39:34.03-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503509974.038729906","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"IDLE_TIMEOUT":"30s"}}
   2017-08-23T11:39:34.03-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503509974.038788557","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"OMS_BATCH_TIME":"10s"}}
   2017-08-23T11:39:34.03-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503509974.039063692","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"CF_ENVIRONMENT":"cf"}}
   2017-08-23T11:39:34.03-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503509974.039081335","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"OMS_MAX_MSG_NUM_PER_BATCH":1000}}
   2017-08-23T11:39:34.03-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503509974.039090157","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"LOG_EVENT_COUNT":true}}
   2017-08-23T11:39:34.03-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503509974.039097309","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"LOG_EVENT_COUNT_INTERVAL":"1m0s"}}
   2017-08-23T11:39:34.03-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503509974.039103985","source":"oms-nozzle","message":"oms-nozzle.config EVENT_FILTER is nil. all events will be published","log_level":1,"data":{}}
   2017-08-23T11:39:34.03-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503509974.039147615","source":"oms-nozzle","message":"oms-nozzle.getting nozzle instance name","log_level":1,"data":{"name":"pid-14@d918cbcb-ce5b-4223-512b-eb07"}}
   2017-08-23T11:39:34.04-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503509974.047873259","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"OMS_POST_TIMEOUT":"10s"}}
   2017-08-23T11:39:34.04-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503509974.047947884","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"SKIP_SSL_VALIDATION":true}}
   2017-08-23T11:39:34.04-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503509974.047963858","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"IDLE_TIMEOUT":"30s"}}
   2017-08-23T11:39:34.04-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503509974.047972679","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"OMS_BATCH_TIME":"10s"}}
   2017-08-23T11:39:34.04-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503509974.048011541","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"CF_ENVIRONMENT":"cf"}}
   2017-08-23T11:39:34.04-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503509974.048020124","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"OMS_MAX_MSG_NUM_PER_BATCH":1000}}
   2017-08-23T11:39:34.04-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503509974.048029184","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"LOG_EVENT_COUNT":true}}
   2017-08-23T11:39:34.04-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503509974.048036814","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"LOG_EVENT_COUNT_INTERVAL":"1m0s"}}
   2017-08-23T11:39:34.04-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503509974.048044205","source":"oms-nozzle","message":"oms-nozzle.config EVENT_FILTER is nil. all events will be published","log_level":1,"data":{}}
   2017-08-23T11:39:34.04-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503509974.048091412","source":"oms-nozzle","message":"oms-nozzle.getting nozzle instance name","log_level":1,"data":{"name":"pid-19@04d45746-0029-4546-526e-59bd"}}
   2017-08-23T11:39:34.08-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503509974.082948685","source":"oms-nozzle","message":"oms-nozzle.error creating cfclient","log_level":3,"data":{"error":"Could not get api /v2/info: json: cannot unmarshal number into Go value of type cfclient.Endpoint","trace":"goroutine 1 [running]:\ngithub.com/Azure/oms-log-analytics-firehose-nozzle/vendor/code.cloudfoundry.org/lager.(*logger).Fatal(0xc4200186c0, 0x563812bcd55d, 0x17, 0x5638130bb760, 0xc4201f0050, 0x0, 0x0, 0x0)\n\t/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/vendor/code.cloudfoundry.org/lager/logger.go:131 +0xc9\ngithub.com/Azure/oms-log-analytics-firehose-nozzle/caching.(*Caching).Initialize(0xc420014bc0)\n\t/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/caching/caching.go:39 +0x5ae\ngithub.com/Azure/oms-log-analytics-firehose-nozzle/omsnozzle.(*OmsNozzle).Start(0xc42008e750, 0xc42000de40, 0xc4200dce00)\n\t/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/omsnozzle/oms_nozzle.go:67 +0x3d\nmain.main()\n\t/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/main.go:162 +0x1f13\n"}}
   �[31m2017-08-23T11:39:34.08-0600 [APP/PROC/WEB/1] ERR panic: Could not get api /v2/info: json: cannot unmarshal number into Go value of type cfclient.Endpoint�[0m
   �[31m2017-08-23T11:39:34.08-0600 [APP/PROC/WEB/1] ERR goroutine 1 [running]:�[0m
   �[31m2017-08-23T11:39:34.08-0600 [APP/PROC/WEB/1] ERR panic(0x563812e9b0a0, 0xc4201f0050)�[0m
   �[31m2017-08-23T11:39:34.08-0600 [APP/PROC/WEB/1] ERR 	/tmp/app/.cloudfoundry/0/go1.7.6/go/src/runtime/panic.go:500 +0x1a5�[0m
   �[31m2017-08-23T11:39:34.08-0600 [APP/PROC/WEB/1] ERR github.com/Azure/oms-log-analytics-firehose-nozzle/vendor/code.cloudfoundry.org/lager.(*logger).Fatal(0xc4200186c0, 0x563812bcd55d, 0x17, 0x5638130bb760, 0xc4201f0050, 0x0, 0x0, 0x0)�[0m
   �[31m2017-08-23T11:39:34.08-0600 [APP/PROC/WEB/1] ERR 	/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/vendor/code.cloudfoundry.org/lager/logger.go:152 +0x41e�[0m
   �[31m2017-08-23T11:39:34.08-0600 [APP/PROC/WEB/1] ERR github.com/Azure/oms-log-analytics-firehose-nozzle/caching.(*Caching).Initialize(0xc420014bc0)�[0m
   �[31m2017-08-23T11:39:34.08-0600 [APP/PROC/WEB/1] ERR 	/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/caching/caching.go:39 +0x5ae�[0m
   �[31m2017-08-23T11:39:34.08-0600 [APP/PROC/WEB/1] ERR github.com/Azure/oms-log-analytics-firehose-nozzle/omsnozzle.(*OmsNozzle).Start(0xc42008e750, 0xc42000de40, 0xc4200dce00)�[0m
   �[31m2017-08-23T11:39:34.08-0600 [APP/PROC/WEB/1] ERR 	/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/omsnozzle/oms_nozzle.go:67 +0x3d�[0m
   �[31m2017-08-23T11:39:34.08-0600 [APP/PROC/WEB/1] ERR main.main()�[0m
   �[31m2017-08-23T11:39:34.08-0600 [APP/PROC/WEB/1] ERR 	/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/main.go:162 +0x1f13�[0m
   2017-08-23T11:39:34.08-0600 [APP/PROC/WEB/1] OUT Exit status 2
   2017-08-23T11:39:34.09-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503509974.095694542","source":"oms-nozzle","message":"oms-nozzle.error creating cfclient","log_level":3,"data":{"error":"Could not get api /v2/info: json: cannot unmarshal number into Go value of type cfclient.Endpoint","trace":"goroutine 1 [running]:\ngithub.com/Azure/oms-log-analytics-firehose-nozzle/vendor/code.cloudfoundry.org/lager.(*logger).Fatal(0xc4200186c0, 0x558a1869255d, 0x17, 0x558a18b80760, 0xc4201c4050, 0x0, 0x0, 0x0)\n\t/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/vendor/code.cloudfoundry.org/lager/logger.go:131 +0xc9\ngithub.com/Azure/oms-log-analytics-firehose-nozzle/caching.(*Caching).Initialize(0xc420014bc0)\n\t/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/caching/caching.go:39 +0x5ae\ngithub.com/Azure/oms-log-analytics-firehose-nozzle/omsnozzle.(*OmsNozzle).Start(0xc42008e750, 0xc42000de40, 0xc4200dce00)\n\t/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/omsnozzle/oms_nozzle.go:67 +0x3d\nmain.main()\n\t/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/main.go:162 +0x1f13\n"}}
   �[31m2017-08-23T11:39:34.09-0600 [APP/PROC/WEB/0] ERR panic: Could not get api /v2/info: json: cannot unmarshal number into Go value of type cfclient.Endpoint�[0m
   �[31m2017-08-23T11:39:34.09-0600 [APP/PROC/WEB/0] ERR goroutine 1 [running]:�[0m
   �[31m2017-08-23T11:39:34.09-0600 [APP/PROC/WEB/0] ERR panic(0x558a189600a0, 0xc4201c4050)�[0m
   �[31m2017-08-23T11:39:34.09-0600 [APP/PROC/WEB/0] ERR 	/tmp/app/.cloudfoundry/0/go1.7.6/go/src/runtime/panic.go:500 +0x1a5�[0m
   �[31m2017-08-23T11:39:34.09-0600 [APP/PROC/WEB/0] ERR github.com/Azure/oms-log-analytics-firehose-nozzle/vendor/code.cloudfoundry.org/lager.(*logger).Fatal(0xc4200186c0, 0x558a1869255d, 0x17, 0x558a18b80760, 0xc4201c4050, 0x0, 0x0, 0x0)�[0m
   �[31m2017-08-23T11:39:34.09-0600 [APP/PROC/WEB/0] ERR 	/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/vendor/code.cloudfoundry.org/lager/logger.go:152 +0x41e�[0m
   �[31m2017-08-23T11:39:34.09-0600 [APP/PROC/WEB/0] ERR github.com/Azure/oms-log-analytics-firehose-nozzle/caching.(*Caching).Initialize(0xc420014bc0)�[0m
   �[31m2017-08-23T11:39:34.09-0600 [APP/PROC/WEB/0] ERR 	/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/caching/caching.go:39 +0x5ae�[0m
   �[31m2017-08-23T11:39:34.09-0600 [APP/PROC/WEB/0] ERR github.com/Azure/oms-log-analytics-firehose-nozzle/omsnozzle.(*OmsNozzle).Start(0xc42008e750, 0xc42000de40, 0xc4200dce00)�[0m
   �[31m2017-08-23T11:39:34.09-0600 [APP/PROC/WEB/0] ERR 	/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/omsnozzle/oms_nozzle.go:67 +0x3d�[0m
   �[31m2017-08-23T11:39:34.09-0600 [APP/PROC/WEB/0] ERR main.main()�[0m
   �[31m2017-08-23T11:39:34.09-0600 [APP/PROC/WEB/0] ERR 	/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/main.go:162 +0x1f13�[0m
   2017-08-23T11:39:34.10-0600 [APP/PROC/WEB/0] OUT Exit status 2
   2017-08-23T11:39:34.10-0600 [CELL/1] OUT Exit status 0
   2017-08-23T11:39:34.11-0600 [CELL/0] OUT Exit status 0
   2017-08-23T11:39:34.11-0600 [CELL/1] OUT Destroying container
   2017-08-23T11:39:34.12-0600 [CELL/0] OUT Destroying container
   2017-08-23T11:39:34.14-0600 [API/0] OUT Process has crashed with type: "web"
   2017-08-23T11:39:34.15-0600 [API/0] OUT App instance exited with guid a1e48e70-b22d-4bc1-b1d1-d1451823782a payload: {"instance"=>"04d45746-0029-4546-526e-59bd", "index"=>1, "reason"=>"CRASHED", "exit_description"=>"2 error(s) occurred:\n\n* Exited with status 2\n* cancelled", "crash_count"=>5, "crash_timestamp"=>1503509974120302399, "version"=>"6f568a4d-fce4-4879-a8e3-caeb22a0c267"}
   2017-08-23T11:39:34.17-0600 [API/1] OUT Process has crashed with type: "web"
   2017-08-23T11:39:34.18-0600 [API/1] OUT App instance exited with guid a1e48e70-b22d-4bc1-b1d1-d1451823782a payload: {"instance"=>"d918cbcb-ce5b-4223-512b-eb07", "index"=>0, "reason"=>"CRASHED", "exit_description"=>"2 error(s) occurred:\n\n* Exited with status 2\n* cancelled", "crash_count"=>5, "crash_timestamp"=>1503509974134852099, "version"=>"6f568a4d-fce4-4879-a8e3-caeb22a0c267"}
   2017-08-23T11:39:34.57-0600 [CELL/1] OUT Successfully destroyed container
   2017-08-23T11:39:34.62-0600 [CELL/0] OUT Successfully destroyed container
   2017-08-23T11:42:03.54-0600 [CELL/1] OUT Creating container
   2017-08-23T11:42:03.54-0600 [CELL/0] OUT Creating container
   2017-08-23T11:42:03.91-0600 [CELL/1] OUT Successfully created container
   2017-08-23T11:42:03.93-0600 [CELL/0] OUT Successfully created container
   2017-08-23T11:42:04.21-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503510124.211110353","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"OMS_POST_TIMEOUT":"10s"}}
   2017-08-23T11:42:04.21-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503510124.211193800","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"SKIP_SSL_VALIDATION":true}}
   2017-08-23T11:42:04.21-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503510124.211219311","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"IDLE_TIMEOUT":"30s"}}
   2017-08-23T11:42:04.21-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503510124.211343050","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"OMS_BATCH_TIME":"10s"}}
   2017-08-23T11:42:04.21-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503510124.211359501","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"CF_ENVIRONMENT":"cf"}}
   2017-08-23T11:42:04.21-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503510124.211366892","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"OMS_MAX_MSG_NUM_PER_BATCH":1000}}
   2017-08-23T11:42:04.21-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503510124.211381435","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"LOG_EVENT_COUNT":true}}
   2017-08-23T11:42:04.21-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503510124.211394787","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"LOG_EVENT_COUNT_INTERVAL":"1m0s"}}
   2017-08-23T11:42:04.21-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503510124.211401463","source":"oms-nozzle","message":"oms-nozzle.config EVENT_FILTER is nil. all events will be published","log_level":1,"data":{}}
   2017-08-23T11:42:04.21-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503510124.211454630","source":"oms-nozzle","message":"oms-nozzle.getting nozzle instance name","log_level":1,"data":{"name":"pid-19@c6ccb8f3-21fb-4baa-7780-f990"}}
   2017-08-23T11:42:04.24-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503510124.243658066","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"OMS_POST_TIMEOUT":"10s"}}
   2017-08-23T11:42:04.24-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503510124.243902445","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"SKIP_SSL_VALIDATION":true}}
   2017-08-23T11:42:04.24-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503510124.243919849","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"IDLE_TIMEOUT":"30s"}}
   2017-08-23T11:42:04.24-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503510124.243936300","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"OMS_BATCH_TIME":"10s"}}
   2017-08-23T11:42:04.24-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503510124.243957520","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"CF_ENVIRONMENT":"cf"}}
   2017-08-23T11:42:04.24-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503510124.243982077","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"OMS_MAX_MSG_NUM_PER_BATCH":1000}}
   2017-08-23T11:42:04.24-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503510124.244013309","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"LOG_EVENT_COUNT":true}}
   2017-08-23T11:42:04.24-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503510124.244047165","source":"oms-nozzle","message":"oms-nozzle.config","log_level":1,"data":{"LOG_EVENT_COUNT_INTERVAL":"1m0s"}}
   2017-08-23T11:42:04.24-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503510124.244056225","source":"oms-nozzle","message":"oms-nozzle.config EVENT_FILTER is nil. all events will be published","log_level":1,"data":{}}
   2017-08-23T11:42:04.24-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503510124.244127035","source":"oms-nozzle","message":"oms-nozzle.getting nozzle instance name","log_level":1,"data":{"name":"pid-19@c1f6cb17-9ea9-4cc1-49b5-d501"}}
   2017-08-23T11:42:04.24-0600 [APP/PROC/WEB/1] OUT {"timestamp":"1503510124.248155594","source":"oms-nozzle","message":"oms-nozzle.error creating cfclient","log_level":3,"data":{"error":"Could not get api /v2/info: json: cannot unmarshal number into Go value of type cfclient.Endpoint","trace":"goroutine 1 [running]:\ngithub.com/Azure/oms-log-analytics-firehose-nozzle/vendor/code.cloudfoundry.org/lager.(*logger).Fatal(0xc420018660, 0x55ed1f1b355d, 0x17, 0x55ed1f6a1760, 0xc420156b00, 0x0, 0x0, 0x0)\n\t/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/vendor/code.cloudfoundry.org/lager/logger.go:131 +0xc9\ngithub.com/Azure/oms-log-analytics-firehose-nozzle/caching.(*Caching).Initialize(0xc420014bc0)\n\t/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/caching/caching.go:39 +0x5ae\ngithub.com/Azure/oms-log-analytics-firehose-nozzle/omsnozzle.(*OmsNozzle).Start(0xc42008e750, 0xc42000de30, 0xc4200dce00)\n\t/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/omsnozzle/oms_nozzle.go:67 +0x3d\nmain.main()\n\t/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/main.go:162 +0x1f13\n"}}
   �[31m2017-08-23T11:42:04.25-0600 [APP/PROC/WEB/1] ERR panic: Could not get api /v2/info: json: cannot unmarshal number into Go value of type cfclient.Endpoint�[0m
   �[31m2017-08-23T11:42:04.25-0600 [APP/PROC/WEB/1] ERR goroutine 1 [running]:�[0m
   �[31m2017-08-23T11:42:04.25-0600 [APP/PROC/WEB/1] ERR panic(0x55ed1f4810a0, 0xc420156b00)�[0m
   �[31m2017-08-23T11:42:04.25-0600 [APP/PROC/WEB/1] ERR 	/tmp/app/.cloudfoundry/0/go1.7.6/go/src/runtime/panic.go:500 +0x1a5�[0m
   �[31m2017-08-23T11:42:04.25-0600 [APP/PROC/WEB/1] ERR github.com/Azure/oms-log-analytics-firehose-nozzle/vendor/code.cloudfoundry.org/lager.(*logger).Fatal(0xc420018660, 0x55ed1f1b355d, 0x17, 0x55ed1f6a1760, 0xc420156b00, 0x0, 0x0, 0x0)�[0m
   �[31m2017-08-23T11:42:04.25-0600 [APP/PROC/WEB/1] ERR 	/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/vendor/code.cloudfoundry.org/lager/logger.go:152 +0x41e�[0m
   �[31m2017-08-23T11:42:04.25-0600 [APP/PROC/WEB/1] ERR github.com/Azure/oms-log-analytics-firehose-nozzle/caching.(*Caching).Initialize(0xc420014bc0)�[0m
   �[31m2017-08-23T11:42:04.25-0600 [APP/PROC/WEB/1] ERR 	/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/caching/caching.go:39 +0x5ae�[0m
   �[31m2017-08-23T11:42:04.25-0600 [APP/PROC/WEB/1] ERR github.com/Azure/oms-log-analytics-firehose-nozzle/omsnozzle.(*OmsNozzle).Start(0xc42008e750, 0xc42000de30, 0xc4200dce00)�[0m
   �[31m2017-08-23T11:42:04.25-0600 [APP/PROC/WEB/1] ERR 	/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/omsnozzle/oms_nozzle.go:67 +0x3d�[0m
   �[31m2017-08-23T11:42:04.25-0600 [APP/PROC/WEB/1] ERR main.main()�[0m
   �[31m2017-08-23T11:42:04.25-0600 [APP/PROC/WEB/1] ERR 	/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/main.go:162 +0x1f13�[0m
   2017-08-23T11:42:04.25-0600 [APP/PROC/WEB/1] OUT Exit status 2
   2017-08-23T11:42:04.26-0600 [CELL/1] OUT Exit status 0
   2017-08-23T11:42:04.27-0600 [CELL/1] OUT Destroying container
   2017-08-23T11:42:04.27-0600 [APP/PROC/WEB/0] OUT {"timestamp":"1503510124.275655746","source":"oms-nozzle","message":"oms-nozzle.error creating cfclient","log_level":3,"data":{"error":"Could not get api /v2/info: json: cannot unmarshal number into Go value of type cfclient.Endpoint","trace":"goroutine 1 [running]:\ngithub.com/Azure/oms-log-analytics-firehose-nozzle/vendor/code.cloudfoundry.org/lager.(*logger).Fatal(0xc420018660, 0x5643c60ea55d, 0x17, 0x5643c65d8760, 0xc42000da10, 0x0, 0x0, 0x0)\n\t/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/vendor/code.cloudfoundry.org/lager/logger.go:131 +0xc9\ngithub.com/Azure/oms-log-analytics-firehose-nozzle/caching.(*Caching).Initialize(0xc420110280)\n\t/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/caching/caching.go:39 +0x5ae\ngithub.com/Azure/oms-log-analytics-firehose-nozzle/omsnozzle.(*OmsNozzle).Start(0xc4201205a0, 0xc4200fc478, 0xc42011c380)\n\t/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/omsnozzle/oms_nozzle.go:67 +0x3d\nmain.main()\n\t/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/main.go:162 +0x1f13\n"}}
   �[31m2017-08-23T11:42:04.27-0600 [APP/PROC/WEB/0] ERR panic: Could not get api /v2/info: json: cannot unmarshal number into Go value of type cfclient.Endpoint�[0m
   �[31m2017-08-23T11:42:04.27-0600 [APP/PROC/WEB/0] ERR goroutine 1 [running]:�[0m
   �[31m2017-08-23T11:42:04.27-0600 [APP/PROC/WEB/0] ERR panic(0x5643c63b80a0, 0xc42000da10)�[0m
   �[31m2017-08-23T11:42:04.27-0600 [APP/PROC/WEB/0] ERR 	/tmp/app/.cloudfoundry/0/go1.7.6/go/src/runtime/panic.go:500 +0x1a5�[0m
   �[31m2017-08-23T11:42:04.27-0600 [APP/PROC/WEB/0] ERR github.com/Azure/oms-log-analytics-firehose-nozzle/vendor/code.cloudfoundry.org/lager.(*logger).Fatal(0xc420018660, 0x5643c60ea55d, 0x17, 0x5643c65d8760, 0xc42000da10, 0x0, 0x0, 0x0)�[0m
   �[31m2017-08-23T11:42:04.27-0600 [APP/PROC/WEB/0] ERR 	/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/vendor/code.cloudfoundry.org/lager/logger.go:152 +0x41e�[0m
   �[31m2017-08-23T11:42:04.27-0600 [APP/PROC/WEB/0] ERR github.com/Azure/oms-log-analytics-firehose-nozzle/caching.(*Caching).Initialize(0xc420110280)�[0m
   �[31m2017-08-23T11:42:04.27-0600 [APP/PROC/WEB/0] ERR 	/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/caching/caching.go:39 +0x5ae�[0m
   �[31m2017-08-23T11:42:04.27-0600 [APP/PROC/WEB/0] ERR github.com/Azure/oms-log-analytics-firehose-nozzle/omsnozzle.(*OmsNozzle).Start(0xc4201205a0, 0xc4200fc478, 0xc42011c380)�[0m
   �[31m2017-08-23T11:42:04.27-0600 [APP/PROC/WEB/0] ERR 	/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/omsnozzle/oms_nozzle.go:67 +0x3d�[0m
   �[31m2017-08-23T11:42:04.27-0600 [APP/PROC/WEB/0] ERR main.main()�[0m
   �[31m2017-08-23T11:42:04.27-0600 [APP/PROC/WEB/0] ERR 	/tmp/gobuildpack.gopath858617533/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/main.go:162 +0x1f13�[0m
   2017-08-23T11:42:04.27-0600 [APP/PROC/WEB/0] OUT Exit status 2
   2017-08-23T11:42:04.29-0600 [CELL/0] OUT Exit status 0
   2017-08-23T11:42:04.30-0600 [API/0] OUT Process has crashed with type: "web"
   2017-08-23T11:42:04.30-0600 [CELL/0] OUT Destroying container
   2017-08-23T11:42:04.31-0600 [API/0] OUT App instance exited with guid a1e48e70-b22d-4bc1-b1d1-d1451823782a payload: {"instance"=>"c6ccb8f3-21fb-4baa-7780-f990", "index"=>1, "reason"=>"CRASHED", "exit_description"=>"2 error(s) occurred:\n\n* Exited with status 2\n* cancelled", "crash_count"=>6, "crash_timestamp"=>1503510124283089599, "version"=>"6f568a4d-fce4-4879-a8e3-caeb22a0c267"}
   2017-08-23T11:42:04.34-0600 [API/1] OUT Process has crashed with type: "web"
   2017-08-23T11:42:04.35-0600 [API/1] OUT App instance exited with guid a1e48e70-b22d-4bc1-b1d1-d1451823782a payload: {"instance"=>"c1f6cb17-9ea9-4cc1-49b5-d501", "index"=>0, "reason"=>"CRASHED", "exit_description"=>"2 error(s) occurred:\n\n* Exited with status 2\n* cancelled", "crash_count"=>6, "crash_timestamp"=>1503510124310964299, "version"=>"6f568a4d-fce4-4879-a8e3-caeb22a0c267"}
   2017-08-23T11:42:04.73-0600 [CELL/1] OUT Successfully destroyed container
   2017-08-23T11:42:04.78-0600 [CELL/0] OUT Successfully destroyed container

I used the latest commit from master.

Slow consumer alert from traffic controller might has been changed

Previous alerts in two ways:
the nozzle receives a WebSocket close error with error code ClosePolicyViolation (1008)
the nozzle receives a CounterEvent with the name TruncatingBuffer.DroppedMessages

The latest document has changed to:
If a nozzle falls behind, Loggregator alerts the nozzle in two ways:
TruncatingBuffer alerts: If the nozzle consumes messages more slowly than they are produced, the Loggregator system may drop messages. In this case, Loggregator sends the log message, TB: Output channel too full. Dropped N messages, where N is the number of dropped messages. Loggregator also emits a CounterEvent with the name doppler_proxy.slow_consumer. The nozzle receives both messages from the Firehose, alerting the operator to the performance issue.

How to confirm if there's no problem on Nozzle in case of log message lost

I'm facing an issue that some of log messages generated by "cf push"-ed application might be lost when I try to confirm them from Azure OMS by using KUSTO query.

Here's a detail regarding my facing issue:
1, An application, hereafter calling it as A, generates some logs and these are forwarded to OMS via Nozzle.
2, I assume that all generated logs from A are preserved at least when they pass through Nozzle, as I confirmed them with "cf app-nozzle A" plugin.
3, However, when I try to find them through Azure OMS with KUSTO query, some of them are lost. (=wasn't displayed as the result of query.)

As of now, what I've done is:
1, Ask Microsoft to investigate it from OMS side, and I shared syslog of VMs which Nozzle is deployed on because of their request, the results of "cf app-nozzle A" and that of "cf logs A".
They can't specify the root cause so far.
2, They recommended me to raise a case here in GitHub, and I'm just issuing now.

My questions are as follows:
1, I expect that I can affirm there's no problem in or inside Nozzle as all logs are preserved when they're passing through Nozzle. However, I should confirm if there's other way to confirm there's no problem as well.
Do you have any log that would help my investigation?
2, Do you have any idea which logs I should check regarding this case?

Thank you in advance.

Add an exclusion list property for excluding apps belonging to a set of orgs and spaces

We have a use-case for separating logs from different spaces of a CF foundation.

For this, we use two instances of the Nozzle (replicating the Azure Log Analytics Nozzle tile), and we apply two different (but complementary) configurations to each one.

For example, let's say we have a Foundation with an ACME org, and spaces like development, staging, preprod and prod. When we need to isolate the logs for prod space, we setup this space_whitelist for the second Nozzle:

space_whitelist:
  ACME.prod

But for the main Nozzle, we must built a kind of “catch-all” space_whitelist to capture everything else.

space_whitelist:
  system.*,
  eventhub-firehose.*,
  credhub-service-broker-org.*,
  ACME.development,
  ACME.staging,
  ACME.preprod

The drawback is that any time anyone (or anything) adds a new org or a new space to the ACME org, then we must re-configure the main Nozzle to include it in the “catch-all” space_whitelist of the main Nozzle. This breaks the self-service nature of Cloud Foundry, that lets developers create orgs and spaces without any extra toil for platform engineers.

The missing feature that we need here would be a space_exclusion_list that would do the opposite of space_whitelist. Then, the config of the main Nozzle would be super easy.

space_exclusion_list:
  ACME.prod

And we wouldn't have any new toil anytime a new org or a new space is created in the ACME org.

Don't hesitate to ask for any further information in order to fully understand this feature request.

Best,
Benjamin

Filtering specific metric keys

Regarding to the filtering,

EVENT_FILTER : Event types to be filtered out. The format is a comma separated list, valid event types are METRIC,LOG,HTTP

This setting will stop the whole inject to each table (e.g. CF_ValueMetric_CL ) in Log analytics.
I have many, however not all, metric keys that are not used in ValueMetric table.
In order to reduce the azure usage cost from the log injection, we would like to filter some specific metric keys.

Is there any way to do that?

incompatible type (incompatible type for method 'Debug' (different parameter types)), in main.go

Hi.
I was not able run the main executable, on Ubuntu 20.04 x86_64:

$ go run main.go

command-line-arguments

./main.go:169:39: error: argument 1 has incompatible type
./main.go:169:71: error: argument 3 has incompatible type (incompatible type for method 'Debug' (different parameter types))
./main.go:171:76: error: argument 4 has incompatible type (incompatible type for method 'Debug' (different parameter types))
./main.go:184:38: error: argument 1 has incompatible type
./main.go:184:54: error: argument 2 has incompatible type (incompatible type for method 'Debug' (different parameter types))
./main.go:185:35: error: argument 1 has incompatible type (incompatible type for method 'Debug' (different parameter types))

$ go version
go version go1.15.2 gollvm LLVM 12.0.0git linux/amd64

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/oceanfish81/.cache/go-build"
GOENV="/home/oceanfish81/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/oceanfish81/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/oceanfish81/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/oceanfish81/gollvm_dist"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/oceanfish81/gollvm_dist/tools"
GCCGO="/home/oceanfish81/gollvm_dist/bin/llvm-goc"
AR="ar"
CC="/usr/bin/clang"
CXX="/usr/bin/clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build809405926=/tmp/go-build -gno-record-gcc-switches -funwind-tables"

Nozzle App is crashing constantly

Nozzle App is crashing constantly we deployed the newer version of PCF tile 1.4.2. App is not stable and crashing often.

ERROR:
2020-11-24T11:57:58.386+05:30 [APP/PROC/WEB/2] [OUT] {'timestamp':'1606199278.385876656','source':'oms-nozzle','message':'oms-nozzle.Error while reading from the firehose','log_level':2,'data':{'error':'websocket: close 1006 (abnormal closure): unexpected EOF'}}

2020-11-30T14:52:42.622+05:30 [APP/PROC/WEB/12] [OUT] {"timestamp":"1606728162.622236252","source":"oms-nozzle","message":"oms-nozzle.App info not found for GUID","log_level":1,"data":{"guid":"5ce0c225-ccf4-4389-97a8-6a43c1ec7849"}}
2020-11-30T14:52:42.626+05:30 [APP/PROC/WEB/0] [OUT] {"timestamp":"1606728162.626188993","source":"oms-nozzle","message":"oms-nozzle.adding to app info cache","log_level":0,"data":{"guid":"","info":{"name":"","org":"","orgId":"","space":"","spaceId":"","monitored":true}}}
2020-11-30T14:52:42.626+05:30 [APP/PROC/WEB/0] [OUT] {"timestamp":"1606728162.626352787","source":"oms-nozzle","message":"oms-nozzle.App info not found for GUID","log_level":1,"data":{"guid":"5ce0c225-ccf4-4389-97a8-6a43c1ec7849"}}
2020-11-30T14:52:42.642+05:30 [APP/PROC/WEB/3] [OUT] {"timestamp":"1606728162.641838789","source":"oms-nozzle","message":"oms-nozzle.adding to app info cache","log_level":0,"data":{"guid":"","info":{"name":"","org":"","orgId":"","space":"","spaceId":"","monitored":true}}}
2020-11-30T14:52:42.642+05:30 [APP/PROC/WEB/3] [OUT] {"timestamp":"1606728162.641998053","source":"oms-nozzle","message":"oms-nozzle.App info not found for GUID","log_level":1,"data":{"guid":"5ce0c225-ccf4-4389-97a8-6a43c1ec7849"}}
2020-11-30T14:52:42.783+05:30 [APP/PROC/WEB/10] [OUT] {"timestamp":"1606728162.783232689","source":"oms-nozzle","message":"oms-nozzle.adding to app info cache","log_level":0,"data":{"guid":"","info":{"name":"","org":"","orgId":"","space":"","spaceId":"","monitored":true}}}
2020-11-30T14:52:42.783+05:30 [APP/PROC/WEB/10] [OUT] {"timestamp":"1606728162.783447504","source":"oms-nozzle","message":"oms-nozzle.App info not found for GUID","log_level":1,"data":{"guid":"8b84e70c-62f8-4dab-9326-f4e6e23a3eda"}}

column explanation document

Is there any document to explain means of column for Developers ?
For example, TimeGenerated and EventTime are confusing for Developers without reading source code.

I want to some kind of list to explain columns

Failed to compile due to the removal of the support for Go 1.7.x in go_buildpack

       FAILED  

          2018-10-15T05:04:53.99+0000 [STG/0] OUT -----> Go Buildpack version 1.8.28  
          2018-10-15T05:04:53.99+0000 [STG/0] OUT        [31;1m**WARNING**[0m buildpack version changed from 1.8.27 to 1.8.28  
          2018-10-15T05:04:54.66+0000 [STG/0] OUT        [31;1m**ERROR**[0m Unable to determine Go version to install: no match found for 1.7.x in [1.8.6 1.8.7 1.9.6 1.9.7 1.10.3 1.10.4 1.11 1.11.1]  
          2018-10-15T05:04:54.67+0000 [STG/0] ERR Failed to compile droplet: Failed to run all supply scripts: exit status 16

supporting PCF 2.0

will this nozzle be "officially" compatible for PCF 2.0 (meaning be downloadable as "depends on PCF 2.0.x" any time soon?

Offline environment will not be able to use Pivotal tile

Offline environment will not be able to download buildpack

Log Analytics Nozzle should support Azure GovCloud

I have a customer looking to use the nozzle on Azure GovCloud. Currently the Nozzle does not support Azure GovCloud endpoints for the Log Analytics Workspace.

@nsagoo-pivotal for additional details

Nozzle app is crashing with DNS lookup issues

2020-01-20T15:50:59.71-0800 [APP/PROC/WEB/4] ERR panic: Could not get api /v2/info: Get https://api.system.prod.**.**.com/v2/info: dial tcp: lookup api.system.prod.stratus.albertsons.com on 169.254.0.2:53: read udp 10.255.79.151:42306->169.254.0.2:53: i/o timeout
2020-01-20T15:50:59.71-0800 [APP/PROC/WEB/4] ERR goroutine 1 [running]:
2020-01-20T15:50:59.71-0800 [APP/PROC/WEB/4] ERR github.com/Azure/oms-log-analytics-firehose-nozzle/vendor/code.cloudfoundry.org/lager.(*logger).Fatal(0xc4201204e0, 0x5595c357001b, 0x17, 0x5595c38f8940, 0xc420160850, 0x0, 0x0, 0x0)
2020-01-20T15:50:59.71-0800 [APP/PROC/WEB/4] ERR /tmp/gobuildpack.gopath476918737/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/vendor/code.cloudfoundry.org/lager/logger.go:152 +0x4d4
2020-01-20T15:50:59.71-0800 [APP/PROC/WEB/4] ERR github.com/Azure/oms-log-analytics-firehose-nozzle/caching.(*Caching).Initialize(0xc420172540, 0x5595c31f4101)
2020-01-20T15:50:59.71-0800 [APP/PROC/WEB/4] ERR /tmp/gobuildpack.gopath476918737/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/caching/caching.go:99 +0x39f
2020-01-20T15:50:59.71-0800 [APP/PROC/WEB/4] ERR github.com/Azure/oms-log-analytics-firehose-nozzle/omsnozzle.(*OmsNozzle).Start(0xc420186000, 0xc420156320, 0xc420184000)
2020-01-20T15:50:59.71-0800 [APP/PROC/WEB/4] ERR /tmp/gobuildpack.gopath476918737/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/omsnozzle/oms_nozzle.go:69 +0x3f
2020-01-20T15:50:59.71-0800 [APP/PROC/WEB/4] ERR main.main()
2020-01-20T15:50:59.71-0800 [APP/PROC/WEB/4] ERR /tmp/gobuildpack.gopath476918737/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/main.go:187 +0x1af2

This repo is missing important files

There are important files that Microsoft projects should all have that are not present in this repository. A pull request has been opened to add the missing file(s). When the pr is merged this issue will be closed automatically.

Microsoft teams can learn more about this effort and share feedback within the open source guidance available internally.

Merge this pull request

Support app rename

Hello,

The Nozzle does not seem to support the renaming of a PCF application as its name is cached once on the start of the [Nozzle] instance or when the first message goes through it and is never invalidated. This behavior causes a mismatch in the ApplicationName field of indexed logs when you do blue-green deployments on PCF.

Steps to reproduce

Create and start an application named appA that emits some logs
Renaming this app into appB
Check logs in the OMS Workspace, they are still incorrectly indexed with appA as the ApplicationName

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.