exporter/stackdriver: creation fails with AlreadyExists for new metrics about opencensus-go

[songy23]

Hi @ChrisBartonLC , this error is probably because you updated your view definition after exporting it to Stackdriver Monitoring. Currently our exporter doesn't support updating a view (or creating a new view with existing names). For now please try deleting the conflicted metric descriptors from Stackdriver before exporting.

[rakyll]

This is coming from CreateMetricDescriptor if the metric descriptor is just newly created. Needs more investigation...

[songy23]

I suppose #162 should fix this? For Stackdriver, it's ok to create the exact same metric descriptor multiple times, however we'll get an AlreadyExists error if we try to create a metric descriptor that has the same name but with different attributes (e.g aggregation or window).

[rakyll]

I cannot reproduce it anymore hence closing, I will reopen it if I see it again.

[lc-chrisbarton]

We get the following errors:

2018/04/03 07:02:37 Failed to export to Stackdriver Monitoring: rpc error: code = AlreadyExists desc = Errors during metric descriptor creation: {(metric: custom.googleapis.com/opencensus/cloud.google.com/go/pubsub/stream_request_count, error: The metric already exists.)}.
2018/04/03 07:02:47 Failed to export to Stackdriver Monitoring: rpc error: code = AlreadyExists desc = Errors during metric descriptor creation: {(metric: custom.googleapis.com/opencensus/cloud.google.com/go/pubsub/stream_response_count, error: The metric already exists.)}.
2018/04/03 06:59:33 Failed to export to Stackdriver Monitoring: rpc error: code = AlreadyExists desc = Errors during metric descriptor creation: {(metric: custom.googleapis.com/opencensus/cloud.google.com/go/pubsub/stream_open_count, error: The metric already exists.)}.

When using code like this with go.opencensus.io master and cloud.google.com/go master deps:

	stackdriverExporter, err := stackdriver.NewExporter(stackdriver.Options{ProjectID: os.Getenv("GCP_PROJECT_ID")})
	if err != nil {
		share_log.Logger.Fatal(ctx, "Failed initializing statistics - opencensus", share_log.Logger.FmtError(err))
	}
	view.RegisterExporter(stackdriverExporter)
	trace.RegisterExporter(stackdriverExporter)
	for _, v := range []*view.View{
		pubsub.PullCountView,
		pubsub.AckCountView,
		pubsub.NackCountView,
		pubsub.ModAckCountView,
		pubsub.StreamOpenCountView,
		pubsub.StreamRetryCountView,
		pubsub.StreamRequestCountView,
		pubsub.StreamResponseCountView,
	} {
		if err := v.Subscribe(); err != nil {
			share_log.Logger.Fatal(ctx, "Failed subscribing to statistics view - opencensus", share_log.Logger.FmtError(err))
		}
		share_log.Logger.Info(ctx, "Subscribed to statistics view - opencensus", share_log.Logger.FmtString("View", v.Name))
	}
	share_log.Logger.Info(ctx, "Initialized statistics - opencensus")

[lc-chrisbarton]

Ok deleted them after modifying the demo code from "google.golang.org/api/monitoring/v3" to "monitoring "google.golang.org/api/monitoring/v3" and now the error is:

 Failed to export to Stackdriver Monitoring: stackdriver metric descriptor was not created with aggregation type view.AggType

[songy23]

@ChrisBartonLC Thanks for reporting, this is related to a recent change to Aggregation. #664 should fix it.

[lc-chrisbarton]

Thanks! Works now after using your latest code on master :)

[lc-chrisbarton]

Hi I updated this morning and swapped to to using the new 'register' api for setting up the view.

Unfortunately I can no longer delete metrics even though the delete does not return an error:

INFO  2018/04/05 09:01:31.888517 BACKGROUND Deleting metric { "metric": "projects/localcover-55/metricDescriptors/custom.googleapis.com/opencensus/cloud.google.com/go/pubsub/stream_request_count" } main.main line 64 /Users/tanh/projects/localcover/dev/code_lc/lc/lc-api/lc-api-customer/src/lc-api-customer/tools/stackdriver-metric-delete/main.go
INFO  2018/04/05 09:01:34.330270 BACKGROUND Deleted metric { "metric": "projects/localcover-55/metricDescriptors/custom.googleapis.com/opencensus/cloud.google.com/go/pubsub/stream_request_count", "response": {} } main.main line 71 /Users/tanh/projects/localcover/dev/code_lc/lc/lc-api/lc-api-customer/src/lc-api-customer/tools/stackdriver-metric-delete/main.go

Metric still exists:

2018/04/05 09:00:45 Failed to export to Stackdriver Monitoring: rpc error: code = AlreadyExists desc = Errors during metric descriptor creation: {(metric: custom.googleapis.com/opencensus/cloud.google.com/go/pubsub/stream_request_count, error: The metric already exists.)}.

[songy23]

Hi @ChrisBartonLC , can you show us how you defined StreamRequestCountView?

[lc-chrisbarton]

	stackdriverExporter, err := stackdriver.NewExporter(stackdriver.Options{ProjectID: os.Getenv("GCP_PROJECT_ID")})
	if err != nil {
		share_log.Logger.Fatal(ctx, "Failed initializing stackdriver", share_log.Logger.FmtError(err))
	}

	trace.RegisterExporter(stackdriverExporter)
	share_log.Logger.Info(ctx, "Registered stackdriver tracing")

	view.RegisterExporter(stackdriverExporter)
	if err := view.Register(
		pubsub.PullCountView,
		pubsub.AckCountView,
		pubsub.NackCountView,
		pubsub.ModAckCountView,
		pubsub.StreamOpenCountView,
		pubsub.StreamRetryCountView,
		pubsub.StreamRequestCountView,
		pubsub.StreamResponseCountView,
	); err != nil {
		share_log.Logger.Fatal(ctx, "Failed registering stackdriver statistics", share_log.Logger.FmtError(err))
	}
	share_log.Logger.Info(ctx, "Registered stackdriver statistics")

[songy23]

Is this issue specific to StreamRequestCountView, or all other views have the same issue?

I suspect this is because StreamRequestCountView is registered somewhere more than once and have different definitions. Might be related to
googleapis/google-cloud-go@b475e33#diff-38bb0d2f62f8d9a759252a81a1086486?

/cc @rakyll

[lc-chrisbarton]

Yep we are only getting errors related to StreamRequestCountView.

Here are all the metrics from "https://cloud.google.com/monitoring/api/ref_v3/rest/v3/projects.metricDescriptors/list" using the filter "metric.type = starts_with("custom.googleapis.com/")"

{
  "metricDescriptors": [
    {
      "name": "projects/localcover-55/metricDescriptors/custom.googleapis.com/opencensus/cloud.google.com/go/pubsub/ack_count",
      "labels": [
        {
          "key": "subscription"
        },
        {
          "key": "opencensus_task",
          "description": "Opencensus task identifier"
        }
      ],
      "metricKind": "CUMULATIVE",
      "valueType": "INT64",
      "unit": "1",
      "description": "Number of PubSub messages acked",
      "displayName": "OpenCensus/cloud.google.com/go/pubsub/ack_count",
      "type": "custom.googleapis.com/opencensus/cloud.google.com/go/pubsub/ack_count"
    },
    {
      "name": "projects/localcover-55/metricDescriptors/custom.googleapis.com/opencensus/cloud.google.com/go/pubsub/mod_ack_count",
      "labels": [
        {
          "key": "subscription"
        },
        {
          "key": "opencensus_task",
          "description": "Opencensus task identifier"
        }
      ],
      "metricKind": "CUMULATIVE",
      "valueType": "INT64",
      "unit": "1",
      "description": "Number of ack-deadlines modified",
      "displayName": "OpenCensus/cloud.google.com/go/pubsub/mod_ack_count",
      "type": "custom.googleapis.com/opencensus/cloud.google.com/go/pubsub/mod_ack_count"
    },
    {
      "name": "projects/localcover-55/metricDescriptors/custom.googleapis.com/opencensus/cloud.google.com/go/pubsub/nack_count",
      "labels": [
        {
          "key": "subscription"
        },
        {
          "key": "opencensus_task",
          "description": "Opencensus task identifier"
        }
      ],
      "metricKind": "CUMULATIVE",
      "valueType": "INT64",
      "unit": "1",
      "description": "Number of PubSub messages nacked",
      "displayName": "OpenCensus/cloud.google.com/go/pubsub/nack_count",
      "type": "custom.googleapis.com/opencensus/cloud.google.com/go/pubsub/nack_count"
    },
    {
      "name": "projects/localcover-55/metricDescriptors/custom.googleapis.com/opencensus/cloud.google.com/go/pubsub/pull_count",
      "labels": [
        {
          "key": "subscription"
        },
        {
          "key": "opencensus_task",
          "description": "Opencensus task identifier"
        }
      ],
      "metricKind": "CUMULATIVE",
      "valueType": "INT64",
      "unit": "1",
      "description": "Number of PubSub messages pulled",
      "displayName": "OpenCensus/cloud.google.com/go/pubsub/pull_count",
      "type": "custom.googleapis.com/opencensus/cloud.google.com/go/pubsub/pull_count"
    },
    {
      "name": "projects/localcover-55/metricDescriptors/custom.googleapis.com/opencensus/cloud.google.com/go/pubsub/stream_open_count",
      "labels": [
        {
          "key": "subscription"
        },
        {
          "key": "opencensus_task",
          "description": "Opencensus task identifier"
        }
      ],
      "metricKind": "CUMULATIVE",
      "valueType": "INT64",
      "unit": "1",
      "description": "Number of calls opening a new streaming pull",
      "displayName": "OpenCensus/cloud.google.com/go/pubsub/stream_open_count",
      "type": "custom.googleapis.com/opencensus/cloud.google.com/go/pubsub/stream_open_count"
    },
    {
      "name": "projects/localcover-55/metricDescriptors/custom.googleapis.com/opencensus/cloud.google.com/go/pubsub/stream_request_count",
      "labels": [
        {
          "key": "subscription"
        },
        {
          "key": "opencensus_task",
          "description": "Opencensus task identifier"
        }
      ],
      "metricKind": "CUMULATIVE",
      "valueType": "INT64",
      "unit": "1",
      "description": "Number gRPC StreamingPull request messages sent",
      "displayName": "OpenCensus/cloud.google.com/go/pubsub/stream_request_count",
      "type": "custom.googleapis.com/opencensus/cloud.google.com/go/pubsub/stream_request_count"
    },
    {
      "name": "projects/localcover-55/metricDescriptors/custom.googleapis.com/opencensus/cloud.google.com/go/pubsub/stream_response_count",
      "labels": [
        {
          "key": "subscription"
        },
        {
          "key": "opencensus_task",
          "description": "Opencensus task identifier"
        }
      ],
      "metricKind": "CUMULATIVE",
      "valueType": "INT64",
      "unit": "1",
      "description": "Number of gRPC StreamingPull response messages received",
      "displayName": "OpenCensus/cloud.google.com/go/pubsub/stream_response_count",
      "type": "custom.googleapis.com/opencensus/cloud.google.com/go/pubsub/stream_response_count"
    },
    {
      "name": "projects/localcover-55/metricDescriptors/custom.googleapis.com/opencensus/cloud.google.com/go/pubsub/stream_retry_count",
      "labels": [
        {
          "key": "subscription"
        },
        {
          "key": "opencensus_task",
          "description": "Opencensus task identifier"
        }
      ],
      "metricKind": "CUMULATIVE",
      "valueType": "INT64",
      "unit": "1",
      "description": "Number of retries of a stream send or receive",
      "displayName": "OpenCensus/cloud.google.com/go/pubsub/stream_retry_count",
      "type": "custom.googleapis.com/opencensus/cloud.google.com/go/pubsub/stream_retry_count"
    },
    {
      "name": "projects/localcover-55/metricDescriptors/custom.googleapis.com/opencensus/grpc.io/client/error_count/cumulative",
      "labels": [
        {
          "key": "canonical_status"
        },
        {
          "key": "method"
        },
        {
          "key": "opencensus_task",
          "description": "Opencensus task identifier"
        }
      ],
      "metricKind": "CUMULATIVE",
      "valueType": "DISTRIBUTION",
      "unit": "1",
      "description": "RPC Errors",
      "displayName": "OpenCensus/grpc.io/client/error_count/cumulative",
      "type": "custom.googleapis.com/opencensus/grpc.io/client/error_count/cumulative"
    },
    {
      "name": "projects/localcover-55/metricDescriptors/custom.googleapis.com/opencensus/grpc.io/client/request_count/cumulative",
      "labels": [
        {
          "key": "method"
        },
        {
          "key": "opencensus_task",
          "description": "Opencensus task identifier"
        }
      ],
      "metricKind": "CUMULATIVE",
      "valueType": "DISTRIBUTION",
      "unit": "1",
      "description": "Number of client RPC request messages",
      "displayName": "OpenCensus/grpc.io/client/request_count/cumulative",
      "type": "custom.googleapis.com/opencensus/grpc.io/client/request_count/cumulative"
    }
  ]
}

[lc-chrisbarton]

Actually it is ok now, sorry :( seems another developer must have been using the old code so we kept clashing

[songy23]

Cool :)