Fix flaky monitoring integration test about openjdk-runtime

[meltsufin]

@nkubala has anyone successfully implemented this test from your common integration testing framework?

[balopat]

@nkubala I created a PR to tackle these: GoogleCloudPlatform/runtimes-common#306

[nkubala]

has anyone successfully implemented this test from your common integration testing framework?

@meltsufin @balopat I wrote the sample integration test application for Python myself, so you can use that as an example if you run into issues. Here's the monitoring test: https://github.com/GoogleCloudPlatform/python-runtime/blob/master/tests/integration/server.py#L140

[balopat]

thanks @nkubala based on what I see there is no significant difference from my implementation.
The 400 error seems like a timing issue, it takes a long time for Stackdriver to actually expose the new custom metric, and even the 80 seconds are not enough sometimes (8s x 10 retry). I'm wondering if the creation of a new metric at every request is necessary? I'd think that not having to create the new metric would speed things up, and would still count as a good integration test as long as we write the unique token into the timeseries.
If we don't do that, then I'd have to resort to waiting until the metric is created in the test app code.

[nkubala]

@balopat creating a new metric descriptor isn't necessary, but thought it would be a valid component of the test. if it's totally blocking your test from passing then we can revisit it, but I'd rather not make that change as other runtimes have been able to write the metrics successfully. could you maybe come up with a lower bound on the wait time for the metric to propagate to Stackdriver, and then we can just up the wait time in the test?

[balopat]

@nkubala I am working on a test suite to narrow down the issues with the StackDriver monitoring when creating a metric - sometimes even the 80 seconds are not enough, when I start reading "too quickly" after the metric was created. This suggests that creating new metrics is a "heavy" operation and can interfere with subsequent reads within a certain "cool down period", while with an already created metric I had absolutely no problem.
But there is another (minor) problem with custom metrics: there is a quota policy for custom metrics - which I'm not sure everyone is aware of and would cause surprises down the line unless we setup cleanup logic for them.

[nkubala]

Yeah I think the main issue with the reading of the metric is that the propagation time/time it takes to flush the writes to Stackdriver varies greatly, so I think that sufficiently increasing the timeout there should probably solve that problem.

As for the quota, yeah that's something we'll need to address. We should probably just cleanup the metric descriptors we create in the tests. #302