How to debug 'Unable to contact server'? about cognitive-services-speech-sdk-js

[gleitz]

This is really embarrassing. 2+ days with no resolution? I realize you're working to resolve but this level of outage does not inspire confidence to continue building anything mission critical on this service, let alone pay for it.

[amenella]

Hi,

Unfortunately we are actually facing the issue (once again)...

The additional logs gave us the following details:

• Reason: Error
• Error details: Unable to contact server. StatusCode: 1006, undefined Reason: Failed to obtain OCSP response: 520
• Error code: ConnectionFailure
• Session id: 5********************************************4

Those additional logs are just confirming @rhurey comment about the OCSP:

The service that hosts Microsoft's OCSP server is down, so when the NodeJS client tries to connect it cannot validate that the certificate hasn't been revoked and fails the connection.

Now that we are sure about this, how could we move forward?
Is there an alternative reliable service?
Would it be possible to bypass the OCSP service when they are down?

[amenella]

Hi, we are regularly facing the current issue, it is randomly raising then magically disappearing after some delays.

Here is the evolution for yesterday:

First error appeared at 3pm CET and disappeared around 4pm CET (for a total of 31 occurences).

It is currently happening on our production.

What can be done to properly fix this issue?

[rhurey]

CRL fallback is more complex than anticipated, so we're looking at a multi-part improvement.

Part 1 is to allow the OCSP check to be disabled. If SPEECH_CONDUCT_OCSP_CHECK is set to 0, certificate revocation checks will be skipped. This change is in master.

Part 2 will be to cache OCSP responses for their validity period.

Part 3 will be CRL failback.

[rhurey]

It's not specific to you. I'm trying to figure out if it's something in the OCSP request we're issuing or if our cert issuer is at fault.

[iguana]

Is there a reference for the speech web socket API so that we may write our own client?

[rhurey]

First, thanks for the graphs. I've attached them to the internal ticket and made sure it's correctly owned by the team that manages our certificate serverices.

I can't recommend disabling the certificate checks in NodeJS as it's a big hammer that removes all checks, including service identity and issuer. It basically leaves clients wide open to being man-in-the-middle snooped.

We had an item put in our backlog earlier to implement fallback to CRL's when OCSP fails. I'm working on a prototype now.

[rhurey]

There’s a service issue currently with the TTS service accepting authorization tokens from a browser. A fix is in the deployment pipeline.

[oscholz]

@gleitz Sorry you are hitting this again. We are as upset about this as I imagine you are. We are going to make this a high priority fix in our current sprint.
Frankly, we didn't see any problems with this until it hit in March, and now it seems to be a regular thing. We are also going to dig deeper internally to figure out how to fix the underlying issue.
Again, our apologies for this issue! We are definitely invested in the JS Speech SDK, and we hope you continue to be, too! I know this sucks right now!

[eldakms]

Which data center region are you using?

[lcswillems]

I use francecentral

[eldakms]

Can you try other data center? there was no changes in francecentral in the recent days. I checked your session id - it does not present on the server side, so it seems the connection cannot even be established.

[lcswillems]

Okay, thank you for your fast answers! I will try another region

[lcswillems]

I have tried other regions and I have exactly the same issue...

[eldakms]

Hmm, looks like a client problem then. Maybe others can help

[lcswillems]

Okay, but this is strange because I am able, from my node application, to request other websites...

[eldakms]

which other data regions did you try?

[eldakms]

I see traffic on france central, are you sure you have not updated the version of JavaScript speech engine that you are using?

[lcswillems]

I have tried all the european regions, and one in each continents. For canadacentral, the recognizer took a lot of time to load but still finished with same error:

SpeechRecognitionCanceledEventArgs {
  privSessionId: 'D2D9FA5A0D27427EA7075E780E9FA5EE',
  privOffset: undefined,
  privReason: 0,
  privErrorDetails: 'Unable to contact server. StatusCode: 1006, undefined Reason: ',
  privErrorCode: 4
}

I will try a VPN

[lcswillems]

I have tried with another internet network but I still have the same issue... How could I debug this? Microsoft doesn't get any of my requests?

[eldakms]

I do not see any of the session ids on the server side (D2D9FA5A0D27427EA7075E780E9FA5EE, 7E58E62A2C99403681671304D6E7A233)

[lcswillems]

Okay, thank you. What this error means? It means that Microsoft doesn't get requested? Is there a way to know the requests that are sent to Microsoft?

[eldakms]

JavaScript sdk is open source, you can have a look at the sources. could this one be related?
Azure-Samples/cognitive-services-speech-sdk#292

[lcswillems]

No, it doesn't help, I use the good key. And the exact same code was working yesterday.

[lcswillems]

I tried to see the request. Here is what happens:

I send a request to Microsoft servers and they send me back a "Origin Error", so Microsoft servers are reached and they answer something.

[rhurey]

There's an underlying problem connecting to the OCSP server to validate the service's certificate.

[lcswillems]

What does it mean? Does it mean the issue is on my side? How I can I debug this?

[rhurey]

It's looking like a problem with the certificate issuer, working on escalating to the right team in Azure that manages things.

Thanks for the report.

[lcswillems]

Okay, thank you for your support!

[rhurey]

PR #158 will improve the logging for connection errors if you're using node.

[gcampax]

Given that this is an issue on your end (indeed, http://ocsp.msocsp.com/ seems to timeout half the times I try it from curl), would it be possible to update https://status.azure.com/en-us/status to indicate disruption or degraded service? Thanks!

[lcswillems]

This morning, the issue disappeared, but it is back again now...

[lcswillems]

Could you, at least, update the status here: https://status.azure.com/en-us/status ?

This is really problematic that your service is not reliable at all. I have production servers that use your API...

[lcswillems]

I have tried my code on AWS machines. It doesn't work either... How is it possible? Are some people able to use your API?

[rhurey]

I'm sorry about the latency in updates.

What's happening is only affecting NodeJS clients.

When an SDK client connects to the Speech Service the certificate for the service in checked to ensure it hasn't been revoked. This is done in various ways based on the platform and SDK Language.

For the languages that are based on a native implementation (Basically everything but JavaScript) certificate validation is either offloaded to the platform OS (Schannel on Windows, MacOS and iOS have something I don't remember the name of) or handled by a library the SDK depends on. (OpenSSL on Linux based OS's). Most of these implementations use Certificate Revocation Lists to validate the certificate.

The JavaScript version of the SDK when run in a browser depends on the browser to conduct certificate validation.

In NodeJS the decision was made to use OCSP to validate certificates.

The service that hosts Microsoft's OCSP server is down, so when the NodeJS client tries to connect it cannot validate that the certificate hasn't been revoked and fails the connection.

We've been in contact with the team that owns OCSP, and they're engaged trying to solve the problem. They've attempted a couple mitigations and while things worked for a bit the situation isn't resolved.

[lcswillems]

Okay, thank you a lot for the update! It helps me to take decisions. Thank you :)

[gleitz]

I am told that upgrading to 1.10.1 will fix this issue. I have not yet verified but communication on this ticket seems slow from Microsoft so I put it out there.

[rhurey]

1.10.1 has fixes for apps build using react.js, but doesn't address the OCSP service issue.

As of this morning the OCSP server is responding, but slowly.

[gcampax]

Is there a reason to use OCSP at all? As you mentioned, all other platforms use platform-specific certificate verification mechanisms. Node.js is no exception: it uses OpenSSL and will check the certificates internally. If OpenSSL is normally good enough, why is it not good enough here?

[rhurey]

It's complicated.

NodeJS will check the certificates for validity, but not revocation. This thread has details:
nodejs/node#16338

OpenSSL requires you to get the CRL from the certificate issuer, it doesn't do it for you. In the native platform where OpenSSL is used the SDK hooks into OpenSSL and provides CRL's on demand for OpenSSL by downloading them.

NodeJS allows you to pass a CRL on connect that it will pass to OpenSSL for a revocation check, but that requires you to know what CRL matches the certificate you're going to connect to. As a client, the SDK doesn't know what certificate it's going to get, or what CRL is the correct one to download before it connects.

There was a library available that did OSCP checks that was easy to integrate into the project and acted as an httpAgent.

I'm looking into the complexity of writing an httpAgent that would do CRL checks, but it's not a quick fix.

[gcampax]

I understand, thank you for the explanation. I was not aware that nodejs has no certificate revocation check by default at all (this seems bad). Looking forward for a more permanent fix then.

[rhurey]

We don't currently document the Web socket protocol used.

And I suspect the code in this project could be better commented.

[amenella]

Hi again,

As said in the latter comment, we faced yesterday the current issue (once again):

The first error appeared at 1:55 pm CET and last error has been seen at 5:10 pm.

There has been a total of 221 occurences of the error, for 2165 sessions started on the same period, that means around 10% of failure.

Based on recommendations from Microsoft support, we are going to try with the latest commit of master branch for which they've added additionnal logging:

Are you willing to use either a private drop or try from our current GitHub master branch?

We've added more logging to enable us to find which error code is being returned from NodeJS.

The logging will be in our next release, but it isn't on npm yet.

Hope this helps

[fabiobiondi]

Hi,
I'm new on cognitive services and i'm trying most of them.
Today is the time to SPEECH API and I'm simply trying the hello world JS/PHP example.

Speech SDK version: SpeechSDK-JavaScript-1.12.0

I get this error:

SpeechSynthesisResult {privResultId: "EDE7CDC7ED0542F4BDB7124476E5CF9B", privReason: 1, privAudioData: undefined, privErrorDetails: "Unable to contact server. StatusCode: 1006, undefined Reason:  undefined", privProperties: PropertyCollection}
audioData: (...)
errorDetails: (...)
privAudioData: undefined
privErrorDetails: "Unable to contact server. StatusCode: 1006, undefined Reason:  undefined"
privProperties: PropertyCollection {privKeys: Array(1), privValues: Array(1)}
privReason: 1
privResultId: "EDE7CDC7ED0542F4BDB7124476E5CF9B"
properties: (...)
reason: (...)
resultId: (...)
__proto__: Object

Since I don't understand how to fix it (but I had no problems with other cognitive services) can you confirm me there is something to fix in SDK?
I ask you this just to avoid to waste time on it.

Thank you
Fabio

[gleitz]

@rhurey we're still seeing regular failures from the node sdk. Is Microsoft pulling support for this product?

[gleitz]

If you would like to use this service you need to bypass the failing OCSP check which seems to fail about 15% of the time. Please set the environment variable process.env.SPEECH_CONDUCT_OCSP_CHECK to 0.

Found at 3ad2b4b

[rhurey]

An update: The PR below has a caching change to enable the SDK to cache OCSP responses from the service. The default validity period for a response is 4 days, and the cache will begin to attempt to refresh when there's ~1 day of validity left.

Thought I'd share it now before we get it committed to garner more feedback if there was any.

#196

[jonnyparris]

@oscholz any update on this? the sdk seems completely unusable in its current state :(

[oscholz]

@rhurey is your PR part of 1.13?
@jonnyparris we are putting the finishing touches on 1.13, and will release it very soon.

[rhurey]

Yes, the OCSP Cache is in 1.13.

[daifuyang]

sorry!

[rhurey]

The region needed for the Speech SDK is the region the Azure Speech Resource was created in. It’ll be something like uswest or chinaeast.

[jonnyparris]

Still getting the same error despite upgrade to 1.13 ☹️, same as @daifuyang

I'm pretty sure my region is the same as where I created the resource; although please share any details if it's not as simple as matching with the resource information taken direct from my Azure portal resource page.

[rhurey]

Since browser based connections rely on the browser for certificate verification you're having a different root cause than a lot of the folks in this thread...

Looking at the error shown, "Unable to contact server" there should be a log line one above it (the one ending with X-ConnectionId) that has the URL the SDK is trying to connect to.

[jonnyparris]

this one?

[rhurey]

Can you get a connection to https://westeurope.stt.speech.microsoft.com/speech/recognition/interactive/cognitiveservices/v1/ with the browser? In FF for me that winds up with a blank page, but no error visible. (The network tab in the debugger should show a 401 error)

[jonnyparris]

Yep, I get the same as you: blank page, 401 error in the network tab

[rhurey]

That's puzzling, because at that point the SDK is depending on the browser to open the connection, and the browser is saying it can't. But you can get to the same URL yourself.

[Cincan]

hi,
my Speech SDK version is 1.13.1
I get this error:
SpeechSynthesisResult {
privResultId: '13F4195BBBC34EB0A4A73670E2F41EAA',
privReason: 1,
privAudioData: undefined,
privErrorDetails: 'Unable to contact server. StatusCode: 1006, undefined Reason: Failed to obtain OCSP response: 504',
privProperties: PropertyCollection {
privKeys: [ 'CancellationErrorCode' ],
privValues: [ 'ConnectionFailure' ]
}
}

[brandom-msft]

@Cincan closing this issue and I've copied your post to a new item here #291