TApplicationException: Missing result about thriftpy2

[ethe]

Generally, "Missing result" exception means you do not use the same thrift IDL in both server and client side, please check it. If you still got this exception, you can post a sample here, I will try to reproduce it.

[zyi1992]

The code is very simple function for return a string, but the problem was i can't reproduce it , In my project , this exception happened very rare.

[ethe]

I can not check or fix an irreproducible error.

[zyi1992]

Sorry for that, Hope anyone else bear this problem can provide some suggest.

[wooparadog]

@zyi1992 Can you tell us which client you are using? The client in thriftpy is quite low level, and doesn't handle many edge cases.

[zyi1992]

from thriftpy.rpc import make_client
client = make_client(service, host, port, timeout=timeout)

This was the client define code.

[wooparadog]

@zyi1992 , @ethe has came up with a possible explanation yesterday. The problem is that make_client in thriftpy doesn't handle timeout by itself, so weird things might happen if a timeout occurs.

Background: The client produced by make_client is basically based on a single connection, and won't handle any exceptions during communication(according to the original python-thrift).

What happens:

1. First rpc call made to server
2. Server hangs for more than timeout set by client.
3. Client raise socket.timeout error. but the connection is still open
4. Server finishes the first rpc response and send it to client. Data will be stored in client's buffer.
5. Client makes a second request, and try to get its response.
6. Client reads from buffer which contains response for the first one.
7. Raise weird protocol exceptions(including missing result)

The fix is to do proper connection management, you can write it according to your scenarios. We are using thrift_connector in that matter.

[zyi1992]

Thanks for you reply, I will have a try on thrift_connector. It seems like thrift_connector didn't deal with thriftpy server restart exception, When use thriftpy with gunicron_thrift, the server side restart will cause client side to raise a TTransportException excetion, is there a best practice for auto reconnection without restart exception.

[wooparadog]

@zyi1992 Actually, thrift_connector does handle this situation by preventing it rather than raising a TTransportException.

Before every rpc, thrift_connector will try to validate the connection by calling ping thrift method, then the actual rpc. If any error occurs during ping, it'll simply create a new connection for this rpc.

Code: https://github.com/Thriftpy/thrift_connector/blob/master/thrift_connector/connection_pool.py#L82

There are many practices out there:

1. Validating connection before every call. (Default in thrift_connector. In our server implementation, server will raise a ServerShutdownException in ping if server is gracefully shutting down.)
2. Maintaining a thread to make sure all connections in connection pool are valid. (Also implemented in thrift_connector, but for performance issues, we don't use it)
3. Use event polling to discover closed connections and process them.

The best method is 3, but we havn't got time to dig deeper than doing a POC.

[zyi1992]

Thanks for you explain.