Under high load cassandra faces connection issues with pycassa interface:
INFO:pycassa.pool:Connection 66232208 (:9160) in None (id = 51385744) failed: timed out
ERROR:root:Uncaught exception POST / (IP)HTTPRequest(protocol='http', host='IP:8888', method='POST', uri='/', version='HTTP/1.0', remote_ip='IP', remote_ip='IP', body='\x12\x0cdatastore_v3\x1a\x08RunQuery"@\n\x12appscalebenchmark4\x1a\x08TestItem#0\x03r\x0f\x1a\x05prop1 \x00_\x04\x08\xda\xe4)$\x80\x01d\xc8\x01\x01\xea\x01\x04100
K', headers={'X-Real-Ip': 'IP', 'Content-Length': '90', 'Accept-Encod
ing': 'identity', 'Appdata': 'appscalebenchmark4', 'Protocolbuffertype': 'Request', 'X-Forwarded-For': 'IP', 'Host': 'IP:8888'})
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/tornado/web.py", line 688, in _execute getattr(self, self.request.method.lower())(_args, *_kwargs)
File "/usr/lib/python2.6/site-packages/tornado/web.py", line 774, in wrapper
return method(self, *args, *_kwargs) File "/root/appscale/AppDB/datastore_server.py", line 1813, in post
self.remote_request(app_id, http_request_data)
File "/root/appscale/AppDB/datastore_server.py", line 1613, in remote_request http_request_data)
File "/root/appscale/AppDB/datastore_server.py", line 1710, in run_query
app_datastore._Dynamic_Run_Query(app_id, query, clone_qr_pb)
File "/root/appscale/AppDB/datastore_server.py", line 1551, in _Dynamic_Run_Quer
y
result = self.__GetQueryResults(query)
File "/root/appscale/AppDB/datastore_server.py", line 1539, in __GetQueryResults
results = strategy(self, query, filter_info, order_info)
File "/root/appscale/AppDB/datastore_server.py", line 1121, in __SinglePropertyQ
uery
startrow)
File "/root/appscale/AppDB/datastore_server.py", line 1252, in __ApplyFilters
end_inclusive=end_inclusive)
File "/root/appscale/AppDB/cassandra/cassandra_interface.py", line 224, in range
_query
for key in keyslices:
File "/usr/local/lib/python2.6/dist-packages/pycassa-1.3.0-py2.6.egg/pycassa/col
umnfamily.py", line 797, in get_range key_slices = self.pool.execute('get_range_slices', cp, sp, key_range, cl) File "/usr/local/lib/python2.6/dist-packages/pycassa-1.3.0-py2.6.egg/pycassa/pool.py", line 572, in execute return getattr(conn, f)(_args, *_kwargs) File "/usr/local/lib/python2.6/dist-packages/pycassa-1.3.0-py2.6.egg/pycassa/pool.py", line 145, in new_f
return new_f(self, _args, *_kwargs)
File "/usr/local/lib/python2.6/dist-packages/pycassa-1.3.0-py2.6.egg/pycassa/poo
l.py", line 145, in new_f
return new_f(self, _args, *_kwargs)
File "/usr/local/lib/python2.6/dist-packages/pycassa-1.3.0-py2.6.egg/pycassa/poo
l.py", line 145, in new_f
return new_f(self, _args, *_kwargs)
File "/usr/local/lib/python2.6/dist-packages/pycassa-1.3.0-py2.6.egg/pycassa/pool.py", line 145, in new_f
return new_f(self, _args, *_kwargs)
File "/usr/local/lib/python2.6/dist-packages/pycassa-1.3.0-py2.6.egg/pycassa/pool.py", line 145, in new_f
return new_f(self, _args, *_kwargs)
File "/usr/local/lib/python2.6/dist-packages/pycassa-1.3.0-py2.6.egg/pycassa/pool.py", line 140, in new_f
(self._retry_count, exc.class.name, exc))
MaximumRetryException: Retried 6 times. Last failure was timeout: timed out
ERROR:root:500 POST / (IP) 3649.25ms