fsx / momoko Goto Github PK
View Code? Open in Web Editor NEWWraps (asynchronous) Psycopg2 for Tornado.
Home Page: http://momoko.61924.nl/
License: Other
Wraps (asynchronous) Psycopg2 for Tornado.
Home Page: http://momoko.61924.nl/
License: Other
after updating to v1.0b, I have found there is no db.batch exists any more. would you please add this method back again?
The dynamic connection pool is causing some difficulties. See the last four comments in issue #34 for information.
Currently connections are considered "busy" if they are executing a query or are in a transaction. They stay in the same list at the connections that are not busy. The connection pool is cleaned every N seconds. And new connections are created when needed.
The problem is that connections are only "busy" when executing queries and not when results are being fetched from the cursor. When the connection is closed at that time the cursor throws an error. To prevent this the connection needs to know if its cursors are still being used. Psycopg2 does a get and put action on a connection pool. I don't see that really working with Tornado. And I couldn't find a way to see if a cursor still needs to connection.
I would like to remove the dynamic connection pool and replace it with a static one. This simplifies things a bit and there's no need for a pool cleaner that closes a connection when it's still needed. A simple queue can be used to queue operations when all the connections are in use. Like Tornado's HTTP client does.
Why would I write this here? I you to know what I'm doing and I want some feedback if possible.
Psycopg2 interprets a single connection to be a transaction - see http://initd.org/psycopg/docs/usage.html#transactions-control
One nice way to solve this would be the QueryChain to reuse a single connection. This way, it's at least possible to do transactions.
A further extension would be a way to insert commands rather than just queries.
We had a staging issue where the IP address for our node was not in the databases OK list. momoko made the connection, but then ofc it was refused. You can replicate exactly what we got if you try to connect with a bad user name:
See traceback:
https://gist.github.com/kacieh80/644445ef51162e7faada
Please don't judge me, I did not name our service.
We are on 1.0 still. We tried upgrading to the latest version and we got the same error.
So what happens when it connects, but then is kicked out is pools == []. Our hacky solution is to throw out the pools - throw a pretty error in the logs about issues connecting to the database - and try to connect again when the user requests a query. This should really be fixed though. Even though we catch the error thrown by the stacktrace, we then get an OSError file not found here: https://github.com/FSX/momoko/blob/v1.0.0/momoko/connection.py#L257
Because the code isn't aware of being kicked out yet? That's for you guys to figure out.
We did not test the second OS error on line 257 in the new version and I can't give you a traceback as I was troubleshooting on another devs box and this service is in 2.7 and my box only does 3 because pls pls move to 3 everyone! Also we got our IP address in so everything is working now. It would be nice to have this error more gracefully though.
Thanks.
I am new to Python and Tornado and I am not sure if this is a bug or I am doing some mistake. I have source code where I send SQL to PostgreSQL database and get back an exception, I catch it and send to the browser appropriate json - everything is quite OK. If I wait 10 seconds until connection cleanup, I get a new connection from pool and everything is OK again. But if I wait only 2 seconds I get unexpected error "Cannot send error response after headers written". Following is minimal source code that is able to reproduce the error:
#!/usr/bin/env python
import sys
import traceback
import tornado.httpserver
import tornado.ioloop
import tornado.options
import tornado.web
from tornado import gen
import psycopg2
import psycopg2.extras
import momoko
class TestHandler(tornado.web.RequestHandler):
@tornado.web.asynchronous
@gen.engine
def get(self):
try:
cursor = yield gen.Task(self.application.db["nameservice"].execute, "select 1/0", {})
except Exception as err:
self.write({"err": "message"})
self.finish()
def main():
try:
tornado.options.parse_command_line()
routes = [
(r'/test', TestHandler),
]
application = tornado.web.Application(
routes,
debug=True,
cookie_secret='bla bla'
)
application.db = {
"nameservice": momoko.AsyncClient({
'host': "localhost",
'database': "nameservice",
'user': "miiza",
'password': "aaaaaaaaaaa",
'min_conn': 0,
'max_conn': 25,
'cleanup_timeout': 8,
'connection_factory': psycopg2.extras.NamedTupleConnection
})
}
http_server = tornado.httpserver.HTTPServer(application)
http_server.listen(8888)
print "Tornado web server now runs."
tornado.ioloop.IOLoop.instance().start()
except KeyboardInterrupt:
print('Exit')
if __name__ == '__main__':
main()
After first run everything is quite OK, exception was cought by my exception handler and the request is finished:
[I 130303 23:07:47 web:1462] 304 GET /test (127.0.0.1) 33.01ms
But when I send from my browser the same request after 2 seconds I get this error. It seems like if the exception hangs in the connection:
[E 130303 23:07:48 web:1085] Uncaught exception GET /test (127.0.0.1)
HTTPRequest(protocol='http', host='localhost:8888', method='GET', uri='/test', version='HTTP/1.1', remote_ip='127.0.0.1', body='', headers={'Accept-Language': 'cs,en-us;q=0.7,en;q=0.3', 'Accept-Encoding': 'gzip, deflate', 'Host': 'localhost:8888', 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:18.0) Gecko/20100101 Firefox/18.0', 'Connection': 'keep-alive', 'Cache-Control': 'max-age=0', 'If-None-Match': '"62e84d336c5973652977bb0379ea846f163bcfaf"'})
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/tornado/web.py", line 1021, in _stack_context_handle_exception
raise_exc_info((type, value, traceback))
File "/usr/local/lib/python2.7/dist-packages/tornado/stack_context.py", line 232, in wrapped
callback(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/momoko/pools.py", line 313, in _io_callback
state = self._conn.poll()
DataError: division by zero
[E 130303 23:07:48 web:739] Cannot send error response after headers written
So, it is strange. I think my exception handler should catch the database exception anytime I send request from my browser. But the reality is that in case of db connection reuse it is not true. It seems like if connection which an exception went through becomes ill and breaks it completely.
When trying to initialize the a momoko pool while the database is down, an exception will be raised, but it is uncatchable by the caller.
After the call the caller does not know whether or not the pool was successfully initialized. Calling for example execute on the failed pool will hang trying to queue the command, spamming the log with this:
WARNING:momoko:Execute: no connection available, operation queued.
I put together simple handler to test momoko and hit it with ApacheBench, and after a certain level of load, it throws an OperationalError in the Poller object and puts the connection in a bad state. I can't make any more queries with the connection, and I have no way to catch the exception to re-establish the connection. Any suggestions?
[E 110526 20:57:16 web:900] Uncaught exception GET /test (127.0.0.1)
HTTPRequest(protocol='http', host='127.0.0.1:10000', method='GET', uri='/test', version='HTTP/1.0', remote_ip='127.0.0.1', body='', headers={'Host': '127.0.0.1:10000', 'Accept': '*/*', 'User-Agent': 'ApacheBench/2.3'})
Traceback (most recent call last):
File "/Users/carlo/Projects/python/lib/python2.6/site-packages/tornado-1.2.1-py2.6.egg/tornado/stack_context.py", line 171, in wrapped
callback(*args, **kwargs)
File "/Users/carlo/Projects/python/lib/python2.6/site-packages/momoko/momoko.py", line 394, in _io_callback
self._update_handler()
File "/Users/carlo/Projects/python/lib/python2.6/site-packages/momoko/momoko.py", line 383, in _update_handler
state = self._connection.poll()
OperationalError: asynchronous connection failed
This is the script:
import logging
# async postgres client
from momoko import Momoko
from tornado.httpserver import HTTPServer
from tornado.ioloop import IOLoop
from tornado.options import define, parse_command_line, options
from tornado.web import Application, RequestHandler, asynchronous
class MainHandler(RequestHandler):
def initialize(self):
self._db = None
@property
def db(self):
if not self._db:
# Setup the db connection when first accessed instead of at
# initialization so that each child process gets its own
# connection.
self._db = Momoko({
'host': 'localhost',
'database': 'db',
'user': 'user',
'password': 'password',
'min_conn': 5,
'max_conn': 20,
'cleanup_timeout': 10
})
return self._db
@asynchronous
def get(self):
self.db.execute("""
select * from test
order by random()
limit 1
""", callback=self._on_response)
def _on_response(self, cursor):
columns = [f[0] for f in cursor.description]
row = cursor.fetchone()
self.write(dict(zip(columns, row)))
self.finish()
def start(port, reload_on_change=False, num_processes=None):
logging.basicConfig(format="[%(asctime)s] %(levelname)s: %(message)s")
settings = {
'debug': reload_on_change
}
app = Application([
(r"/test", MainHandler),
], **settings)
logging.info('Starting test on port {0}'.format(port))
server = HTTPServer(app)
server.bind(port)
server.start(num_processes)
IOLoop.instance().start()
def main():
define("port", type=int, default=10000, help="Port to listen on")
define("reload", type=bool, default=False, help="Automatically restart the server when code changes are detected")
define("num_processes", type=int, default=None, help="Number of child processes to prefork. Defaults to the number of cpus on the machine.")
parse_command_line()
start(
port=options.port,
reload_on_change=options.reload,
num_processes=options.num_processes
)
if __name__ == '__main__':
main()
It important to return connection back to the pool once you’ve done with it,
even if an error occurs in the middle of your work.
Use either putconn() method or manage() manager to return the connection.
so. i tried this example:
@gen.coroutine
def get(self):
chunk = 1000
try:
connection = yield momoko.Op(self.db.getconn)
with self.db.manage(connection):
yield momoko.Op(connection.execute, "BEGIN")
...
except Exception as error:
self.write(str(error))
finally:
yield momoko.Op(self.db.putconn, connection)
but error occured:
TypeError: putconn() got an unexpected keyword argument 'callback'
and i found solution, add param callback=None
to putconn
function at connection.py
line 444:
def putconn(self, connection, callback=None):
and it should be fixed, but i've not test this
Greetings, I've read through the documents and stumbled upon this note in psycopg2 faqs, namely this one:
Why does psycopg2 leave database sessions “idle in transaction”?
Psycopg normally starts a new transaction the first time a query is executed, e.g. calling cursor.execute(), even if the command is a SELECT. The transaction is not closed until an explicit commit() or rollback().If you are writing a long-living program, you should probably make sure to call one of the transaction closing methods before leaving the connection unused for a long time (which may also be a few seconds, depending on the concurrency level in your database). Alternatively you can use a connection in autocommit mode to avoid a new transaction to be started at the first command.
So when I use db.execute
to run a INSERT
or UPDATE
statement, should I commit it immediately in another db.execute
call? Or should I just ignore it?
I found this isn't mentioned in the docs. So I'd ask it here. Thanks in advance.
When the application is started and the first query is going to be executed the connection pool is also created. It looks like there is not yet a connection available and a new one is created.
psychopg2
connects synchronously to local socket. When Pool
is initialized and database is down, the pool fails to start because psycopg2
raises connect error immediately.
The solution would be probably to catch connect errors early and immediately mark connection as dead.
I tried to install new momoko in heroku, look the error:
remote: -----> Python app detected
remote: -----> Using Python runtime (python-3.3.1)
remote: -----> Installing dependencies using Pip (1.3.1)
remote: Downloading/unpacking Momoko==1.0.0b2 (from -r requirements.txt (line 1))
remote: Could not find a version that satisfies the requirement Momoko==1.0.0b2 ( (from -r requirements.txt (line 1)) (from versions: 0.4.0, 0.3.0, 0.2.0, 1.0.0, 0.1.0, 0.5.0, 0.1.0, 0.5.0, 0.4.0, 0.2.0, 0.3.0, 1.0.0, 0.2.0, 0.1.0, 0.4.0, 0.3.0, 0.5.0, 0.1.0, 0.5.0, 0.4.0, 0.2.0, 0.3.0, 1.0.0, 0.1.0, 0.5.0, 0.4.0, 0.2.0, 0.3.0, 1.0.0, 0.4.0, 0.3.0, 0.2.0, 1.0.0, 0.1.0, 0.5.0, 0.2.0, 0.4.0, 0.3.0, 0.5.0, 0.1.0, 1.0.0)
To fix it, I needed to change the "pip freeze" file to momoko == 1.0.0 instead momoko == 1.0.0b2
Tests covering all functionality and fixed bugs.
Tornado provides similar functionality now with gen
.
Running
yield momoko.Op(pool.transaction([u"SELECT 1"])
Throws:
ProgrammingError: syntax error at or near "S"
LINE 1: S
(mypy)hzg@debian:~$ pip install momoko
Downloading/unpacking momoko
Downloading Momoko-0.3.0.tar.gz (109Kb): 109Kb downloaded
Running setup.py egg_info for package momoko
Installing collected packages: momoko
Running setup.py install for momoko
File "/home/hzg/mypy/lib/python2.5/site-packages/momoko/client.py", line 365
self._db.execute(*query, callback=self._collect)
^
SyntaxError: invalid syntax
Successfully installed momoko
I'm using AsyncHTTPTestCase
and some other stuff so that I can do integration style tests by GET fetching URLs on the application in my tests. The code looks something like this:
from tornado.testing import AsyncHTTPTestCase
class BaseHTTPTestCase(AsyncHTTPTestCase, HTTPClientMixin):
def setUp(self):
super(BaseHTTPTestCase, self).setUp()
self.client = TestClient(self) # from tornado-utils
def get_app(self):
# pick up the defaults
config = TEST_DATABASE_CONFIG
return app.Application(config)
class CrashesCommentsTestCase(BaseHTTPTestCase):
def test_get_no_data(self):
url = self.reverse_url('crashes:comments', 'signature/FooBar')
response = self.client.get(url)
The app works and its inner core looks something like this:
class BaseHandler(tornado.web.RequestHandler):
@property
def db(self):
# Create a database connection when a request handler is called
# and store the connection in the application object.
if not hasattr(self.application, 'db'):
config = self.application.settings
self.application.db = momoko.AsyncClient({
'host': config['database_host'],
'database': config['database_name'],
'user': config['database_user'],
'password': config['database_password'],
'min_conn': config['database_min_connections'],
'max_conn': config['database_max_connections'],
'cleanup_timeout': config['database_cleanup_timeout']
})
return self.application.db
@route('/crashes/comments/(.*)', name='crashes:comments')
class CrashesCommentsHandler(BaseHandler):
@tornado.web.asynchronous
@tornado.gen.engine
def get(self, parameters):
print "pre-cursor"
cursor = yield tornado.gen.Task(
self.db.execute,
...,
...
)
print "cursor", repr(cursor)
total, = cursor.fetchone()
print "total", repr(total)
...
self.write(stuff)
self.finish()
Whenever I try to run the tests I get:
pre-cursor
F
======================================================================
FAIL: test_get_no_data (tests.test_handlers.CrashesCommentsTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/peterbe/dev/MOZILLA/SOCORRO/socorro/socorro/middleware2/tests/test_handlers.py", line 8, in test_get_no_data
response = self.client.get(url)
File "/Users/peterbe/virtualenvs/socorro/lib/python2.6/site-packages/tornado_utils/http_test_client.py", line 58, in get
follow_redirects=follow_redirects)
File "/Users/peterbe/virtualenvs/socorro/lib/python2.6/site-packages/tornado_utils/http_test_client.py", line 22, in get
follow_redirects=follow_redirects)
File "/Users/peterbe/virtualenvs/socorro/lib/python2.6/site-packages/tornado_utils/http_test_client.py", line 39, in _fetch
return self.wait()
File "/Users/peterbe/virtualenvs/socorro/lib/python2.6/site-packages/tornado/testing.py", line 202, in wait
self.__rethrow()
File "/Users/peterbe/virtualenvs/socorro/lib/python2.6/site-packages/tornado/testing.py", line 146, in __rethrow
raise_exc_info(failure)
File "/Users/peterbe/virtualenvs/socorro/lib/python2.6/site-packages/tornado/testing.py", line 183, in timeout_func
timeout)
AssertionError: Async operation timed out after 5 seconds
So, is there something I need know about running tests "on top of" momoko?
Any other hints?
This is actually 2 issues.
1:
The first is that errors cannot be caught, for example, if I provide the wrong database login and start an AsyncPool, the momoko event handler is just called repeatedly and the IOLoop keeps printing the stack trace "Asynchronous connection failed". A mechanism is required to handle errors gracefully, remove the bad connections from the IOLoop and notify the caller that something went wrong.
2:
The request StackContext is not maintained. Assuming that the AsyncPool can always handle all it's exceptions, this is still a problem, for example:
@gen.engine
def get(self):
result = yeild gen.Task(pool.new_cursor, 'execute', "select 1")
# Don't like it
raise web.HTTPError(404)
The 404 is logged but the client is left hanging, since the StackContext was lost. The fix is wrapping user callbacks with tornado.stackcontext.wrap().
Regards
ERROR:root:Cannot send error response after headers written
ERROR:root:Uncaught exception POST /url/is/here/ (127.0.0.1)
HTTPRequest(protocol='http', host='localhost:8888', method='POST', uri='/url/is/here/', version='HTTP/1.1', remote_ip='127.0.0.1', body='some_data_here', headers={'Content-Length': '96', 'Connection': 'close', 'Content-Type': 'application/x-www-form-urlencoded', 'Host': 'localhost:8888', 'Accept-Encoding': 'gzip'})
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/tornado/web.py", line 1021, in _stack_context_handle_exception
raise_exc_info((type, value, traceback))
File "/usr/lib/python2.7/site-packages/tornado/stack_context.py", line 259, in _nested
yield vars
File "/usr/lib/python2.7/site-packages/tornado/stack_context.py", line 229, in wrapped
callback(_args, *_kwargs)
File "/usr/lib/python2.7/site-packages/momoko/pools.py", line 313, in _io_callback
state = self._conn.poll()
OperationalError: asynchronous connection failed
more details here http://stackoverflow.com/questions/14494885/asynchronous-connection-failed-by-momoko
Make some documentation with Sphinx and put it on Github pages and the PyPi page.
Greetings. As the title suggests, getconn()
doesn't play well with NamedTupleCursor
.
Here's a test case:
import psycopg2
import momoko
from psycopg2.extras import NamedTupleCursor
from tornado.testing import AsyncTestCase, gen_test
import unittest
import os
class ConnNamedTuple(AsyncTestCase):
def setUp(self):
super().setUp()
self.db = momoko.Pool(
dsn='dbname=dbdb user=%s host=localhost port=5432' % os.environ['USER'],
size=3,
ioloop=self.io_loop,
callback=self.stop,
cursor_factory=NamedTupleCursor
)
self.wait()
def tearDown(self):
self.db.close()
self.io_loop.close()
@gen_test
def test_conn_namedtuple(self):
yield momoko.Op(self.db.getconn)
unittest.main()
Here's the stack trace:
======================================================================
ERROR: test_conn_namedtuple (__main__.ConnNamedTuple)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/lib/python3.4/dist-packages/tornado/testing.py", line 427, in wrapper
functools.partial(f, self), timeout=timeout)
File "/usr/local/lib/python3.4/dist-packages/tornado/ioloop.py", line 389, in run_sync
return future_cell[0].result()
File "/usr/local/lib/python3.4/dist-packages/tornado/concurrent.py", line 129, in result
raise_exc_info(self.__exc_info)
File "<string>", line 3, in raise_exc_info
File "/usr/local/lib/python3.4/dist-packages/tornado/gen.py", line 206, in handle_exception
if runner is not None and runner.handle_exception(typ, value, tb):
File "/usr/local/lib/python3.4/dist-packages/tornado/gen.py", line 580, in handle_exception
self.run()
File "/usr/local/lib/python3.4/dist-packages/tornado/gen.py", line 529, in run
yielded = self.gen.throw(*exc_info)
File "tmp/conn_namedtuple.py", line 26, in test_conn_namedtuple
yield momoko.Op(self.db.getconn)
File "/usr/local/lib/python3.4/dist-packages/tornado/stack_context.py", line 302, in wrapped
ret = fn(*args, **kwargs)
File "/usr/local/lib/python3.4/dist-packages/momoko/connection.py", line 594, in io_callback
self.callback(None)
File "/usr/local/lib/python3.4/dist-packages/momoko/connection.py", line 640, in _ping_callback
cursor.fetchall()
File "/usr/local/lib/python3.4/dist-packages/psycopg2/extras.py", line 317, in fetchall
nt = self.Record = self._make_nt()
File "/usr/local/lib/python3.4/dist-packages/psycopg2/extras.py", line 340, in _make_nt
return namedtuple("Record", [d[0] for d in self.description or ()])
File "/usr/lib/python3.4/collections/__init__.py", line 338, in namedtuple
'identifiers: %r' % name)
ValueError: Type names and field names must be valid identifiers: '?column?'
----------------------------------------------------------------------
Ran 1 test in 0.031s
FAILED (errors=1)
I'm on Python 3.4. pip freeze output, all are PyPI releases.
Momoko==1.1.2
psycopg2==2.5.2
tornado==3.2
I think it's easy to workaround but it's a little bit annoying. Thanks in advance.
Using python 2.7, tornado 3.2 momoko 1.0.0 psycopg2 2.5, with Op, WaitOp, and WaitAllOp examples in tutorial, Response fails with:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tornado/web.py", line 1232, in _when_complete
raise ValueError("Expected Future or None, got %r" % result)
ValueError: Expected Future or None, got <generator object get at 0x1027b8fa0>
Bug? Suggested workaround?
Hi, all
I am trying to perform some reconnect-procedures of my program.
Assume the DB is down at startup, I will try to reconnect the db when I found the db is down which means the _pool size is 0 at the execution point.
def init_setting(self, db_host, db_port, db_name, user_name, user_pwd, pool_size):
""" Initialize the DB connect setting
"""
self.cache_dsn = 'dbname=%s user=%s password=%s host=%s port=%s' % (db_name, user_name, user_pwd, db_host, db_port)
self.cache_size = pool_size
self.db = momoko.Pool(dsn=self.cache_dsn, size=self.cache_size)
@gen.coroutine
def execute(self, *commands):
""" execute commands
Args:
commands: {tuple of SQL commands}
"""
error_occur = False
while len(self.db._pool) == 0 or error_occur:
logger.debug("Connection Pool is empty!! Re-connect the db. cmd=%s", commands)
yield gen.Task(IOLoop.instance().add_timeout, time.time() + 2)
try:
self.db = momoko.Pool(dsn=self.cache_dsn, size=self.cache_size)
except:
error_occur = True
logger.debug("Reconnect the DB Failed!!")
yield gen.Task(IOLoop.instance().add_timeout, time.time() + 1)
# ...... skip the rest
result = yield momoko.WaitAllOps(tuple(op_name_list))
raise gen.Return(result[-1])
When I run the execute function above, when the pool size is 0, I will try to re-initialize the momoko.Pool and I always get the following exception.
Traceback (most recent call last):
File "/lib/python2.7/site-packages/tornado/web.py", line 1115, in _stack_context_handle_exception
raise_exc_info((type, value, traceback))
File "/lib/python2.7/site-packages/tornado/stack_context.py", line 343, in _handle_exception
if tail.exit(*exc):
File "/lib/python2.7/site-packages/tornado/stack_context.py", line 186, in exit
return self.exception_handler(type, value, traceback)
File "lib/python2.7/site-packages/tornado/gen.py", line 204, in handle_exception
future.set_exc_info((typ, value, tb))
File "/lib/python2.7/site-packages/tornado/concurrent.py", line 125, in set_exc_info
self.set_exception(exc_info[1])
File "/lib/python2.7/site-packages/tornado/concurrent.py", line 87, in set_exception
self._set_done()
File "/lib/python2.7/site-packages/tornado/concurrent.py", line 95, in _set_done
for cb in self._callbacks:
TypeError: 'NoneType' object is not iterable
Assume the DB is down during the above test.
I found out the problem is the connection recreation: "momoko.Pool" inside the while loop of execute function. When I comment the out, the exception is gone and also cannot reconnect .
And if the DB is down at startup and I restart the db before the execute function, the above connection recreation procedure works fine.
Does anyone has any suggestions?
Ideas?
Cheers
Ivan
For example I have code like this and only one connection in pool:
try:
cursor = yield momoko.Op(self.db.execute, 'SELECT %s FROM %s'
,()) # rises TypeError
except Exception as e:
print e
cursor = yield momoko.Op(self.db.execute, 'SELECT 1'
,())
print 'Done' #unreachable
This code rises TypeError, but second query can't obtain connection.
Debug log is:
[D 141115 18:00:26 connection:234] Connection attempt complete. Success: True
[D 141115 18:00:31 connection:252] Getting connection
[D 141115 18:00:31 connection:327] Connection obtained, proceeding
tuple index out of range
[D 141115 18:00:31 connection:252] Getting connection
[D 141115 18:00:31 connection:272] There are busy connections
[D 141115 18:00:31 connection:321] No connection available right now - will try again later
Changing psycopg2.Error to Exception in _catch_early_errors decorator fixes that issue, but I not sure that it right.
Chain multiple queries and execute them one by one, passing the results to the next one and/or execute a callback for a certain query when specified.
It's too much work to manually make callbacks in a request handler for queries.
Hi. I use tornado + momoko + postgresql 8.4 and all works fine. Then, I drop postgresql 8.4 and install 9.1 and momoko cannot connect to postgresql:
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/tornado/stack_context.py", line 202, in wrapped
callback(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/momoko/utils.py", line 140, in _io_callback
self._update_handler()
File "/usr/local/lib/python2.7/dist-packages/momoko/utils.py", line 127, in _update_handler
state = self._connection.poll()
OperationalError: asynchronous connection failed
BaseHandler:
class BaseHandler(tornado.web.RequestHandler):
@property
def db(self):
if not hasattr(self.application, 'db'):
self.application.db = momoko.AdispClient({
'host': settings.host,
'port': settings.port,
'database': settings.database,
'user': settings.user,
'password': settings.password,
'min_conn': settings.min_conn,
'max_conn': settings.max_conn,
'cleanup_timeout': settings.cleanup_timeout
})
return self.application.db
Query:
@tornado.web.asynchronous
@momoko.process
def get(self):
cursor = yield self.db.execute('select 1,2,3');
print cursor.fetchall()
What I`m doing wrong ?
I was evaluating momoko for a project and noticed it was throwing errors if i instantiate the connection pool, trying to track this down i took the example from the momoko manual and that is also throwing the error,
this is running on OSX with the home brew version of python 2.7.3
$ cat test.py
from tornado import gen
from tornado.ioloop import IOLoop
from tornado.httpserver import HTTPServer
from tornado.options import parse_command_line
from tornado.web import *
import psycopg2
import momoko
class BaseHandler(RequestHandler):
@property
def db(self):
return self.application.db
class TutorialHandler(BaseHandler):
def get(self):
self.write('Some text here!')
self.finish()
if __name__ == '__main__':
parse_command_line()
application = Application([
(r'/', TutorialHandler)
], debug=True)
application.db = momoko.Pool(
dsn='dbname=your_db user=your_user password=very_secret_password '
'host=localhost port=5432',
size=1
)
http_server = HTTPServer(application)
http_server.listen(8888, 'localhost')
IOLoop.instance().start()
$ python -m test
[I 141119 14:42:04 connection:536] Opening new database connection
[E 141119 14:42:04 ioloop:585] Exception in callback (8, <function null_wrapper at 0x104b337d0>)
Traceback (most recent call last):
File "/Users/daniel/.venvs/foo/lib/python2.7/site-packages/tornado/ioloop.py", line 837, in start
handler_func(fd_obj, events)
File "/Users/daniel/.venvs/foo/lib/python2.7/site-packages/tornado/stack_context.py", line 275, in null_wrapper
return fn(*args, **kwargs)
File "/Users/daniel/.venvs/foo/lib/python2.7/site-packages/momoko/connection.py", line 599, in io_callback
self.ioloop.update_handler(self.fileno, IOLoop.WRITE)
File "/Users/daniel/.venvs/foo/lib/python2.7/site-packages/tornado/ioloop.py", line 681, in update_handler
self._impl.modify(fd, events | self.ERROR)
File "/Users/daniel/.venvs/foo/lib/python2.7/site-packages/tornado/platform/kqueue.py", line 45, in modify
self.unregister(fd)
File "/Users/daniel/.venvs/foo/lib/python2.7/site-packages/tornado/platform/kqueue.py", line 50, in unregister
self._control(fd, events, select.KQ_EV_DELETE)
File "/Users/daniel/.venvs/foo/lib/python2.7/site-packages/tornado/platform/kqueue.py", line 64, in _control
self._kqueue.control([kevent], 0)
OSError: [Errno 2] No such file or directory
^CTraceback (most recent call last):
File "/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 162, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/Users/daniel/foo/test.py", line 37, in <module>
IOLoop.instance().start()
File "/Users/daniel/.venvs/foo/lib/python2.7/site-packages/tornado/ioloop.py", line 812, in start
event_pairs = self._impl.poll(poll_timeout)
File "/Users/daniel/.venvs/foo/lib/python2.7/site-packages/tornado/platform/kqueue.py", line 67, in poll
kevents = self._kqueue.control(None, 1000, timeout)
KeyboardInterrupt
$ python --version
Python 2.7.3
$ pip freeze
Momoko==1.1.5
backports.ssl-match-hostname==3.4.0.2
brukva==0.0.1
certifi==14.05.14
geoip2==2.0.2
psycopg2==2.5.4
requests==2.4.3
tornado==4.0.2
wsgiref==0.1.2
I keep seeing this in my logs:
Traceback (most recent call last):
File "/usr/local/lib/python3.2/dist-packages/tornado/web.py", line 1115, in _stack_context_handle_exception
raise_exc_info((type, value, traceback))
File "", line 3, in raise_exc_info
File "/usr/local/lib/python3.2/dist-packages/tornado/stack_context.py", line 302, in wrapped
ret = fn(*args, **kwargs)
File "/usr/local/lib/python3.2/dist-packages/momoko/connection.py", line 414, in exec_statement
self.execute(operation, parameters, cursor_factory, exec_statement)
File "/usr/local/lib/python3.2/dist-packages/momoko/connection.py", line 289, in execute
cursor.execute(operation, parameters)
psycopg2.ProgrammingError: execute cannot be used while an asynchronous query is underway
It looks to me like Connection.busy() should maybe return true while the Connection is in the middle of a transaction(). Otherwise perhaps you can get into the situation where two asynchronous web requests can end up trying to use the same underlying psycopg connection?
I have a max_conn
of 1000 (just for argument's sake) and I run out of cursors. This is the big batch that I run:
cursors = yield gen.Task(self.db.batch, queries)
for key,cursor in cursors.items():
worddata = {}
if verse[0] not in results.keys():
results[verse[0]] = []
datapoints = cursor.fetchone()
results[verse[0]].append( {
'wordid': datapoints[0],
'word': datapoints[1],
'inflectedmeaning': datapoints[2],
'root': datapoints[3],
'rootmeaning': datapoints[4],
'case': datapoints[5],
'person': datapoints[6],
'gender': datapoints[7],
'number': datapoints[8],
'tense': datapoints[9],
'voice': datapoints[10],
'mood': datapoints[11],
} )
I use this to render a page and each word has a link. At this point, I'm getting regular errors to the effect of PoolError: connection pool exausted
or InterfaceError: cursor already closed
. I'm pretty sure that I'm doing something wrong but am unsure as to what it is.
def get(self, word):
queries = {}
queries['worddetails'] = ['select * from worddetailsbyword(%s)', (word,)]
queries['wordlocations'] = ['select v.id, reference, value from verse v join version vv on v.version_id = vv.id where to_tsvector(analysis_text) @@ to_tsquery(%s)', (word,)]
cursors = yield gen.Task(self.db.batch, queries)
for key, cursor in cursors.items():
self.write('Query results: %s = %s<br>' % (key, cursor.fetchall()))
self.finish()
After 2 days of clean work, I am getting these rows in my log file:
WARNING:momoko:Execute: no connection available, operation queued.
Thousands of them actually, which eventually kills my application and i get 500/502 responses or no responses at all from server. Then, after restart application runs normally. I wonder if this is momoko bug or problem with postgresql or my code.
I use momoko in rather an odd way:
class MomokoWrapper(object):
def __init__(self):
pass
def pool(self):
self.db = momoko.Pool(
dsn='dbname=%s user=%s password=%s host=%s port=%s' % (
database['name'], database['user'], database['password'],
database['host'], database['port']
),
size=1
)
return self
def sample_fun(self, params):
return momoko.Op(self.db.execute, 'my query')
Then in tornado Application init
self.db = MomokoWrapper().pool()
And I am able to execute wrappers functions like this:
yield self.db.sample_fun(params)
Actually, I doubt that code organization and momoko usage is the problem (despite the fact that this may be good/wrong layout). I presume that postgresql or pool configuration might be an issue in my case. I am dealing with these annoying warnings/errors for 2 weeks now and have absolutely no ideas left. All suggestions are appreciated.
Catching exceptions is not possibly (yet), because the exception if raised in the Poller class and not in the request handler where the query is executed. tornado.stack_context
should be used to fix this.
Here is a sample
import tornado.gen
import tornado.ioloop
import momoko
from psycopg2.extras import RealDictConnection, RealDictCursor
@tornado.gen.coroutine
def query():
db = momoko.Pool('dbname=jvhua user=jvhua', connection_factory=RealDictConnection)
cur = yield momoko.Op(db.execute, 'SELECT * FROM users;')
print(cur.fetchone())
tornado.ioloop.IOLoop.instance().run_sync(query)
tornado.ioloop.IOLoop.instance().start()
What it prints is a tuple, while expected a dict.If I add cursor_factory=RealDictCursor to momoko.Op, things work fine.
Psycopg should be referencing http://initd.org/psycopg/
, without the www
. www.initd.org
is not working.
For my unit tests I have another db
that I use to stuff in fixture data. Unlike the app, I want this one to be blocking since I don't want additional hassle as I'm setting up the test. Here's what I tried:
#tests/base.py
@property
def db(self):
# Create a database connection when a request handler is called
# and store the connection in the application object.
if not hasattr(self._app, 'db'):
from socorro.unittest.config.commonconfig import (
databaseHost, databaseName, databaseUserName, databasePassword)
self._app.db = momoko.BlockingClient({
'host': databaseHost.default,
'database': databaseName.default,
'user': databaseUserName.default,
'password': databasePassword.default,
'min_conn': 1,
'max_conn': 20,
'cleanup_timeout': 10
})
return self._app.db
#sometest.py
def setUp(self):
...
with self.db.connection() as conn:
cursor = conn.cursor()
cursor.execute("""
INSERT INTO users ....
But unfortunately I'm getting this:
======================================================================
ERROR: test_get_no_data (tests.test_handlers.CrashesCommentsTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/peterb.../middleware2/tests/test_handlers.py", line 20, in setUp
with self.db.connection() as conn:
TypeError: 'GeneratorContextManager' object is not callable
How save binary file in a database(postgresql) with momoko ?
Just thought I'll play with latest tornado. Seem that some of the upstream tornado changes (gen.Task is now a function not a class) has broken Momoko. Hope this bug report can help with the fix.
(ve3_t4)➜ momoko git:(master) python setup.py test
Using psycopg2
running test
running egg_info
creating Momoko.egg-info
writing dependency_links to Momoko.egg-info/dependency_links.txt
writing top-level names to Momoko.egg-info/top_level.txt
writing requirements to Momoko.egg-info/requires.txt
writing Momoko.egg-info/PKG-INFO
writing manifest file 'Momoko.egg-info/SOURCES.txt'
reading manifest file 'Momoko.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'Momoko.egg-info/SOURCES.txt'
running build_ext
Traceback (most recent call last):
File "setup.py", line 58, in
'Topic :: Database :: Front-Ends'
File "/usr/local/Cellar/python3/3.4.0_1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/distutils/core.py", line 149, in setup
dist.run_commands()
File "/usr/local/Cellar/python3/3.4.0_1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/distutils/dist.py", line 955, in run_commands
self.run_command(cmd)
File "/usr/local/Cellar/python3/3.4.0_1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/distutils/dist.py", line 974, in run_command
cmd_obj.run()
File "/Users/martinslabber/Documents/coding/momoko/ve3_t4/lib/python3.4/site-packages/setuptools/command/test.py", line 138, in run
self.with_project_on_sys_path(self.run_tests)
File "/Users/martinslabber/Documents/coding/momoko/ve3_t4/lib/python3.4/site-packages/setuptools/command/test.py", line 118, in with_project_on_sys_path
func()
File "/Users/martinslabber/Documents/coding/momoko/ve3_t4/lib/python3.4/site-packages/setuptools/command/test.py", line 164, in run_tests
testLoader = cks
File "/usr/local/Cellar/python3/3.4.0_1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/unittest/main.py", line 92, in init
self.parseArgs(argv)
File "/usr/local/Cellar/python3/3.4.0_1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/unittest/main.py", line 139, in parseArgs
self.createTests()
File "/usr/local/Cellar/python3/3.4.0_1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/unittest/main.py", line 146, in createTests
self.module)
File "/usr/local/Cellar/python3/3.4.0_1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/unittest/loader.py", line 146, in loadTestsFromNames
suites = [self.loadTestsFromName(name, module) for name in names]
File "/usr/local/Cellar/python3/3.4.0_1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/unittest/loader.py", line 146, in
suites = [self.loadTestsFromName(name, module) for name in names]
File "/usr/local/Cellar/python3/3.4.0_1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/unittest/loader.py", line 105, in loadTestsFromName
module = import('.'.join(parts_copy))
File "/Users/martinslabber/Documents/coding/momoko/momoko/tests.py", line 44, in
import momoko
File "/Users/martinslabber/Documents/coding/momoko/momoko/momoko/init.py", line 15, in
from .connection import Pool, Connection
File "/Users/martinslabber/Documents/coding/momoko/momoko/momoko/connection.py", line 33, in
from .utils import log
File "/Users/martinslabber/Documents/coding/momoko/momoko/momoko/utils.py", line 28, in
class Op(gen.Task):
TypeError: function() argument 1 must be code, not str
(ve3_t4)➜ momoko git:(master) pip freeze
Momoko==1.1.3
certifi==14.05.14
psycopg2==2.5.3
tornado==4.0
Using Python 3.4.0 on Mac OS X 10.9.4
Steps to reproduce:
Quite sure it is related to this upstream change:
tornadoweb/tornado@518ecf8
according to the psycopg2.extras:
Note
Not very useful since Psycopg 2.5: you can use psycopg2.connect(dsn, cursor_factory=DictCursor) instead of DictConnection.
and psycopg2.connection.cursor_factory:
The default cursor factory used by cursor() if the parameter is not specified.
Adding this would save time on explicitly specifying cursor_factory in every execute() call.
http://pypi.python.org/pypi/Momoko/ is out of date
I am crating AsyncPool instance with argument ioloop set from tornado.ioloop.IOLoop.instance() and I am damn sure it is ok.
The problem is, I don't know what to put into new_cursor( function ...). I do not know where the execute/batch? I am not sure, but those methods are inaccesible from AsyncPool instance.
I think they should be -> In my understanding AsyncPool is "all in one object".
Documentation about making tests. With some code example ofcourse.
I didn't find answer in docs.
thanks,
duyue
Came across an opportunity to restart PostgreSQL and did so. My Tornado app ceased functioning and did not recover. Logs reveal the following exception:
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/tornado/web.py", line 1045, in wrapper
return method(self, *args, **kwargs)
File "/usr/lib/python2.6/site-packages/tornado/gen.py", line 91, in wrapper
Runner(gen).run()
File "/usr/lib/python2.6/site-packages/tornado/gen.py", line 318, in run
self.yield_point.start(self)
File "/usr/lib/python2.6/site-packages/tornado/gen.py", line 207, in start
self.func(*self.args, **self.kwargs)
File "/usr/lib/python2.6/site-packages/momoko/clients.py", line 125, in execute
self._pool.new_cursor('execute', (operation, parameters), callback)
File "/usr/lib/python2.6/site-packages/momoko/pools.py", line 215, in new_cursor
cursor = connection.cursor()
File "/usr/lib/python2.6/site-packages/psycopg2/extras.py", line 116, in cursor
return _connection.cursor(self, cursor_factory=DictCursor)
InterfaceError: connection already closed
Given that Momoko maintains a pool of connections, catching and recovering from this in client code would be difficult.
It looks like register_hstore
is in the Pool constructor any more.
$python gen_example.py
Traceback (most recent call last):
File "gen_example.py", line 194, in <module>
main()
File "gen_example.py", line 183, in main
size=1
TypeError: __init__() got an unexpected keyword argument 'register_hstore'
Checked on 1.0.0, but looks like its the issue on master as well.
Add a set of unit tests.
A class for listening to asynchronous notifications. The developer can specify a callback that gets called when a notification is received on a certain channel or all.
There's currently support for transactions in the master branch contributed by @aleksj. You put a bunch of queries in and get a bunch of queries out. I'm planning to implement the same method in the rewrite.
Something that might also be useful is getting a connection from the connection pool, use it for the entire request and then put it back in the pool.
Here's a pseudo code example:
def get(self):
conn = self.pool.get() # Connection is removed from the pool
idnr = conn.execute('INSERT SOME STUFF RETURN id;')
conn.execute('INSERT SOME STUFF BASED ON %s;', (idnr,))
self.pool.put(conn) # Connection is put back into the pool
This would be asynchronous of course, but I left the callbacks away for simplicity.
Feedback?
Hi,
I need to use the register_composite() function from psychopg2 and it ask for a cursor or a connection. I wich to do this just after creating a Momoko pool:
application.db = momoko.Pool(
dsn="dbname=assurance user=assurance password=assurance host=localhost port=5432",
minconn=1,
maxconn=10,
cleanup_timeout=10)
psycopg2.extras.register_composite('_result', application.db._get_connection(), globally=True)
but i got this error : AttributeError: 'NoneType' object has no attribute 'cursor'
even if Momoko works great in other parts of the application. So where can I access a connection ?
By the way thanks for the great work, Momoko is awesome!
I got following error when I started my web app with momoko added:
Traceback (most recent call last):
File "/usr/local/lib/python2.6/dist-packages/tornado-2.0-py2.6.egg/tornado/web.py", line 927, in _execute
getattr(self, self.request.method.lower())(*args, **kwargs)
File "hw.py", line 130, in post
user_id = self.db.get(query)
File "hw.py", line 37, in db
'cleanup_timeout': 10
File "/usr/local/lib/python2.6/dist-packages/momoko/client.py", line 29, in __init__
self._pool = Pool(**settings)
File "/usr/local/lib/python2.6/dist-packages/momoko/client.py", line 193, in __init__
self._new_conn()
File "/usr/local/lib/python2.6/dist-packages/momoko/client.py", line 209, in _new_conn
conn = psycopg2.connect(async=1, *self._args, **self._kwargs)
TypeError: 'async' is an invalid keyword argument for this function
Since the blocking-style API is done it's easier to write top-down code.
Maybe Peewee can be adapted so it uses Momoko.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.