admirito / pymongowatch Goto Github PK

The aggregate command returns a CommandCursor but pymongowatch only supports normal Cursor.

Maybe we have to create a base for all sort of cursors and then reuse it for both Cursor and CommandCursor.

It would be also nice to add the ability to modify operation results in "operations" configuration to be a cursor so operations with cursor results will be like other operations.

If you add multiple log handlers with multiple WatchQueues only the last queue will handle set_ready correctly, the others will only generate logs on timeout.

The problem seems to be that WatchMessage only supports one call back method for log readiness, so only that last one added by the watcher class will be in effect.

Basic collection operations such as insert/delete/update has to be logged.

After adding support for multiprocessing in a69b1a9, the current approach for mutable logs started to show its problems. A simple example was 1541887 which tries to fix some bugs in set_ready and maintaining a multiprocessing proxy for the WatchMessage over complicates the development.

The mutable logs approach is also the source of other complexities and other bugs such as #1 which is not even due to multiprocessing complication.

The current issue propose a more effective/flexible and much simpler approach. We can generate a unique ID (e.g. using pymongo's native ObjectID) say WatchID and it could be stored in the WatchMessage as usual.

Each watcher technically will not update (mutate) the logs but it can generate new ones by preserving the same WatchID. Then the WatchQueue can merge the new data into the previous non-ready logs.

This makes it much easier to implement multiprocessing. The WatchMessage can be passed explicitly to the queue and without the need for a proxy. Furthermore, the user can have more choices and, if necessary, can receive all or some of the middle logs. We can even create a dedicated tool to combine duplicate logs in the final output file.

Three elements are required to add support for docker and kubernetes:

1. provider: a service that makes pymongowatch and its mask source available to be mounted in other containers.

2. inspector: a service that search through a list of containers provided by user, creates ephemeral containers with the same image but with a custom entrypoint/command to inspect where we can install the pymongowatch and its mask.

3. patcher: an authorized service that will patch the containers with volumes / volume-mounts to put pymonowatch and its mask on the right path.

Without any doubt the most problematic part of the pymongowatch design that made us rewrite and rewrite all over again through the course of development is using multiprocessing.

It seems that this story will never end. multiprocessing is known to have problems with gevent (although you can use the great project geventmp) and problems appear one after another.

But theoretically even in a multi-process environment each process could use its own logger handlers without using the multiprocessing module at all. Maybe it is the way to go.