admirito / pymongowatch Goto Github PK
View Code? Open in Web Editor NEWA pymongo extension for watching and auditing the database operations
License: Other
A pymongo extension for watching and auditing the database operations
License: Other
The aggregate command returns a CommandCursor but pymongowatch only supports normal Cursor
.
Maybe we have to create a base for all sort of cursors and then reuse it for both Cursor
and CommandCursor
.
It would be also nice to add the ability to modify operation results in "operations" configuration to be a cursor so operations with cursor results will be like other operations.
If you add multiple log handlers with multiple WatchQueues
only the last queue will handle set_ready
correctly, the others will only generate logs on timeout.
The problem seems to be that WatchMessage
only supports one call back method for log readiness, so only that last one added by the watcher class will be in effect.
Basic collection operations such as insert/delete/update has to be logged.
After adding support for multiprocessing in a69b1a9, the current approach for mutable logs started to show its problems. A simple example was 1541887 which tries to fix some bugs in set_ready
and maintaining a multiprocessing proxy for the WatchMessage
over complicates the development.
The mutable logs approach is also the source of other complexities and other bugs such as #1 which is not even due to multiprocessing complication.
The current issue propose a more effective/flexible and much simpler approach. We can generate a unique ID (e.g. using pymongo's native ObjectID
) say WatchID
and it could be stored in the WatchMessage
as usual.
Each watcher technically will not update (mutate) the logs but it can generate new ones by preserving the same WatchID
. Then the WatchQueue
can merge the new data into the previous non-ready logs.
This makes it much easier to implement multiprocessing. The WatchMessage
can be passed explicitly to the queue and without the need for a proxy. Furthermore, the user can have more choices and, if necessary, can receive all or some of the middle logs. We can even create a dedicated tool to combine duplicate logs in the final output file.
Three elements are required to add support for docker and kubernetes:
provider: a service that makes pymongowatch
and its mask source available to be mounted in other containers.
inspector: a service that search through a list of containers provided by user, creates ephemeral containers with the same image but with a custom entrypoint/command to inspect where we can install the pymongowatch
and its mask
.
patcher: an authorized service that will patch the containers with volumes / volume-mounts to put pymonowatch
and its mask on the right path.
Without any doubt the most problematic part of the pymongowatch
design that made us rewrite and rewrite all over again through the course of development is using multiprocessing
.
It seems that this story will never end. multiprocessing
is known to have problems with gevent
(although you can use the great project geventmp) and problems appear one after another.
But theoretically even in a multi-process environment each process could use its own logger handlers without using the multiprocessing
module at all. Maybe it is the way to go.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.