Comments (14)
has there been any traction on this? 7+ years open, PR request pending almost 3 years now...
from logstash-output-mongodb.
Up!
I'm also interested in this, it's very limiting the absence of the mongo upsert in the output plugin.
from logstash-output-mongodb.
Hi all, I'm using the HTTP input plugin to receive HTTP POST/PATCH requests, and would like to map the PATCH in the mongodb output plugin to an update on an existing document based on the $set operator. I'm not familiar with Ruby, but checking the latest mongodb.rb, it looks like only insert_one() is used. Are there any plans to support update_one() or other mechanisms to update an existing Mongo document?
from logstash-output-mongodb.
@stevecalvert - In general the LS outputs are append only. Consider this blog article from LinkedIn. You turn your collection into a journal showing how the document changed over time.
from logstash-output-mongodb.
@guyboertje on the other hand, we support action
of delete, insert, create, update --- for the elasticsearch output. I'm open to doing so for the mongodb output; in which case we'll need to add another setting to let users set whatever the equivalent to the 'document_id' setting would be.
from logstash-output-mongodb.
@stevecalvert @jordansissel - in Mongo we could use update with the upsert option. https://docs.mongodb.org/manual/reference/method/db.collection.update/#upsert-parameter
Obviously driven by config options, to allow the user to specify which field contains the query key for the upsert. e.g.
config :upsert_query_key, :validate => :string, :default => "_id"
config :upsert, :validate => :boolean, :default => false.
from logstash-output-mongodb.
Has this been ruled out as something that will be added? I'm needing upsert functionality for a project I'm working on so I'm evaluating options. This plugin works great but I need some way to handle a record getting sent thru with an update....
from logstash-output-mongodb.
Hi, Is there any update on this issue ?
from logstash-output-mongodb.
@joelpermanas any update will be publicly visible here. PR contributions are welcome :)
from logstash-output-mongodb.
I would like to have this feature also, please.
Thank you for the great product.
from logstash-output-mongodb.
from logstash-output-mongodb.
Did anybody start with an implementation or plans to do so?
from logstash-output-mongodb.
I am implementing this and I plan to use these config flags:
config :action, :validate => :string, :required => true
config :query_key, :validate => :string, :required => false, :default => "_id"
config :query_value, :validate => :string, :required => false
config :upsert, :validate => :boolean, :required => false, :default => false
Where action may be one of insert
(the default, behaves like plugin does now), update
or replace
. For update and replace, query to find document to update/replace is built like this:
:filter => {@query_key => @query_value}
Where _id
would be the default value for @query_key
config setting. By setting upsert
to true one could enable upsert mode for update/replace. For insert this setting wouldn't have any effect.
I tried to align naming of config settings to elasticsearch output but I think, at least for me, upsert
is much more clear than doc_as_upsert
.
Any comments? Suggestions?
from logstash-output-mongodb.
I opened a PR for this (see above). Could anybody please take a look and let me know whether my changes make sense for you, too?
It's working for us so far.
from logstash-output-mongodb.
Related Issues (20)
- bulk size limit HOT 2
- Logstash output getting failed with mongodb as output HOT 1
- Container crashes after plugin installation HOT 3
- Failed to sent event with wrong argument numbers HOT 12
- logstash mongodb output - wrong number of arguments HOT 4
- Plugin version 3.1.6 issue HOT 5
- failed to send event to MongoDB, retrying in 3 seconds {:event=>#LogStash::Event:0x2367ca72, :exception=>#<ArgumentError: wrong number of arguments (2 for 1)>} HOT 2
- MongoDB Atlas Integration with Logstash through logstash-output-mongodb plugin HOT 1
- Logstash not able to load mongodb output plugin HOT 1
- [Enhancement] Allow continue-on-error when insert duplicate records
- :exception=>#<Mongo::Error::NoServerAvailable:
- The process exits when a duplicate _id is inserted
- Failed to send event to MongoDB, retrying in 3 seconds {:event=>#<LogStash::Event:0x115d2456>, :exception=>#<Mongo::Error::NoServerAvailable: No server is available matching preference: #<Mongo::ServerSelector::Primary:0x101bcedd @tag_sets=[], @server_selection_timeout=30, @options={:database=>"Logs", :user=>"username", :password=>"passwd"}>>} HOT 1
- Logstash MongoDB Output plugin 3.1.7 error
- Logstash-output-mongodb error JavaLang::NullPointerException HOT 1
- MongoDB `uri` may contain password where it can be leaked in debug logs.
- warning: BigDecimal.new is deprecated; use BigDecimal() method instead. HOT 1
- DLQ support
- duplicate ID infinite retry (and blocks pipeline)
- MongoDB connection error - Failed to handshake HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from logstash-output-mongodb.