Comments (7)
Hi @swizzley,
I think what you are looking for is available with version 1.1.0.
If the HTTP endpoint returns a proper JSON structure it is also added in the field jsonBody (see https://github.com/christiangalsterer/httpbeat/blob/master/docs/fields.asciidoc). You can also modify if "dots" shall be replaced in the structure and if the structure shall be flattened, see https://github.com/christiangalsterer/httpbeat/blob/master/docs/configuration.asciidoc
from httpbeat.
yeah my problem is that it is returning a json array therefore it prints the json as a string, so the response field has this massive json blob as its' body. Additionally I don't want to create unique IDs in elastic search for each poll interval, I just want the documents inside the json poll to derive their ID from a given field and then simply update that document on subsequent polls. I've figured out how to do this with a for loop to derive the _id then select the document from the response with a lil jq before adding the metadata before sending in _bulk, but ideally I'd like to integrate these methods into your beat , what do you think?
from httpbeat.
maybe an option to deactivate the body field in the response would be an option.
I just ran into this limitation with a very big response (using graylog as a target for the logstash output):
message [java.lang.IllegalArgumentException: Document contains at least one immense term in field="beat_response_body" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped. Please correct the analyzer to not produce such terms. The prefix of the first immense term is: '...', original message: bytes can be at most 32766 in length; got 167036]
worked after deactivating the body manually and recompiling.
http://stackoverflow.com/questions/24019868/utf8-encoding-is-longer-than-the-max-length-32766
from httpbeat.
We actually could set it to nil if unmarshalling succeeds.
from httpbeat.
Hi everybody,
thank you for all your feedback. I was also wondering if I should not slightly change the behaviour in a way that you either return the jsonBody or body field.
Here I see the options:
- Always return both fields
- Set body to nil, if unmarshalling to json is successful
- Users need to specify the return format. body field is then either string or json.
Would be great to get some feedback on your preferences.
from httpbeat.
Thanks for the fast response.
I personally don't see any value in 1. as the same information is sent twice.
I'm actually not sure why anyone would want to send json info as a string, but having an option (3) could also be used to not try to unmarshal non-json output in the first place and could be a minor performance improvement. Due to the fact that already a json related setting exists (dot mode) I would vote vor 3.
from httpbeat.
Released with version 3.0.0.
There is now a new "output_format" parameter which allows to specify the format of the response body. If it is not set or "string" then the body is returned as string in the "body" field. If set to "json" then the response body is returned as json in the "jsonBody" field.
from httpbeat.
Related Issues (20)
- support for beats 5.0.0 ? HOT 9
- json unmarshalling of large numbers HOT 2
- Elasticsearch plugin issue, missing json file HOT 6
- https issues HOT 14
- Problem in parsing Spring Boot Metrics HOT 3
- http beat consume > 2Gb of memory HOT 6
- Current date in httpbeat.yml HOT 1
- Connection error when sampling many URLs HOT 5
- Is it possible to include response time? HOT 1
- Does httpbeat support dynamic reloading of configuration files? HOT 1
- rpm package version doesn't match release version HOT 2
- timestamp config
- Making the template work for kibana HOT 20
- HTTP Status code in event doesn't match documentation
- Support returning HTTP status phrase
- Is it possible to set up a `config_dir` for conf files HOT 5
- NTLM Authentication HOT 1
- Add ability to prevent sending certain headers?
- No method specified error
- problems about Build, Test, Run HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from httpbeat.