Comments (12)
Gojay is used in production on a highly instrumented system handling thousands of requests per second. In the context it's being used it does not leak memory, if it would, we would have noticed. It could be linked with something you're doing specifically. Let me try to reproduce what you are noticing.
from gojay.
So the reason is you are reusing the same decoder. It's actually a duplicate of this issue: #79
At the time I suggested to add a reset method. But the OP implemented his own solution. As I want gojay to stick more to the behaviour of the standard API, I will implement the reset after each decode so that the buffer doesn't grow forever if you reuse the Decoder, in the meantime you can get a new decoder between each loop and test.
from gojay.
@francoispqt I cannot get new decoder because I have a byte stream of JSON messages and on dropping the old decoder I will crop a part of the next message (I've already tried that before creating the issue).
from gojay.
Yes, of course, because it reads the bytes available in the reader into the decoder's internal buffer.
So basically, what we'd need to do, is everytime an item is read, drop the bytes already read by resetting the cursor, this is actually what the stream api already does. But you're not using the stream api. I will implement that in the regular API and evaluate the perf impact.
from gojay.
But you're not using the stream api. I will implement that in the regular API and evaluate the perf impact.
In my quick-research I did not understand how to use the stream API for my case. If that is the API that I was supposed to use for my case then don't waste time on me: I will just try to use it.
So am I understand correctly that I just should use the stream API? :)
from gojay.
Yes, the best for you would be to use the stream api, it will basically push messages to a channel whenever it's there, so you just have to read from that channel.
But if you want, I have just implemented the reset to reuse the Decoder
because I do think there is a usecase for that and I want it to be consistent with the standard json. I have tested and it works correctly with no impact on performance, I've added a test as well. You can try it in the feature/reset-decoder
branch.
If it's ok for you, I will merge it to master and tag a new release tomorrow.
from gojay.
I will test it tomorrow, thank you very much :)
from gojay.
Well, I've switched the branch and it did not help.
gojay# git log | head -2
commit 1cb50f7b876bd5abaa4ffa66c9e28cbb0c3547fe
Author: francoispqt <[email protected]>
# grep RSS /proc/`pgrep logger`/status
VmRSS: 3023848 kB
I will try to use the steam API. Anyway, thanks :)
from gojay.
Are you running it in a unit test (mocking your udp connection) or you're really testing your stream?
from gojay.
Are you running it in a unit test (mocking your udp connection) or you're really testing your stream?
The second one:
I have a testing environment with real traffic. So I just run my application and watch how it works.
$ go version
go version go1.12 linux/amd64
from gojay.
Ok. Let me setup an environment quickly. It shouldn't grow like that. I know it doesn't with the stream api, as we are using it, and you can try the example in examples/websocket
, just run go run main.go
and you'll see it doesn't grow.
from gojay.
Ok, so I have pushed updates to the branch feature/reset-decoder
.
I've added an example which replicates what you are doing, you can find it here: https://github.com/francoispqt/gojay/blob/9df89bce7ea024f84eb4d44efb460bbf755e7b58/examples/reuse-decoder/main.go
You can run the example doing: go run main.go
The memory usage does not grow. Please let me know if it works in your case.
from gojay.
Related Issues (20)
- Decoder returns misleading error at EOF
- gojay can't generator code for two package
- encoding ints to strings HOT 1
- Unmarshal modifies input data HOT 1
- wrong code generated with custom slice types
- Generator fails for map[string]interface{} field HOT 1
- TestMessage_Unmarshal and TestMessage_Marshal fail at test
- panic on example stream program
- NullBytes appearing in strings HOT 1
- Go test build fails under 32-bit systems
- Is the project dead? HOT 1
- Streaming reflective JSON encoder
- Decoder.{{Type}}Null does not behave as documented when encountering incorrect typed json elements.
- Return length of decoded input
- Return error on unknown object key
- gojay generated code doesn't work for type names starting with K
- Support naming helper Array types with correct plural (Currencies, not Currencys)
- Manually adding an error to an Encoder
- Stream encoding with gojay seems to be a problem. HOT 1
- Upgrade protobuf dependency to >= 1.3.2 to get rid of CVE-2021-3121 vulnerability
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from gojay.