Comments (21)
Hey there @jingyuanswitchco. I am actually not sure of the behavior of non-integer counters. This likely undefined behavior for us because when we think of counters we increment them in whole numbers.
Thanks for opening the issue, we will write up a test case and check.
As for the change in behavior, we didn't explicitly change anything that I think would have affected this behavior, but since it's undefined perhaps we did!
from veneur.
Thanks for the super fast response @cory-stripe!. I will also try to check with the latest code and verify whether it fixed the issue and update you if I find something.
from veneur.
No problem! If you find anything suspicious do let us know. Otherwise we'll hopefully find it by writing some test cases.
from veneur.
I saw the latest code is using golang 1.9. However on the official website: https://golang.org/dl/. It is listing go1.9 as unstable version. I am wondering why does Veneur need to upgrade to 1.9? Or is it possible I can still compile it using go1.8.3?
from veneur.
It's possible to still use 1.8.3, but we use 1.9 for our internal stuff.
from veneur.
I am trying to compile the latest code with 1.8.3. But I run into the error when running command gofmt -w .:
vendor/golang.org/x/net/context/go19.go:15:14: expected type, found '='
vendor/golang.org/x/net/context/go19.go:20:17: expected type, found '='
vendor/golang.org/x/net/ipv4/batch.go:59:14: expected type, found '='
vendor/golang.org/x/net/ipv4/batch.go:78:2: expected declaration, found 'if'
vendor/golang.org/x/net/ipv6/batch.go:56:14: expected type, found '='
vendor/golang.org/x/net/ipv6/batch.go:69:2: expected declaration, found 'if'
I understand it is a formatting command. Just a FYI. I skipped the formatting and it compiled successfully now.
from veneur.
Ah, yes. I believe we had to make some changes for go fmt
in 1.9 now that you mention it. Sorry about that!
from veneur.
No worries:) I just tried to run the latest code. The same issue is still there. Do you plan to get this fixed?
from veneur.
I plan to write some tests to exercise it, but I have no ETA on this. We've got some other stuff in progress at the moment.
from veneur.
Is it possible you can quickly point me to the code where we are handling this. And I can investigate it.
from veneur.
I'm unsure. You could write a test in the sampler testing code to see if perhaps the counter doesn't work. Outside of that you'd have to test the JSON marshaling code. I'm also unsure if Datadog even works with a float counter, but your issue implies it does.
Othewise, I'm sorry but I can't tell you a specific time. This being OSS we have other obligations and these come as we can get to them!
from veneur.
Thanks Cory! Yes datadog agent worked fine with float counters.
from veneur.
Hi Cory,
I found:
Line 84 in 47f0a97
this line is essentially change a float to an integer. Which would convert float number that is smaller than 1 to 0. Since the sample was passed in as a float64, I am wondering why do we want to force it to convert it to int64? If I want to submit a change and not converting sample to int64, is it an acceptable change?
Thanks,
Jingyuan
from veneur.
Line 79 in 47f0a97
It seems like it is because Counter is defined as an int64. Not sure if I changed it to float64 something will be broken.
from veneur.
Gotya, in rereading this issue I now see that I misunderstood the original ask. You are pointing out that Veneur differs from the official DogStatsD client in that it does not work with counters. I misunderstood your original statements to mean that veneur's behavior had changed.
Ok, we'll work on this soon. I'm not sure how much work it entails. I'll report back to you!
from veneur.
Hi Cory,
I tried a fix today:
diff --git a/samplers/samplers.go b/samplers/samplers.go
index aa5dfa6..6076fae 100644
--- a/samplers/samplers.go
+++ b/samplers/samplers.go
@@ -76,12 +76,12 @@ type JSONMetric struct {
type Counter struct {
Name string
Tags []string
- value int64
- value float64
}
// Sample adds a sample to the counter.
func (c *Counter) Sample(sample float64, sampleRate float32) {
- c.value += int64(sample) * int64(1/sampleRate)
- c.value += sample * (1/float64(sampleRate))
}
// Flush generates a DDMetric from the current state of this Counter.
@@ -119,7 +119,7 @@ func (c *Counter) Export() (JSONMetric, error) {
// Combine merges the values seen with another set (marshalled as a byte slice)
func (c *Counter) Combine(other []byte) error {
- var otherCounts int64
- var otherCounts float64
buf := bytes.NewReader(other)
err := binary.Read(buf, binary.LittleEndian,
And after I changed the int64 to float64, it seems fixed the issue when I tested this on our infra. Not sure this may violate any design principle for Veneur. Please let me know if this helps.
Best,
Jingyuan
from veneur.
Unfortunately I don't think this change is simple. Because of our naive serialization (the bytes above) trying to decode this would fail unless all veneurs were update at the same time. In other words, this would fail if someone sent a byte-encoded int64 to a box expecting a float64.
We likely need to redesign this to use floats for everything or something like that. Doing so will require a new import API. I don't think we'll have a quick solution hereβ¦
from veneur.
There is a way to allow deserialization of either float or int types simultaneously, allowing for a gradual upgrade, but it'd be kind of cumbersome and a potentially problematic performance hit. We'd have to profile it and test it out to know whether or not it'd be problematic to update it in production.
from veneur.
Neat!
from veneur.
Hi Cory,
Just saw you put this as an enhancement. Just want to touch base with you whether this will be fixed soon or it will need more time and there is no ETA to put in the fix.
Best,
Jingyuan
from veneur.
No ETA, there's a speed bump here that we don't have an easy way to made this backward compatible, so I'm thinking will look at this for 1.7 or 2.0.
from veneur.
Related Issues (20)
- Veneur forarding to Datadog - avg higher than max, avg missing data
- Support SO_REUSEPORT on darwin/etc HOT 1
- Conflicting documentation
- AddTags only respected when using metric sink routing
- Veneur facing client timeout for large metric count
- Security Vulnerabilities - v14.1.0-release-prod
- security issue: veneur.org is no longer owned by stripe HOT 4
- DogStatsD protocol >= v1.2 support
- Flush sinks on shutdown
- How Can I send metrics collected by datadog agent or apm trace? HOT 4
- Global Histograms & Timers do not work with HTTP forwarding
- "SSL certificate problem: self signed certificate in certificate chain error" while building alpine docker image.
- Debian dockerfile build is failing during the tests HOT 2
- Alpine image for 13.2.0 is missing in Dockerhub HOT 1
- Is there any upgrade plan to golang:1.17 alpine image ? HOT 5
- Release HOT 2
- Veneur is unable to parse packets from the go dogstatsd client
- Veneur does not like New Relic Insert Key config
- Publish multi arch[amd64/arm64] docker images for veneur
- Stripe/veneur docker images fail using default configuration with obtuse error HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. πππ
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google β€οΈ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from veneur.