Comments (7)
Make sure checkpointing correctly. You might also check whether the record processor is restarted. Enable more log for stress test can find the issue.
We had stress tested it before and didn't find any data duplication issue.
from vmware-go-kcl.
Also, https://docs.aws.amazon.com/streams/latest/dev/kinesis-record-processor-duplicates.html
from vmware-go-kcl.
I found the issue.
Fixed provisioning was enabled on DynamoDB and we had exceeded the provisioned throughput.
Setting the DynamoDB to on-demand provisioning fixed the problem.
The fixed provisioning was triggering shardConsumer.getRecords
failures:
Error in refreshing lease on shard: shardId-000000001185 for worker: b3e512c484b4-23a4dbb5-060c-11ea-9728-0242ac110005. Error: ProvisionedThroughputExceededException: The level of configured provisioned throughput for the table was exceeded. Consider increasing your provisioning level with the UpdateTable API.
status code: 400, request id: xxxEDITEDxxx
immediately followed by:
Error in getRecords: ProvisionedThroughputExceededException: The level of configured provisioned throughput for the table was exceeded. Consider increasing your provisioning level with the UpdateTable API.
status code: 400, request id: xxxEDITEDxxx
side note: the same error is logged twice. In that case it might be desirable to not log at the error site. Instead we could return a new error created with fmt.Errorf
that provides additional context, and let the caller logs it.
Make sure checkpointing correctly. You might also check whether the record processor is restarted. Enable more log for stress test can find the issue.
Still, I'm not sure I get why losing and restarting a shardProcessor should lead to record duplication. Ideally the restarted processor should take on where the last one stopped, right?
Also at one point the same cause (ProvisionedThroughputExceededException
) triggered another effect, probably when a new shardProcessor was restarting.
Error: ProvisionedThroughputExceededException: The level of configured provisioned throughput for the table was exceeded. Consider increasing your provisioning level with the UpdateTable API.
status code: 400, request id: xxxEDITEDXXX
Note the space in front of Error
: this detail led me to the only place in the code base where Error:
is used:
vmware-go-kcl/clientlibrary/worker/worker.go
Lines 260 to 268 in d0e9c4c
from vmware-go-kcl.
@taojwmware I know that from the aws document you linked about duplicated records one of the possible case of duplication is a worker terminated unexpectedly, still I'm not sure I understand why it should be the case if checkpointer.GetLease
fails, since the current shard owner hasn't moved its checkpoint yet when it happens. Thus even if a new worker gets started, it should continue where the previous owner stopped, right?
Am I missing something here?
from vmware-go-kcl.
For logging, there were some internal discussions long time before. We decided we'd rather log more instead of logging less to make troubleshooting easy. We use to wrap error and only log in one place but find it hard to trace back exactly error origin.
from vmware-go-kcl.
Depends on how to checkpoint, if you checkpoint it immediately after get record before process and store, restart processor will cause data lose. If only checkpointing the whole records but storing parted of processed record, processor restarts will cause the stored portion of records to be refetched after restart because no checkpoint was done on those record.
from vmware-go-kcl.
Here is the link for Kafka Clients (At-Most-Once, At-Least-Once, Exactly-Once, and Avro Client)
https://dzone.com/articles/kafka-clients-at-most-once-at-least-once-exactly-o
It applies to Kinesis as well.
from vmware-go-kcl.
Related Issues (20)
- Lease stealing code exception HOT 3
- Option to assume role in AWS Kinesis client HOT 1
- Checkpointer interface breaking change in v1.3.0 HOT 1
- worker.go:339 Error in getRecords: unexpected EOF
- How to get the StreamName/ConsumerName inside ProcessRecords method HOT 1
- Unsupported protocol scheme
- Workload skewed across workers HOT 3
- (Still) Too many calls to DescribeStream HOT 3
- Error in getRecords leaves a dangling record processor. HOT 1
- Stuck in waitOnParentShard after resharding HOT 4
- Possible Data Race for input Checkpointer HOT 1
- Semantic versioning friendly version tags HOT 1
- Shard consumer sometimes fails to recover from error refreshing lease HOT 9
- Prometheus metrics should add appname as label and not in the metric name HOT 1
- Possibility to Work w/ DynamoDB Streams HOT 3
- Error in publishing cloudwatch metrics. Error: NoCredentialProviders: no valid providers in chain. HOT 1
- How to use record deaggregator? HOT 2
- Multiple consumers processing records from the same shard. HOT 10
- Tagged toolchain docker container vmware/go-kcl-toolchain:0.1.2 uses golang version 1.12.4
- AWS Go SDK V2 HOT 15
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from vmware-go-kcl.