Comments (4)
The error on CDC sink persists even after stopping the inserts, after a truncate table _cdc_sink.defaultdb_t;
, and cancel and restart the CDC-changefeed job.
from replicator.
Might be a buffer underrun here: https://github.com/cockroachdb/cockroach/blob/a194629fa50dd681329e158f78a2f0988684595b/pkg/sql/copy.go#L474
Will pull on that thread.
from replicator.
Summary of findings:
- In https://github.com/jackc/pgx/blob/23b4656fc573dcecd8bdce9c0235249e73d384f4/copy_from.go#L187-L188 the
buildCopyBuf
function has a test to limit the size of the output buffer, but the generated buffer can still exceed 64k. - When this buffer reaches https://github.com/jackc/pgconn/blob/53f5fed36c570f0b5c98d6ec2415658c7b9bd11c/pgconn.go#L1217-L1222 it gets sent as multiple wire messages. This means that a tuple can span multiple length-delimited chunks.
- The wire format https://www.postgresql.org/docs/current/protocol-message-formats.html allows for 32-bit message lengths in the copy protocol, but it also states that "messages sent by frontends may divide the data stream arbitrarily."
- CRDB's copy-in code https://github.com/cockroachdb/cockroach/blob/65abfccdf6b61495e0f262307e1c2d7d492ce55c/pkg/sql/copy.go#L251 assumes that a row's tuple values don't span wire messages, which appears to contravene the pgwire protocol.
- I can't find a test in CRDB that exercises the binary copy protocol with enough rows or large-enough values to trigger the protocol errors.
- If I patch
pgconn.go
to use an intermediate buffer with a larger capacity, my large-clob test passes.
from replicator.
Upstream fix is available in 21.1.9, so I'm going to roll this hack out of the codebase.
from replicator.
Related Issues (20)
- Kafka: integration testing with OATH2
- Composite checkpoints
- Kafka: provide a grafana dashboard to track kafka connector performance. HOT 1
- Kafka: retrieve offsets in parallel across multiple topics and partitions.
- Rename this repo to Replicator HOT 1
- Panic in mylogical/conn.go; index out of bounds
- Data race in leases_test.go
- Demo for update-only table
- Support data models with natural keys
- mysql GTIDs are incorrect in the memo table
- mysql replication requires target database to be same as the source database.
- Handle non-idempotent Changefeed redelivery HOT 2
- Verify varchar to Oracle clob
- Allow split-mode Replicator to be restarted if the target is down
- Update cdc/server/integration_test.go with workload.Checker
- Lingering global flags
- configureTable.deletesTo should support a callback
- Allow workload subcommand to write to real db
- Load API SQL does not work with MariaDB
- oracle: evaluate expressions in the value field
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from replicator.