conduitio-labs / conduit-connector-azure-storage Goto Github PK
View Code? Open in Web Editor NEWConduit connector for Azure Blob Storage
License: Apache License 2.0
Conduit connector for Azure Blob Storage
License: Apache License 2.0
When the connector is in the CDC mode everything is okay, except when you want to pause a running pipeline and resume it after some time. In this case, the last operation performed before a pipeline pause will be duplicated after the pipeline resume.
It seems that CDC doesn't work properly with Azure's BlockBlob
type, it only works with AppendBlob
.
BlockBlob
type;Testing this connector locally in a conduit pipeline, when not providing the optional configuration field maxResults
, conduit throws an error when trying to parse. Here's an example log:
2024-05-16T14:28:53+00:00 ERR node stopped error="node azure-to-log:example.in stopped with error: could not open source connector: could not configure source connector plugin: configuration error: failed to parse \"maxResults\" config value: strconv.ParseInt: parsing \"4s8\": invalid syntax" component=pipeline.Service node_id=azure-to-log:example.in stack=[{"file":"/Users/rb/code/conduitio/conduit/pkg/pipeline/lifecycle.go","func":"github.com/conduitio/conduit/pkg/pipeline.(*Service).runPipeline.func2","line":583},{"file":"/Users/rb/code/conduitio/conduit/pkg/pipeline/stream/source.go","func":"github.com/conduitio/conduit/pkg/pipeline/stream.(*SourceNode).Run","line":110},{"file":"/Users/rb/code/conduitio/conduit/pkg/connector/source.go","func":"github.com/conduitio/conduit/pkg/connector.(*Source).configure","line":287},{"file":"/Users/rb/code/conduitio/conduit/pkg/plugin/connector/standalone/v1/client.go","func":"github.com/conduitio/conduit/pkg/plugin/connector/standalone/v1.unwrapGRPCError","line":165}]
Workaround: Provide this configuration field manually.
maxResults
Example of a pipeline using this connector as a source that will expose the error:
version: 2.2
pipelines:
- id: azure-to-log
# run pipeline on startup
status: running
description: >
Example pipeline reading from file "example.in" and writing into file "example.out".
connectors:
- id: example.in
type: source
# use the built-in file plugin as the source
plugin: azure-storage
settings:
connectionString: "DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://127.0.0.1:10000/devstoreaccount1;QueueEndpoint=http://127.0.0.1:10001/devstoreaccount1;TableEndpoint=http://127.0.0.1:10002/devstoreaccount1;"
containerName: "test"
- id: example.out
type: destination
# use the built-in file plugin as the destination
plugin: file
settings:
# write data to example.out
path: ./example.out
# output the raw payload as a string
sdk.record.format: template
sdk.record.format.options: '{{ printf "%s" .Payload.After }}'
v0.3.0
We have two acceptance tests related to CDC failing (for more context: #6 (comment)).
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.