Coder Social home page Coder Social logo

conduit-connector-azure-storage's People

Contributors

adamhaffar avatar alyakimenko avatar dependabot[bot] avatar donatorsky avatar lovromazgon avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

conduit-connector-azure-storage's Issues

Bug: record is duplicated after a pause

Bug description

When the connector is in the CDC mode everything is okay, except when you want to pause a running pipeline and resume it after some time. In this case, the last operation performed before a pipeline pause will be duplicated after the pipeline resume.

Steps to reproduce

  1. Create and start a Conduit pipeline with Azure source and any destination;
  2. Wait until the Azure connector is switched to the CDC mode;
  3. Update any existing blob, the update should be written to a destination;
  4. Stop the pipeline;
  5. Resume the pipeline;
  6. The update from step 3 is duplicated and written to a destination again.

Version

db39e39

Bug: CDC doesn't work with Azure's BlockBlob blob type

Bug description

It seems that CDC doesn't work properly with Azure's BlockBlob type, it only works with AppendBlob.

Steps to reproduce

  1. Create a blob with BlockBlob type;
  2. Update the blob;
  3. Connector recognizes the update as insert;
  4. Delete the blob;
  5. Connector doesn't recognize the delete.

Version

0364a85

Bug: can't parse maxResults when not specified

Bug description

Testing this connector locally in a conduit pipeline, when not providing the optional configuration field maxResults, conduit throws an error when trying to parse. Here's an example log:

2024-05-16T14:28:53+00:00 ERR node stopped error="node azure-to-log:example.in stopped with error: could not open source connector: could not configure source connector plugin: configuration error: failed to parse \"maxResults\" config value: strconv.ParseInt: parsing \"4s8\": invalid syntax" component=pipeline.Service node_id=azure-to-log:example.in stack=[{"file":"/Users/rb/code/conduitio/conduit/pkg/pipeline/lifecycle.go","func":"github.com/conduitio/conduit/pkg/pipeline.(*Service).runPipeline.func2","line":583},{"file":"/Users/rb/code/conduitio/conduit/pkg/pipeline/stream/source.go","func":"github.com/conduitio/conduit/pkg/pipeline/stream.(*SourceNode).Run","line":110},{"file":"/Users/rb/code/conduitio/conduit/pkg/connector/source.go","func":"github.com/conduitio/conduit/pkg/connector.(*Source).configure","line":287},{"file":"/Users/rb/code/conduitio/conduit/pkg/plugin/connector/standalone/v1/client.go","func":"github.com/conduitio/conduit/pkg/plugin/connector/standalone/v1.unwrapGRPCError","line":165}]

Workaround: Provide this configuration field manually.

Steps to reproduce

  1. Build binary for this connector
  2. Use it as a source
  3. Don't specify optional configuration field maxResults
  4. See the error

Example of a pipeline using this connector as a source that will expose the error:

version: 2.2
pipelines:
  - id: azure-to-log
    # run pipeline on startup
    status: running
    description: >
      Example pipeline reading from file "example.in" and writing into file "example.out".

    connectors:
      - id: example.in
        type: source
        # use the built-in file plugin as the source
        plugin: azure-storage
        settings:
          connectionString: "DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://127.0.0.1:10000/devstoreaccount1;QueueEndpoint=http://127.0.0.1:10001/devstoreaccount1;TableEndpoint=http://127.0.0.1:10002/devstoreaccount1;"
          containerName: "test"
      - id: example.out
        type: destination
        # use the built-in file plugin as the destination
        plugin: file
        settings:
          # write data to example.out
          path: ./example.out
          # output the raw payload as a string
          sdk.record.format: template
          sdk.record.format.options: '{{ printf "%s" .Payload.After }}'

Version

v0.3.0

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.