Coder Social home page Coder Social logo

Comments (15)

brian-mulier-p avatar brian-mulier-p commented on July 23, 2024 1

Hey ! I'll try to reproduce it later in the day. Can you try the same thing with a local storage (if not I'll reproduce it with S3 but it would help identifying whether it's a UI issue or backend one) ?

from kestra.

brian-mulier-p avatar brian-mulier-p commented on July 23, 2024 1

Thank's for providing effort with local storage, I'm setting up the reproducer 👍

from kestra.

brian-mulier-p avatar brian-mulier-p commented on July 23, 2024 1

For traceability:
the storage-minio was used directly against an S3 bucket (and not a minio one) which led to some issues among which the fact that in the storage-minio plugin we remove any leading slash when querying due to limitations from minio which in the end doesn't work properly with S3.

The right way is to use the storage-s3 plugin but in this case it was causing an issue as the DNS is not configured to reach virtual-host-style S3 URL. The behaviour of the S3 SDK is to detect the domain resolution styling by checking if the url contains any dot (mostly IP) prior to path. If it contains some it will default to path-styling else it goes to virtual-host-styling.

In the case above it's not working as we have a strong DNS limitation and the allowed styling is a fixed one that doesn't go with the above domain resolution rule.

To solve that I introduced a new forcePathStyle configuration for storage-s3 plugin. Putting it to true should solve the issue. Feel free to open again if it doesn't work.

Moreover I enhanced the doc on storages to include all properties currently available

from kestra.

anna-geller avatar anna-geller commented on July 23, 2024

more context https://kestra-io.slack.com/archives/C03FEC452NQ/p1718812771627879

from kestra.

aku avatar aku commented on July 23, 2024

I've found the missing files in S3 storage:

2024-06-19 15:55            3  s3://kestra//test/_files/test.py
2024-06-19 18:26           11  s3://kestra//test/_files/test.txt

It looks like that the bug is in UI

from kestra.

aku avatar aku commented on July 23, 2024

I've created a simple workflow that tries to read the file:

image

As you might notice the file is missing from UI, however the flow is able to access it

image image

If I try to create a file with the same name the system will overwrite the existing file (but the file is still not visible in UI)

from kestra.

aku avatar aku commented on July 23, 2024

Another observation is that the files created via UI are missing from the exported files archive. It contains only yaml flows definitions.

from kestra.

aku avatar aku commented on July 23, 2024

@brian-mulier-p I've tested this case with a dockerized kestra It does not reproduce with a local storage.

docker run --pull=always --rm -it -p 8080:8080 --user=root \
 -v /var/run/docker.sock:/var/run/docker.sock \
 -v /tmp:/tmp kestra/kestra:latest-full server local
image

from kestra.

aku avatar aku commented on July 23, 2024

Also, I've checked export functionality and it was working correctly with a local dockerized kestra - test files were in the downloaded archive

from kestra.

aku avatar aku commented on July 23, 2024

Thank's for providing effort with local storage, I'm setting up the reproducer 👍

To give more context:
I've started with Kestra 16.xx (i don't remember minor version but it was the last release before 17)
I've gradually upgraded my installation from 17.1 to 17.4 (updated binaries and re-installed plugins, I'm running Kestra in VMs)

I.e. if you will fail to reproduce it in 17.4 with a fresh install, it might be something related to migration from a previous versions. In this case I'm willing to do a local debugging/analysis if you provide me with ideas/instruction. Feel free to tag me in Slack

A file will be overwritten in S3 if I create a new one with the same name - it might be another/related bug.

from kestra.

brian-mulier-p avatar brian-mulier-p commented on July 23, 2024

So yes I didn't manage to reproduce the issue. Dumb question but are you installing plugins on your own ? If so make sure your storage-s3 plugin is the 0.17.0 one.
Also can you show the listing content of your S3 bucket under /test/_files ? If it doesn't exist basically under /test

from kestra.

brian-mulier-p avatar brian-mulier-p commented on July 23, 2024

Aside from that can you show your network tab (in your browser) when opening the editor ? Maybe export it in HAR format and provide it here. At least provide the /namespaces/test/files call output

from kestra.

aku avatar aku commented on July 23, 2024

So yes I didn't manage to reproduce the issue. Dumb question but are you installing plugins on your own ? If so make sure your storage-s3 plugin is the 0.17.0 one. Also can you show the listing content of your S3 bucket under /test/_files ? If it doesn't exist basically under /test

I have ansible playbook that installs plugins:

    - name: Install kestra plugins # noqa command-instead-of-shell partial-become
      ansible.builtin.shell:
        cmd: '"{{ kestra_dir }}/kestra" plugins install --plugins="{{ kestra_dir }}/plugins"  "{{ item }}"'
      loop:
        - io.kestra.plugin:plugin-git:LATEST
        - io.kestra.plugin:plugin-fs:LATEST
        - io.kestra.plugin:plugin-elasticsearch:LATEST
        - io.kestra.plugin:plugin-kafka:LATEST
        - io.kestra.plugin:plugin-docker:LATEST
        - io.kestra.plugin:plugin-redis:LATEST
        - io.kestra.plugin:plugin-jdbc-postgres:LATEST
        - io.kestra.plugin:plugin-jdbc-clickhouse:LATEST
        - io.kestra.plugin:plugin-script-node:LATEST
        - io.kestra.plugin:plugin-script-python:LATEST
        - io.kestra.plugin:plugin-script-shell:LATEST
        - io.kestra.plugin:plugin-notifications:LATEST
        - io.kestra.plugin:plugin-serdes:LATEST
        - io.kestra.storage:storage-minio:LATEST
        - io.kestra.storage:storage-s3:LATEST
      become_user: "{{ kestra_sys_user }}"
      changed_when: false
      tags:
        - plugins

When a new version of Kestra is released I do the following:

  • stop systemd services (I have multiple services for each Kestra component)
  • download a fresh binary and update VMs
  • run the code above to re-install plugins
❯ s3cmd  -c s3.cfg ls s3://kestra//test/_files/
                          DIR  s3://kestra//test/_files/test/
                          DIR  s3://kestra//test/_files/test2/
2024-06-19 22:39       DIROBJ  s3://kestra//test/_files/
2024-06-19 18:25        25141  s3://kestra//test/_files/WB.log
2024-06-19 22:39           20  s3://kestra//test/_files/test.py
2024-06-19 18:26           11  s3://kestra//test/_files/test.txt

All of these files/folders were created via UI but are missing in the editor

from kestra.

aku avatar aku commented on July 23, 2024

Aside from that can you show your network tab (in your browser) when opening the editor ? Maybe export it in HAR format and provide it here. At least provide the /namespaces/test/files call output

As I've mentioned in Slack there are no errors in dev console besides Chrome warnings on 3rd party cookies

I've recorded a video and a corresponding HAR file

file_creation.mov

kestra.requests.har.zip

from kestra.

aku avatar aku commented on July 23, 2024

Since you mentioned plugins, I will try to manually delete and re-install s3 plugin. Maybe it will help

UPD:

I've logged into VMs, removed s3 plugins manually (there were 16.x and 17.0 versions) and reinstalled the latest version. Then I've restarted kestra. Unfortunately, it did not help. Here is a list of plugins in the plugins directory:

plugin-docker-0.16.1.jar
plugin-docker-0.17.0.jar
plugin-elasticsearch-0.16.0.jar
plugin-elasticsearch-0.17.0.jar
plugin-fs-0.16.0.jar
plugin-fs-0.16.1.jar
plugin-fs-0.17.0.jar
plugin-git-0.16.1.jar
plugin-git-0.17.0.jar
plugin-jdbc-clickhouse-0.16.1.jar
plugin-jdbc-clickhouse-0.17.0.jar
plugin-jdbc-duckdb-0.16.1.jar
plugin-jdbc-duckdb-0.17.0.jar
plugin-jdbc-mysql-0.16.1.jar
plugin-jdbc-mysql-0.17.0.jar
plugin-jdbc-postgres-0.16.1.jar
plugin-jdbc-postgres-0.17.0.jar
plugin-kafka-0.16.0.jar
plugin-kafka-0.17.0.jar
plugin-malloy-0.16.1.jar
plugin-malloy-0.17.0.jar
plugin-notifications-0.16.0.jar
plugin-notifications-0.17.0.jar
plugin-redis-0.16.0.jar
plugin-redis-0.17.0.jar
plugin-script-groovy-0.16.1.jar
plugin-script-groovy-0.17.0.jar
plugin-script-julia-0.16.1.jar
plugin-script-julia-0.17.0.jar
plugin-script-jython-0.16.1.jar
plugin-script-jython-0.17.0.jar
plugin-script-nashorn-0.16.1.jar
plugin-script-nashorn-0.17.0.jar
plugin-script-node-0.16.1.jar
plugin-script-node-0.17.0.jar
plugin-script-python-0.16.1.jar
plugin-script-python-0.17.0.jar
plugin-script-r-0.16.1.jar
plugin-script-r-0.17.0.jar
plugin-script-shell-0.16.1.jar
plugin-script-shell-0.17.0.jar
plugin-serdes-0.16.0.jar
plugin-serdes-0.17.1.jar
plugins.txt
storage-minio-0.16.1.jar
storage-minio-0.17.0.jar
storage-s3-0.17.0.jar```

from kestra.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.