Saved files disappear from the editor when using S3 storage about kestra

[brian-mulier-p]

Hey ! I'll try to reproduce it later in the day. Can you try the same thing with a local storage (if not I'll reproduce it with S3 but it would help identifying whether it's a UI issue or backend one) ?

[brian-mulier-p]

Thank's for providing effort with local storage, I'm setting up the reproducer 👍

[brian-mulier-p]

For traceability:
the storage-minio was used directly against an S3 bucket (and not a minio one) which led to some issues among which the fact that in the storage-minio plugin we remove any leading slash when querying due to limitations from minio which in the end doesn't work properly with S3.

The right way is to use the storage-s3 plugin but in this case it was causing an issue as the DNS is not configured to reach virtual-host-style S3 URL. The behaviour of the S3 SDK is to detect the domain resolution styling by checking if the url contains any dot (mostly IP) prior to path. If it contains some it will default to path-styling else it goes to virtual-host-styling.

In the case above it's not working as we have a strong DNS limitation and the allowed styling is a fixed one that doesn't go with the above domain resolution rule.

To solve that I introduced a new forcePathStyle configuration for storage-s3 plugin. Putting it to true should solve the issue. Feel free to open again if it doesn't work.

Moreover I enhanced the doc on storages to include all properties currently available

[anna-geller]

more context https://kestra-io.slack.com/archives/C03FEC452NQ/p1718812771627879

[aku]

I've found the missing files in S3 storage:

2024-06-19 15:55            3  s3://kestra//test/_files/test.py
2024-06-19 18:26           11  s3://kestra//test/_files/test.txt

It looks like that the bug is in UI

[aku]

I've created a simple workflow that tries to read the file:

As you might notice the file is missing from UI, however the flow is able to access it

If I try to create a file with the same name the system will overwrite the existing file (but the file is still not visible in UI)

[aku]

Another observation is that the files created via UI are missing from the exported files archive. It contains only yaml flows definitions.

[aku]

@brian-mulier-p I've tested this case with a dockerized kestra It does not reproduce with a local storage.

docker run --pull=always --rm -it -p 8080:8080 --user=root \
 -v /var/run/docker.sock:/var/run/docker.sock \
 -v /tmp:/tmp kestra/kestra:latest-full server local

[aku]

Also, I've checked export functionality and it was working correctly with a local dockerized kestra - test files were in the downloaded archive

[aku]

Thank's for providing effort with local storage, I'm setting up the reproducer 👍

To give more context:
I've started with Kestra 16.xx (i don't remember minor version but it was the last release before 17)
I've gradually upgraded my installation from 17.1 to 17.4 (updated binaries and re-installed plugins, I'm running Kestra in VMs)

I.e. if you will fail to reproduce it in 17.4 with a fresh install, it might be something related to migration from a previous versions. In this case I'm willing to do a local debugging/analysis if you provide me with ideas/instruction. Feel free to tag me in Slack

A file will be overwritten in S3 if I create a new one with the same name - it might be another/related bug.

[brian-mulier-p]

So yes I didn't manage to reproduce the issue. Dumb question but are you installing plugins on your own ? If so make sure your storage-s3 plugin is the 0.17.0 one.
Also can you show the listing content of your S3 bucket under /test/_files ? If it doesn't exist basically under /test

[brian-mulier-p]

Aside from that can you show your network tab (in your browser) when opening the editor ? Maybe export it in HAR format and provide it here. At least provide the /namespaces/test/files call output

[aku]

So yes I didn't manage to reproduce the issue. Dumb question but are you installing plugins on your own ? If so make sure your storage-s3 plugin is the 0.17.0 one. Also can you show the listing content of your S3 bucket under /test/_files ? If it doesn't exist basically under /test

I have ansible playbook that installs plugins:

    - name: Install kestra plugins # noqa command-instead-of-shell partial-become
      ansible.builtin.shell:
        cmd: '"{{ kestra_dir }}/kestra" plugins install --plugins="{{ kestra_dir }}/plugins"  "{{ item }}"'
      loop:
        - io.kestra.plugin:plugin-git:LATEST
        - io.kestra.plugin:plugin-fs:LATEST
        - io.kestra.plugin:plugin-elasticsearch:LATEST
        - io.kestra.plugin:plugin-kafka:LATEST
        - io.kestra.plugin:plugin-docker:LATEST
        - io.kestra.plugin:plugin-redis:LATEST
        - io.kestra.plugin:plugin-jdbc-postgres:LATEST
        - io.kestra.plugin:plugin-jdbc-clickhouse:LATEST
        - io.kestra.plugin:plugin-script-node:LATEST
        - io.kestra.plugin:plugin-script-python:LATEST
        - io.kestra.plugin:plugin-script-shell:LATEST
        - io.kestra.plugin:plugin-notifications:LATEST
        - io.kestra.plugin:plugin-serdes:LATEST
        - io.kestra.storage:storage-minio:LATEST
        - io.kestra.storage:storage-s3:LATEST
      become_user: "{{ kestra_sys_user }}"
      changed_when: false
      tags:
        - plugins

When a new version of Kestra is released I do the following:

• stop systemd services (I have multiple services for each Kestra component)
• download a fresh binary and update VMs
• run the code above to re-install plugins

❯ s3cmd  -c s3.cfg ls s3://kestra//test/_files/
                          DIR  s3://kestra//test/_files/test/
                          DIR  s3://kestra//test/_files/test2/
2024-06-19 22:39       DIROBJ  s3://kestra//test/_files/
2024-06-19 18:25        25141  s3://kestra//test/_files/WB.log
2024-06-19 22:39           20  s3://kestra//test/_files/test.py
2024-06-19 18:26           11  s3://kestra//test/_files/test.txt

All of these files/folders were created via UI but are missing in the editor

[aku]

Aside from that can you show your network tab (in your browser) when opening the editor ? Maybe export it in HAR format and provide it here. At least provide the /namespaces/test/files call output

As I've mentioned in Slack there are no errors in dev console besides Chrome warnings on 3rd party cookies

I've recorded a video and a corresponding HAR file

file_creation.mov

kestra.requests.har.zip

[aku]

Since you mentioned plugins, I will try to manually delete and re-install s3 plugin. Maybe it will help

UPD:

I've logged into VMs, removed s3 plugins manually (there were 16.x and 17.0 versions) and reinstalled the latest version. Then I've restarted kestra. Unfortunately, it did not help. Here is a list of plugins in the plugins directory:

plugin-docker-0.16.1.jar
plugin-docker-0.17.0.jar
plugin-elasticsearch-0.16.0.jar
plugin-elasticsearch-0.17.0.jar
plugin-fs-0.16.0.jar
plugin-fs-0.16.1.jar
plugin-fs-0.17.0.jar
plugin-git-0.16.1.jar
plugin-git-0.17.0.jar
plugin-jdbc-clickhouse-0.16.1.jar
plugin-jdbc-clickhouse-0.17.0.jar
plugin-jdbc-duckdb-0.16.1.jar
plugin-jdbc-duckdb-0.17.0.jar
plugin-jdbc-mysql-0.16.1.jar
plugin-jdbc-mysql-0.17.0.jar
plugin-jdbc-postgres-0.16.1.jar
plugin-jdbc-postgres-0.17.0.jar
plugin-kafka-0.16.0.jar
plugin-kafka-0.17.0.jar
plugin-malloy-0.16.1.jar
plugin-malloy-0.17.0.jar
plugin-notifications-0.16.0.jar
plugin-notifications-0.17.0.jar
plugin-redis-0.16.0.jar
plugin-redis-0.17.0.jar
plugin-script-groovy-0.16.1.jar
plugin-script-groovy-0.17.0.jar
plugin-script-julia-0.16.1.jar
plugin-script-julia-0.17.0.jar
plugin-script-jython-0.16.1.jar
plugin-script-jython-0.17.0.jar
plugin-script-nashorn-0.16.1.jar
plugin-script-nashorn-0.17.0.jar
plugin-script-node-0.16.1.jar
plugin-script-node-0.17.0.jar
plugin-script-python-0.16.1.jar
plugin-script-python-0.17.0.jar
plugin-script-r-0.16.1.jar
plugin-script-r-0.17.0.jar
plugin-script-shell-0.16.1.jar
plugin-script-shell-0.17.0.jar
plugin-serdes-0.16.0.jar
plugin-serdes-0.17.1.jar
plugins.txt
storage-minio-0.16.1.jar
storage-minio-0.17.0.jar
storage-s3-0.17.0.jar```