Worker can exhaust system storage about kestra

[loicmathieu]

Temporary files should be deleted after use, but if you have a trigger that creates a file that, in a single trigger evaluation, would read data that cannot be stored in the local filesystem this can happen.

Can you tell us more on your use case, the version of Kestra, its configuration and the flow that causes the issue?

[aku]

In my case I have a simple workflow that reads data from a kafka (topic produces thousands of records per second), transforms them via Jython FileTransform and sends to another kafka instance. The problem happens even if I just want to log a messageCount property of the trigger. If I do not use maxRecords property, they system gets overloaded easily.

I'm using Kestra 0.17.1 running on several VMs (webapp + scheduler running on 1 VM + 3 VMs with executor+worker processes)

triggers:
  - id: read_orders_from_kafka
    type: io.kestra.plugin.kafka.Trigger
    keyDeserializer: STRING
    valueDeserializer: JSON
    interval: PT5S
    topic: [REDACTED]
    groupId: [REDACTED]
    maxRecords: 5000
    properties:      
      [REDACTED]