BUG: All failed jobs are being restarted about bacalhau

[rossjones]

I've been able to replicate this on the update_docker_executor_for_OOM branch.

I run devstack, and then execute ./bin/darwin/arm64/bacalhau docker run ubuntu tail /dev/zero and it fails, but then retries and continues to do so forever.

When I run bacalhau describe [JOBID] I see all the executions failing but the job progress is still InProgress.

- ComputeReference: e-911c24d4-c086-4658-bd45-76980d8a432c
    CreateTime: "2024-01-10T14:59:55.873367Z"
    DesiredState: 2
    JobID: 2bccf347-eee7-4c57-8c7e-1a45688ec2be
    NodeId: QmdLcbeFi6xEZC3GfzTAAvBdJjReDJXiqksAR1KYKEJPnA
    PublishedResults: {}
    State: Failed
    Status: 'execution error: memory limit exceeded. execution failed'
    UpdateTime: "2024-01-10T14:59:58.344354Z"
    Version: 6
  - ComputeReference: e-911f07ed-0e9f-43ea-97fa-03754407e36b
    CreateTime: "2024-01-10T14:59:52.885959Z"
    DesiredState: 2
    JobID: 2bccf347-eee7-4c57-8c7e-1a45688ec2be
    NodeId: QmQVziCr88FNoxpHx467skzp8HcLSybbqekrqZuRA2fV9Q
    PublishedResults: {}
    State: Failed
    Status: 'execution error: memory limit exceeded. execution failed'
    UpdateTime: "2024-01-10T14:59:55.858067Z"
    Version: 6
  - ComputeReference: e-c20503ef-3580-490f-a39b-d72c14052fdc
    CreateTime: "2024-01-10T14:59:58.362537Z"
    DesiredState: 2
    JobID: 2bccf347-eee7-4c57-8c7e-1a45688ec2be
    NodeId: QmQVziCr88FNoxpHx467skzp8HcLSybbqekrqZuRA2fV9Q
    PublishedResults: {}
    State: Failed
    Status: 'execution error: memory limit exceeded. execution failed'
    UpdateTime: "2024-01-10T15:00:00.961569Z"
    Version: 6
  - ComputeReference: e-cf8183da-23ce-4f8a-959d-d62f05d76baf
    CreateTime: "2024-01-10T15:00:03.807177Z"
    DesiredState: 2
    JobID: 2bccf347-eee7-4c57-8c7e-1a45688ec2be
    NodeId: QmdLcbeFi6xEZC3GfzTAAvBdJjReDJXiqksAR1KYKEJPnA
    PublishedResults: {}
    State: Failed
    Status: 'execution error: memory limit exceeded. execution failed'
    UpdateTime: "2024-01-10T15:00:06.173217Z"
    Version: 6
  - ComputeReference: e-d5030644-3ffd-440f-9f87-a18ca982bb20
    CreateTime: "2024-01-10T15:00:00.979565Z"
    DesiredState: 2
    JobID: 2bccf347-eee7-4c57-8c7e-1a45688ec2be
    NodeId: QmTmjRMf9D4riszGXBjUyJ15rqFkf5wnPYCg7CVKcjDGdD
    PublishedResults: {}
    State: Failed
    Status: 'execution error: memory limit exceeded. execution failed'
    UpdateTime: "2024-01-10T15:00:03.79186Z"
    Version: 6