Streams returning HTTP/500 about opencti

[nino-filigran]

@MaxwellDPS I see that you did not indicate any steps to reproduce: is it because you don't know how to reproduce?

[MaxwellDPS]

Not sure if its related but also seeing this around the same time

{
    "category": "APP",
    "error": {
        "context": {},
        "message": "Cannot read properties of undefined (reading 'length')",
        "name": "TypeError",
        "stack": "TypeError: Cannot read properties of undefined (reading 'length')\n    at taxiiV21DataHandler (/opt/opencti/build/src/manager/ingestionManager.ts:233:20)\n    at processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async Promise.all (index 0)\n    at async Promise.all (index 1)\n    at ingestionHandler (/opt/opencti/build/src/manager/ingestionManager.ts:288:5)\n    at /opt/opencti/build/src/manager/ingestionManager.ts:307:9\n    at pyt.#runHandlerAndScheduleTimeout (/opt/opencti/build/node_modules/set-interval-async/dist/set-interval-async-timer.cjs:36:13)\n    at Timeout._onTimeout (/opt/opencti/build/node_modules/set-interval-async/dist/set-interval-async-timer.cjs:29:13)"
    },
    "level": "error",
    "message": "[OPENCTI-MODULE] Taxii ingestion execution error for <REMOVED>",
    "timestamp": "2024-01-19T00:42:21.112Z",
    "version": "5.12.4"
}

[richard-julien]

"ERR no such key" is an error that occurs when the stream is deleted from the redis. This situation must not happens as flushing directly the redis is not an authorized operation.

"Cannot read properties of undefined (reading 'length')", i think its already fixed in newest version of the platform

[MaxwellDPS]

Hey @richard-julien & @nino-filigran thanks for the reply!

@nino-filigran Yeah, there seems to be an issue with the datastore on this instance (#4936) But this was a unexpected Loss in cabin pressure scenario, so not sure a great way to recreate.

@richard-julien On the Redis issue, this has largely resolved itsself after ~50m. To clarify here, redis was flushed following the initial stall.

Sequence of events

1. Stream Sync RMQ rate drops [Remote Not the problem instance]
2. On the remote instance all the streams returned HTTP/500s
3. Once I saw that + the Redis log I shut down the platform pods
4. I then flushed Redis and issued a restart of the Redis service
5. Once Redis was healthy I rescaled the platform pods and kubernetes rescheduled the workers
6. At this point the stream was still broken

[nino-filigran]

@MaxwellDPS Do you still have an issue with this?

[Jipegien]

closing the issue due to no news. Do not hesitate to reopen the issue if necessary.