[BUG] Opensearch stack is stuck in DELETE_FAILED status about autonomous-driving-data-framework

[dgraeber]

@manojrajpurohit
It appears that the Security Group tied to the Opensearch Deployment has a reference to some entity that has not been removed from it yet(?). Can you please go into the account, find the referred to SG and see what resources are using it by trying to manually deleting the SG. Once you are able to delete the SG, rerun seedfarmer apply

[dgraeber]

@manojrajpurohit Also - check if the stack has the Opensearch proxy has been deployed and not destroyed. I would like to know how the customer got here (i.e. they had a deployment and wanted to swap out OpenSearch, they were trying to delete the entire deployment, etc).

[manojrajpurohit]

Indeed the security group was used in Network Interface. I guess these Network Interface were created as part of modules/demo-only/opensearch-proxy/
So I did the following

1. commented manifests/ros-image-demo/integration-modules.yaml in deployment.yaml and executed seedfarmer, thus, modules/demo-only/opensearch-proxy/ and modules/integration/ddb-to-opensearch/ stacks were deleted
2. deleted opensearch stack from CloudFormation console. Was able to delete it this time.
3. Commented opensearch in manifest and rerun seedfarmer - deployment failed (I thought this was required to put seedfarmer in right state since I had deleted opensearch from console)
4. Included opensearch , modules/demo-only/opensearch-proxy/ and modules/integration/ddb-to-opensearch/ in manifest and rerun seedfarmer - all 3 stacks are successfully created

so I was somehow able to fix this issue.

However, I think, the point to remember is after deploying opensearch, opensearch proxy and ddb-to-opensearch, we should not try to remove opensearch alone , rather all 3 should be removed. I am not sure if this understanding is correct. But if this is correct then I am wondering whether there is a way to hard specify dependency check between stacks.

Anyways closing closing this bug now.

[dgraeber]

Yes...it sounds like you deleted a module that had other modules dependent on it. Currently, seedfarmer does not manage dependencies, so it will let you delete a module even if other modules are referencing it. For now, rule of thumb is to verify module references before removing them from a manifest, and we are exploring ways to mitigate the event you incurred.