DAG triggering without all Datasets being triggered about airflow

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[slords]

[potiuk]

I think you should realise that we have absolutely no idea about your setup and screenshots are not helping to understand that, so people here can't even know what is wrong. People here will simply miss context.

The only way it can be acted upon is if there are correlated logs - i.e. find a concrete DAGs that has been triggered wrongly. You can set debug log level on the whole airflow (but I have no idea how to do in MWAA - likely there should be docs) https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#logging-level and you can get scheduler logs (again no idea how to do it in MWAA) - correlated with the DAG that gets triggered wrongly - showing why it has been scheduled and explaining the context. Explaining what has been triggered, why you think it should not be and showing related log entries.

You have to remember - that here people here when they think they can help - in their free time - and it's up to you to provide detailed information (that they don't have), so such a vague issue does not help those people (who often do it at expense of their free time) does not help them - simply this is not a helpdesk but a forum where people help each other if they can.

But also remember that MWAA is a service you pay for and that they can likely access all the information that people here can't - so if you won't be able to extract and provide the information with more details, then you pay for the service and I guess they should be able to investigate more - with all the access and possibilities to look at your problem.

[slords]

The issue is that DAGs are triggering even though not all of the datasets have been updated. Sometimes the DAGs trigger successfully, but other times they don't. This makes it really hard to figure out what's wrong.

To fix this, we need to find the logs that show why the DAG was scheduled even though not all the datasets had updates. MWAA doesn't let you set the debug flag on any log, only INFO. I'm working with AWS to see if there is an internal way to turn on DEBUG so we can get to these logs.

You don't need to know about my setup to see what the problem is. I've attached some screenshots that clearly show which datasets are required and that, on two of the five occasions that the DAG ran, it didn't have all the required datasets available.

I understand how open source works and I know that most people won't raise bugs. But, I've been contributing to open source projects for years, and I firmly believe that any issue I can help fix makes the product better for everyone. Filing a bug report is a good way to get an issue identified and hopefully resolved. It's also possible that others are experiencing the same problem but haven't said anything yet. If they see this bug report, they can confirm the issue and maybe even provide some additional information that will help us find a solution.

I was disappointed by the unhelpful and dismissive attitude that came across in your response. Not everyone has the same level of technical expertise or resources, and it can be frustrating to be made to feel like a nuisance. Just because I don't have all the answers doesn't mean my bug report isn't valid. I'm providing as much information and context as I can, and I'm happy to cooperate with anyone who wants to help troubleshoot the issue.