user_table not returning all results about steampipe-plugin-jira

[rmaalouf]

Does this approach make sense:

• If retry/backoff limit is reached, user should be made aware that 429 was encountered
• Query should still succeed as is currently the case, but group_names should be left joined instead of inner joined so that the query still returns the full list of users.

[rmaalouf]

Hi @misraved, thank you for taking the time to look into this.
Good news, it appears this PR fixes the issue.

I tested it by first reproducing the issue and then I rebuilt the plugin using this PR and ran the following 10 times in a row:

select * from jira_user
.cache clear

The query returned all users (~700) every time.

[misraved]

Thanks for raising the bug @rmaalouf. Sorry to see that you are bumping into issues.

Could you please share the plugin-level logs when you run the query select * from jira_user?

[e-gineer]

Hey @rmaalouf ... Steampipe does sub-API calls for some columns to get their data. In this case, the problem is the group_names column calling a different API for each row (too quickly).

Notice the Hydrate function in this code:

https://github.com/turbot/steampipe-plugin-jira/blob/main/jira/table_jira_user.go#L69-L75

As a workaround, does this work? (Avoiding the group_names column):

select account_id, display_name from jira_user

For a more permanent solution, I think we need to add retry / backoff to the getUserGroups function:

https://github.com/turbot/steampipe-plugin-jira/blob/main/jira/table_jira_user.go#L148-L164

[rmaalouf]

Thanks for the assistance. On previous days I was able to replicate this on every query. This morning select * from jira_user worked several times in a row. However after a few clear/query cycles I was able to replicate the issue (results are incomplete but no error is returned). Here are the logs:

2022-10-31 13:39:42.466 UTC [ERROR] steampipe-plugin-jira.plugin: [ERROR] 1667223580111: jira_user.getUserGroups: api_error="429 Too Many Requests: local_rate_limited: request failed. Please analyze the request body for more details. Status code: 429"
2022-10-31 13:39:42.466 UTC [ERROR] steampipe-plugin-jira.plugin: [ERROR] 1667223580111: jira_user.getUserGroups: api_error="429 Too Many Requests: local_rate_limited: request failed. Please analyze the request body for more details. Status code: 429"
2022-10-31 13:39:42.466 UTC [ERROR] steampipe-plugin-jira.plugin: [ERROR] 1667223580111: callHydrate getUserGroups finished with error: 429
Too Many Requests: local_rate_limited: request failed. Please analyze the request body for more details. Status code: 429
2022-10-31 13:39:42.466 UTC [ERROR] steampipe-plugin-jira.plugin: [ERROR] 1667223580111: callHydrate getUserGroups finished with error: 429 Too Many Requests: local_rate_limited: request failed. Please analyze the request body for more details. Status code: 429
2022-10-31 13:39:42.466 UTC [ERROR] steampipe-plugin-jira.plugin: [ERROR] 1667223580111: table 'jira_user' failed to get column data, connectionCallId jira-1667223580111: table 'jira_user' column 'group_names' requires hydrate data from getUserGroups, which failed with error 429 Too Many Requests: local_rate_limited: request failed. Please analyze the request body for more details. Status code: 429.
2022-10-31 13:39:42.466 UTC [WARN] steampipe-plugin-jira.plugin: [WARN] 1667223580111: QueryData StreamError table 'jira_user' column 'group_names' requires hydrate data from getUserGroups, which failed with error 429 Too Many Requests: local_rate_limited: request failed. Please analyze the request body for more details. Status code: 429.
2022-10-31 13:39:42.466 UTC [ERROR] steampipe-plugin-jira.plugin: [ERROR] 1667223580111: jira_user.getUserGroups: api_error="429 Too Many Requests: local_rate_limited: request failed. Please analyze the request body for more details. Status code: 429"
2022-10-31 13:39:42.466 UTC [ERROR] steampipe-plugin-jira.plugin: [ERROR] 1667223580111: jira_user.getUserGroups: api_error="429 Too Many Requests: local_rate_limited: request failed. Please analyze the request body for more details. Status code: 429"
2022-10-31 13:39:42.466 UTC [WARN] steampipe-plugin-jira.plugin: [WARN] 1667223580111: hydrate error chan select error 429 Too Many Requests: local_rate_limited: request failed. Please analyze the request body for more details. Status code: 429
2022-10-31 13:39:42.466 UTC [WARN] steampipe-plugin-jira.plugin: [WARN] 1667223580111: getRow failed with error 429 Too Many Requests: local_rate_limited: request failed. Please analyze the request body for more details. Status code: 429
2022-10-31 13:39:42.466 UTC [WARN] steampipe-plugin-jira.plugin: [WARN] 1667223580111: QueryData StreamError 429 Too Many Requests: local_rate_limited: request failed. Please analyze the request body for more details. Status code: 429 (jira-1667223580111)
2022-10-31 13:39:42.466 UTC [WARN] steampipe-plugin-jira.plugin: [WARN] 1667223580111: hydrate error chan select error 429 Too Many Requests: local_rate_limited: request failed. Please analyze the request body for more details. Status code: 429
2022-10-31 13:39:42.466 UTC [WARN] steampipe-plugin-jira.plugin: [WARN] 1667223580111: getRow failed with error 429 Too Many Requests: local_rate_limited: request failed. Please analyze the request body for more details. Status code: 429
2022-10-31 13:39:42.466 UTC [WARN] steampipe-plugin-jira.plugin: [WARN] 1667223580111: QueryData StreamError 429 Too Many Requests: local_rate_limited: request failed. Please analyze the request body for more details. Status code: 429 (jira-1667223580111)
2022-10-31 13:39:42.466 UTC [ERROR] steampipe-plugin-jira.plugin: [ERROR] 1667223580111: callHydrate getUserGroups finished with error: 429 Too Many Requests: local_rate_limited: request failed. Please analyze the request body for more details. Status code: 429
2022-10-31 13:39:42.466 UTC [WARN] steampipe-plugin-jira.plugin: [WARN] 1667223580111: streamRows for jira-1667223580111 - execution has failed (table 'jira_user' column 'group_names' requires hydrate data from getUserGroups, which failed with error 429 Too Many Requests: local_rate_limited: request failed. Please analyze the request body for more details. Status code: 429.
2022-10-31 13:39:42.466 UTC [WARN] steampipe-plugin-jira.plugin: [WARN] 1667223580111: executeForConnection jira returned error table 'jira_user' column 'group_names' requires hydrate data from getUserGroups, which failed with error 429 Too Many Requests: local_rate_limited: request failed. Please analyze the request body for more details. Status code: 429.
2022-10-31 13:39:42.466 UTC [ERROR] steampipe-plugin-jira.plugin: [ERROR] 1667223580111: callHydrate getUserGroups finished with error: 429 Too Many Requests: local_rate_limited: request failed. Please analyze the request body for more details. Status code: 429
2022-10-31 13:39:42.466 UTC [ERROR] steampipe-plugin-jira.plugin: [ERROR] 1667223580111: table 'jira_user' failed to get column data, connectionCallId jira-1667223580111: table 'jira_user' column 'group_names' requires hydrate data from getUserGroups, which failed with error 429 Too Many Requests: local_rate_limited: request failed. Please analyze the request body for more details. Status code: 429.
2022-10-31 13:39:42.466 UTC [WARN] steampipe-plugin-jira.plugin: [WARN] 1667223580111: error channel received table 'jira_user' column 'group_names' requires hydrate data from getUserGroups, which failed with error 429 Too Many Requests: local_rate_limited: request failed. Please analyze the request body for more details. Status code: 429.
2022-10-31 13:39:42.466 UTC [WARN] steampipe-plugin-jira.plugin: [WARN] 1667223580111: QueryData StreamError table 'jira_user' column 'group_names' requires hydrate data from getUserGroups, which failed with error 429 Too Many Requests: local_rate_limited: request failed. Please analyze the request body for more details. Status code: 429.
2022-10-31 13:39:42.466 UTC [WARN] steampipe-plugin-jira.plugin: [WARN] 1667223580111: hydrate error chan select error 429 Too Many Requests: local_rate_limited: request failed. Please analyze the request body for more details. Status code: 429
2022-10-31 13:39:42.466 UTC [WARN] steampipe-plugin-jira.plugin: [WARN] 1667223580111: getRow failed with error 429 Too Many Requests: local_rate_limited: request failed. Please analyze the request body for more details. Status code: 429
2022-10-31 13:39:42.466 UTC [WARN] steampipe-plugin-jira.plugin: [WARN] 1667223580111: QueryData StreamError 429 Too Many Requests: local_rate_limited: request failed. Please analyze the request body for more details. Status code: 429 (jira-1667223580111)
2022-10-31 13:39:42.466 UTC [WARN] steampipe-plugin-jira.plugin: [WARN] 1667223580111: hydrate error chan select error 429 Too Many Requests: local_rate_limited: request failed. Please analyze the request body for more details. Status code: 429
2022-10-31 13:39:42.466 UTC [WARN] steampipe-plugin-jira.plugin: [WARN] 1667223580111: getRow failed with error 429 Too Many Requests: local_rate_limited: request failed. Please analyze the request body for more details. Status code: 429
2022-10-31 13:39:42.466 UTC [WARN] steampipe-plugin-jira.plugin: [WARN] 1667223580111: QueryData StreamError 429 Too Many Requests: local_rate_limited: request failed. Please analyze the request body for more details. Status code: 429 (jira-1667223580111)
2022-10-31 13:39:42.466 UTC [ERROR] steampipe-plugin-jira.plugin: [ERROR] 1667223580111: table 'jira_user' failed to get column data, connectionCallId jira-1667223580111: table 'jira_user' column 'group_names' requires hydrate data from getUserGroups, which failed with error 429 Too Many Requests: local_rate_limited: request failed. Please analyze the request body for more details. Status code: 429.
2022-10-31 13:39:42.466 UTC [WARN] steampipe-plugin-jira.plugin: [WARN] 1667223580111: QueryData StreamError table 'jira_user' column 'group_names' requires hydrate data from getUserGroups, which failed with error 429 Too Many Requests: local_rate_limited: request failed. Please analyze the request body for more details. Status code: 429.
2022-10-31 13:39:42.466 UTC [ERROR] steampipe-plugin-jira.plugin: [ERROR] 1667223580111: table 'jira_user' failed to get column data, connectionCallId jira-1667223580111: table 'jira_user' column 'group_names' requires hydrate data from getUserGroups, which failed with error 429 Too Many Requests: local_rate_limited: request failed. Please analyze the request body for more details. Status code: 429.
2022-10-31 13:39:42.466 UTC [WARN] steampipe-plugin-jira.plugin: [WARN] 1667223580111: QueryData StreamError table 'jira_user' column 'group_names' requires hydrate data from getUserGroups, which failed with error 429 Too Many Requests: local_rate_limited: request failed. Please analyze the request body for more details. Status code: 429.

[rmaalouf]

@e-gineer that appears to do the trick. After hitting the 429 issue several times in a row I avoided selecting the group_names column as you suggested and got the expected full number of records.

[misraved]

Hello @rmaalouf, thanks a lot for extensively testing out the table and providing valuable insights on the error logs 👍 . Apologies for the delay, but I was finally able to get a hold of this issue.

Initially, I was inclined towards reducing the MaxConcurrency parameter to 5, since it worked for you in the first instance, but I finally settled on adding a RetryConfig to the getUserGroups function.

I have created a PR - #72 which fixes the issue of API Rate limiting in the jira_user table. Could you please rebuild the plugin using this PR and let us know if it fixes your issue?