get more than 50 items from `describe_log_streams` about aws-sdk-pandas

[jaidisido]

The wr.cloudwatch.describe_log_streams API already implements next_token and can return all items. 50 is the limit for each pagination call, not the overall limit. It's enforced by the underlying boto3 API (see Limit in the link you shared).

[AlJohri]

Thanks @jaidisido! I realize my confusion is with the word limit which I didn't interpret to be a page size.

I have a log group with hundreds of thousands of log streams. How do I get the latest 1000 log streams ordered by LastEventTime? I would have expected the limit parameter to allow me to stop the pagination after 1000 events collected.

wr.cloudwatch.describe_log_streams(
    log_group_name="/my/log/group",
    order_by='LastEventTime',
    descending=True,
)

[jaidisido]

I see. We are limited by what the underlying boto3 calls exposes I am afraid. The read_logs API might give you more flexibility in the query perhaps.

[AlJohri]

I think the boto3 API exposes enough nobs to achieve this behavior. Instead of running while "nextToken" in response, we can client side stop paginating requests if some condition is met such as a client side limit or a client side LastEventTime

if this makes sense to you, I can re-open this as a feature request for adding a client side filtering / stop pagination parameter in wr.cloudwatch.describe_log_streams for large log groups