ScanCommand with no Limit set only ever reads one page of data about aws-sdk-js-v3

[aBurmeseDev]

Hi @mn-prp - thanks for reaching out.

I'm not able to reproduce it on my end and wanted to verify a few things with you. The expected behavior is that Limit is used for non-pagination requests whereas pageSize can be used when you want to specify results per page.

If you'd like to limit the total returned results, here's what I'd do:

const paginator = paginateScan(
        {
            client: DDBClient,
            pageSize: 1,
        },
        {
            TableName: tableName,
            FilterExpression: 'begins_with(sk, :sk)',
            ExpressionAttributeValues: {
                       ':sk': sk
           }          
        }
    );
    const LIMIT = 2;
    let count = 0;

    for await (const page of paginator) {
        if (++count >= LIMIT) {
            break;
        }
    }
}

Hope that makes sense but let me know if you have any further questions.
Best,
John

[mn-prp]

Thanks for looking into this. My use case is to retrieve all items, without any (final) pagination, since the goal of the query is to sum some values across the whole table for an internal analytics job that is run periodically.

I don't have the time to try to create a reproduction, since it obviously depends on a populated database, but I can comment on the context a bit. We have two environments, one staging and more production, and the staging environment has a table ~1mb in size, whereas production is ~5mb. The "total" values in both tables were known to be increasing, but whereas the staging environment did continue to increase, we noticed that the total of the production database query was stagnating at around some value (let's say 200,000 -- we were seeing totals at 210,000 then 209,000 then 197,000 then 201,000 etc.) This led me to think there was a page size that was being hit.

So I went and looked at the sdk call, which was the first snippet copied in my issue. By adding the Limit: 1000 I got the expected much-higher total. Then I tried Limit: 50 and again, the expected much higher value (but with many more network requests, as my logging showed). But when I removed the line, the value dropped back down to the ~200,000 and my logging revealed only one loop was being executed, with no LastEvaluatedKey.

Is it expected that Limit is required to read the full data set read from a multi-megabyte table? I would have expected that not setting Limit would either (a) retrieve the max page size and return a LastEvaluatedKey or (b) make the sdk continue fetching the next page automatically (and appending it to the result set so far) until there were no more pages.