Pagination with next page token about data

[kamarton]

@samdark I'm creating a pull request for what I mean

[kamarton]

Additional info

Page token require encode before publish, and decode if request:

• has optionally sensitive data
• may even be an attack point
• too long
• binary data

[samdark]

Is it the same as in #1?

[kamarton]

@samdark yes

[kamarton]

Additional info

Very important: The requested elements number does not always match the response elements number.

situation	next page token	description
has next page, request count == response count	✔️	result is full page
not has next page, request count >= response count (include zero!)✝	❌	end of list
has next page, request count >= response count (include zero!)✝	✔️	the result does not contain a full page

✝: Too long scanning (many records) are needed, therefore limit the running time.

[samdark]

Leaving that issue then and closing this one.

[kamarton]

I reviewed code of #1 and thought it over.

The TokenPaginator is a token pager that waits for a token string as input. The token string can generate all the data needed to continue paging, however complex it may be.

This is an opportunity to scroll API endpoints that work with a page token, example:

• custom rest or any API with yii2
• mostly cloud based APIs (Google Cloud, Amazon, Facebook, and more)

Suggest

class TokenPaginator {
  protected $token;
  __constructor...
  public function withPageSize(int ): self;
  public function withToken(String token): self;
  public function read(): iterator {
    
  }
  public function nextToken(): string;

  protected function encodeTokenData(object $data): string {
    return (string) $data;
  }

  protected function decodeTokenData(string $token): object {
    return $token;
  }
}

$dataReader = new ArrayDataReader($this->getDataSet());
// first page
$paginator = new TokenPaginator($dataReader);
$nextPageToken = $paginator->nextToken();

// second page
$paginator = new TokenPaginator($dataReader);
$paginator->withToken($nextPageToken);
$nextPageToken = $paginator->nextToken();

...

Example 1: offset paginator, with public data

class OffsetWothTokenPaginator extends TokenPaginator {
  protected $offsetPaginator;
  public function read() {
    if($pageToken !== null) {
      $parameters = decodeTokenData($pageToken);
      $offsetPaginator->withPageSize($parameters['pageSize'])->withCurrentPage($parameters['pageNumber']);
    }

    return $offsetPaginator->read();
  }
  
  protected function encodeTokenData(object $data): string {
    return join('|', [$data['pageNumber'],$data['pageSize']]);
  }
  protected function decodeTokenData(string $token): object {
    $parts = explode('|', $token);
    return [
      'pageNumber' => $parts[0],
      'pageSize' => $parts[1],
    ]
  }
}


// first page
$paginator = new TokenPaginator($dataReader);
$paginator->withPageSize(100);
$nextPageToken = $paginator->nextToken();  // "1|100"

// second page
$paginator = new TokenPaginator($dataReader);
$paginator->withToken($nextPageToken);
$nextPageToken = $paginator->nextToken();  // "2|100"

Example 2: token with random string

class OffsetWithRandomTokenPaginator extends TokenPaginator {
  protected $offsetPaginator;
  public function read() {
    if($pageToken !== null) {
      $parameters = decodeTokenData($pageToken);
      $offsetPaginator->withPageSize($parameters['pageSize'])->withCurrentPage($parameters['pageNumber']);
    }

    return $offsetPaginator->read();
  }

  protected function encodeTokenData(object $data): string {
    $token = randomString(32);
    $cacheProvider->set($token, $data);
    return $token;
  }

  protected function decodeTokenData(string $token): object {
     $data = $cacheProvider->get($token);
     if(!$data) throw ...
     return $data;
  }
}


// first page
$paginator = new OffsetWithRandomTokenPaginator($dataReader);
$paginator->withPageSize(100);
$nextPageToken = $paginator->nextToken();  // "sdklfsefiwe98fhwegoeighwgowekaslas"

// second page
$paginator = new OffsetWithRandomTokenPaginator($dataReader);
$paginator->withToken($nextPageToken);
$nextPageToken = $paginator->nextToken();  // "weklfneiowj2oifjwöefaiefhweiohqwoeif"

[samdark]

@kamarton in case of offset paginator we need ability to get to any numbered page directly. How would you do that using the API above?

[kamarton]

The example is not the lucky one, first I wanted the examples with KeysetPaginator but I didn't know how much the class would change.
Regardless, the answer is, no way, because I don't want users to page back and forth. For example, the facebook post feed has only scroll down, without paging.

The reasons for using page token may be:

• there is no point in turning back
• it makes no sense to jump between pages
• the data required to generate the next page is too large.
• It cannot be returned from the user without further verification.
• shorter url
• It is not technically allowed to change eg. the filter conditions, so there is no point sending the user back and forth.
• The first pages have an aggressive cache, so I don't want users to jump to the last page out of curiosity

More examples in list view + frontend + load more or scroll down paging context.

page token based API-s

https://cloud.google.com/storage/docs/json_api/v1/objects/list#response has only nextPageToken, and paging is limited to next page.

To do this on the frontend, I need a pager that only scrolls to the next page and expects only the next token as a parameter. The displayed pager has one link: 'load more', or nothing, if auto scroll down paging (javascript).

using with KeysetPaginator

This pagination require lastValue. Suppose that the values of lastValue:

• has sensitive data
• too long (in case of Bigtable can be up N*100-1000 bytes in normally)

Because of the above, I can't put in to URL or HTTP body.

paginator comparsion

condition	OffsetPaginator	KeysetPaginator	TokenPaginator
support dynamic page size	no	yes	yes
bounce between pages	yes	partially	no
short URL with filters	no	no	yes
Filter conditions can be edited by the user (after first page)	yes	yes	no
Can be used with other paginators	no	no	yes
supports global uniqueness	no	no	yes
support TTL	no	no	yes
support new record without missing items	partilly	yes	yes
With the exception of the last page, it is always a fixed page size	yes	no	no

[kamarton]

Suggest

this paginator required TokenableDataInterface. I'm not sure English is the correct tokenable word.

interface TokenableDataInterface
{
   public function withNextToken(string $token);
   public function withPreviousToken(string $token);
}

[samdark]

1. OffsetPaginator supports dynamic page size.
2. "With the exception of the last page, it is always a fixed page size" is true for KeysetPaginator.

The only difference between KeysetPaginator and TokenPaginator seems to be that next page token is encoded in case of TokenPaginator. Is that true?

[kamarton]

OffsetPaginator supports dynamic page size.

This is fails. offset = pageSize * page If you resize the page size on page 2, the items will slide completely.

"With the exception of the last page, it is always a fixed page size" is true for KeysetPaginator.

This is not true. Each page gives you the option to set a different page size. This is a scanner solution where you do not need to have all the pages the same size, especially if I disable reverse paging.
For example, on the landing page, I only operate with one screen page size because most users no longer scroll. But if someone goes on, I'll load a larger amount of items.

The only difference between KeysetPaginator and TokenPaginator seems to be that next page token is encoded in case of TokenPaginator. Is that true?

No.

• TTL
• Can be used with other paginators
• Filter conditions can be edited by the user (after first page) - not supported by token paginator (possibly partially supported by creating a new paginator.)

[kamarton]

The only difference between KeysetPaginator and TokenPaginator seems to be that next page token is encoded in case of TokenPaginator. Is that true?

If I can see clearly in the code, the value of KeysetPaginator#lastValue is limited to a scalar value.
The TokenPaginator can operate on any complex set of data.

[samdark]

What is TTL in a paginator?

How token paginator can be used with other paginators?

The TokenPaginator can operate on any complex set of data.

Do you mean the data is encrypted with a secret key into a string and then decrypted?

[kamarton]

What is TTL in a paginator?

As long as the page itself can be used.

How token paginator can be used with other paginators?

The token paginator can also be used by itself, e.g. if the data source itself is expecting a token. If the source is not capable of handling the token, call the other paginators for help. The task is to convert the other paging data into a token.

class TokenPaginator {
   public $paginator;    // optionally sub paginator config
}

Do you mean the data is encrypted with a secret key into a string and then decrypted?

No. The Token provider is not responsible for the storage process, this process is freely defined. example

• store in database record columns
• or passes a class, which is then stored by the storage procedure (eg cache) and returned to its original form when recreated.

[samdark]

As long as the page itself can be used.

When would it expire?

or passes a class, which is then stored by the storage procedure (eg cache) and returned to its original form when recreated.

Do you mean token and parameters corresponding to it should be stored?

[kamarton]

When would it expire?

Expiration date depends on storage and developer. For example, we use the cache component, then Yii::$app->cache->set($token, $data, $duration). If the storage has expired, the token data cannot be loaded, which will also the page unusable.

It is advisable to create classes for the basic storage methods for better usability, eg. for a cache component.

Do you mean token and parameters corresponding to it should be stored?

Yes. The token must be able to get the parameters by itself or in with a modified process (for example, a cut prefix, remove CRC).

[kamarton]

When would it expire?

Of course, this requires that the storage process can handle TTL. If you cannot handle TTL, then the token paginator will not be able to handle it either.