They are currently used in most places (mainly in paginators)

What steps will reproduce the problem?

Currently cannot implement own filter. For example, I want a regexp filter, or custom group filter with scoring system.

What is the expected result?

        $filter = new CustomRegexpFilter('name' /* field */, '!^a[g]?!' /* pattern */, 'ui' /* modifier*/);
        $reader = (new ArrayDataReader($this->getDataSet()))
            ->withFilter($filter);

        $filter = new CustomScoringGroupFilter(new Filter(), new Filter()...);
        $reader = (new ArrayDataReader($this->getDataSet()))
            ->withFilter($filter);

What do you get instead?

This function throw exception if i using custom filter. As I look at eg. ArrayDataReader, I have no chance of implementing a custom filter (for example, inheritance, but all filtering functions are private).

private function matchFilter(array $item, array $filter): bool

What steps will reproduce the problem?

Currently, paginator classes have no common part, so they cannot be handled uniformly. In addition, implementing a custom paginator is difficult if it comes in this form eg. in the view layer.

I think the first common part should be done with the data needed for paging and the restore from data.

What is the expected result?

$paginatorData = $paginatior->toParameters();
...
$paginator = $pagainator->withParameters($paginatorData);

What do you get instead?

Generate data for paging links:

$pages = [];
if($paginator instanceof KeysetPaginator) {
  $data = [
    'lastValue' => $paginator->getLast(),
    ...
  ];
   ...
} elseif($paginator instanceof OffsetPaginator) {
  ...
} else {
  // custom paginator???
}

...and restoring data to the paginator.

Suggest

interface Paginator {
  function toParamters(int $relativePage): ?array;
  function withParameters(aray $data);
}

example

class KeysetPaginator implements Paginator {
  public function toParameters(int $relativePage): ?array {
    ...
  }
  public function withParameters(array $data): self {
    $new = clone $this;
    if(isset($data['lastValue'])) {
      $new = $new->withLast($data['lastValue']);
    }
    ...
    return $new;
  }
}

$relativePage in toParamters($relativePage)

I suggest that the data be retrieved with a relative number of pages so that the paging view can retrieve the data needed for a page from any distance. If the paginator does not support something, it returns NULL.

example:

$pageData = [];
$relativePages = array_merge([
  PHP_INT_MIN /* first page */,
  PHP_INT_MAX /* last page */,
], 
  range(-5, 5)
  // N-5, N-4, N-3, N-2, N-1,
  // 0 (current page),
  // N+1, N+2, N+3, N+4, N+5,
);

foreace($relativePages as $relativePage) {
  $data = $paginator->toParameters($relativePage);
  if($data === null) {
    /* not available or not supported */
    continue;
  }
  $pageData[$relativePages] = $data;
}

// forming page data keys and creating eg URL

Additional info

Required for:

• page view
• TokenPaginator #5
• adding custom paginator

Creeate new class that implement DataProcessorInterface and use callable parameter to process data.

What steps will reproduce the problem?

in yii-demo modify the following file: src/Blog/Comment/CommentRepository.php from:

    private function getSort(): Sort
    {
        return Sort::only(['id', 'public', 'created_at', 'post_id', 'user_id'])->withOrder(['id' => 'asc']);
    }

to:

    private function getSort(): Sort
    {
        return Sort::only(['id', 'public', 'created_at', 'post_id', 'user_id'])->withOrder(['created_at' => 'desc']);
    }

What is the expected result?

comments are sorted descending order for created_at fields

What do you get instead?

Yiisoft\ErrorHandler\Exception\ErrorException: Undefined property: App\Blog\Entity\Comment Cycle ORM Proxy::$created_at in /app/vendor/cycle/orm/src/Mapper/Proxy/EntityProxyTrait.php:28
Stack trace:
#0 /app/vendor/cycle/orm/src/Mapper/Proxy/EntityProxyTrait.php(28): Yiisoft\ErrorHandler\ErrorHandler::Yiisoft\ErrorHandler\{closure}(2, 'Undefined prope...', '/app/vendor/cyc...', 28)
#1 /app/vendor/yiisoft/arrays/src/ArrayHelper.php(255): App\Blog\Entity\Comment Cycle ORM Proxy->__get('created_at')
#2 /app/vendor/yiisoft/arrays/src/ArrayHelper.php(231): Yiisoft\Arrays\ArrayHelper::getRootValue(Object(App\Blog\Entity\Comment Cycle ORM Proxy), 'created_at', NULL)
#3 /app/vendor/yiisoft/data/src/Paginator/KeysetPaginator.php(316): Yiisoft\Arrays\ArrayHelper::getValue(Object(App\Blog\Entity\Comment Cycle ORM Proxy), 'created_at')
#4 /app/vendor/yiisoft/data/src/Paginator/KeysetPaginator.php(262): Yiisoft\Data\Paginator\KeysetPaginator->getValueFromItem(Object(App\Blog\Entity\Comment Cycle ORM Proxy), 'created_at')
#5 /app/vendor/yiisoft/data/src/Paginator/KeysetPaginator.php(174): Yiisoft\Data\Paginator\KeysetPaginator->readData(Object(Yiisoft\Yii\Cycle\Data\Reader\EntityReader), Object(Yiisoft\Data\Reader\Sort))
#6 /app/views/blog/comments/_comments.php(19): Yiisoft\Data\Paginator\KeysetPaginator->read()

Additional info

We have Readable and Writeable interfaces. But we have no interface for deleting. It might be worth adding.

What steps will reproduce the problem? Updating the following composer file

What is the expected result? A clean run.

What do you get instead? The above error.

Additional info

"yiisoft/data": "dev-master as 1.0",
"yiisoft/yii-cycle":"dev-master",
"yiisoft/yii-dataview": "^3.0",

Hi Vjik.
I am experiencing the above error. Composer is updating with the above combination but then the program runs into this error. I understand there are BC issues at the moment with yiisoft\data Is this related to that? I fortunately have a stable vendor folder so still able to develop.

I suggest adding a method for reading a single element in the DataReaderInterface.
This method will work in much the same way as the read() method with a limit of 1, but will return not the collection of elements, but the first element.

interface DataReaderInterface
{
    public const DEFAULT_LIMIT = 10;

    public function withLimit(int $limit);
    public function read(): iterable;
    public function readOne();
}

This is need for remove confusing with two entities with same name: data processor and filter processor.

• Move Reader\Interable to IterableReader to separate concepts better.
• Merge FilterHandlerInterface into IterableHandlerInterface since it's used in iterable filter only.

What steps will reproduce the problem?

If the field does not exist then an undefined index is generated, I assume this is incorrect.

What is the expected result?

• In most cases: $item[$field] ?? null
  • =, <, <=, >, >=, in
• With like I don't know what to do correct properly.
  • always fails?

$ php -r 'var_dump(stripos(null, ""));'
bool(false)
$ php -r 'var_dump(stripos(null, null));'
bool(false)

• or null LIKE null is true

What do you get instead?

Undefined index: notExistsField

What's wrong?

Currently there is getters and setters support in KeysetPaginator. There's no such thing in other data readers.

Fix 1

We could make these explicit i.e. if you need a getter you state it specifically such as getName(). Note the (). We won't check existence of the method via reflection and won't inflect the name. Just call it directly.

Advantages:

• Explicit.
• Good performance.

Disadvantages:

• Method names will get into URLs when using filters.

Fix 2

Alternative is to extract inflector and reflection based magic from KeysetPaginator and use it everywhere.

Advantages:

• URLs will be short and nice.

Disadvantages:

• Magic.
• Lower performance.

What steps will reproduce the problem?

What is the expected result?

What do you get instead?

Additional info

These interfaces and functions are incorrectly placed under the "Reader" namespace because they can also be used for Writer:

• src/Reader/Filter
• src/Reader/SortableDataInterface.php
• src/Reader/FilterableDataInterface.php
• src/Reader/Sort.php
• DataReaderInterface#withLimit()

Example SQL update:

UPDATE `table` SET values... WHERE <filters> ORDER BY <sorters> LIMIT <limit>

Additional info

After #66 Sort is less usable for non-user input case. You have to specify all fields all the time and it's tedious.

I propose to have two static constructors:

• Sort::fields() for user input with specifying allowed fields.
• Sort::any() for non-user input allowing any fields.

What steps will reproduce the problem?

OffsetPaginator::getCurrentPageSize() work with DataReaderInterface::count() but OffsetPaginator::read() operate with DateReaderInterface::read().

If the data source changes between the two operations, count(OffsetPaginator::read()) !== OffsetPaginator::getCurrentPageSize() may occur.

What is the expected result?

count(OffsetPaginator::read()) === OffsetPaginator::getCurrentPageSize() is true

What do you get instead?

Depending on the circumstances count(OffsetPaginator::read()) === OffsetPaginator::getCurrentPageSize() is false

What steps will reproduce the problem?

Currently, the filtered value is treated as $item[$field]. I'm not really frontend developer, but I think yii2 knew it too:
https://github.com/yiisoft/yii2/blob/f8611d170d77f8b89f24bd80e7f3b6b2a8947380/framework/grid/DataColumn.php#L232

What is the expected result?

Yii3 also uses <field>.<subfield> with ArrayHelper::getValue()

What do you get instead?

Currently not supported.

Additional info

in #28 issue also affect the implementation.

Implementations of PaginatorInterface is very different. Some methods need in one implementation, and not need in other implementations. Some methods should named different in different implementations. Seems PaginatorInterface not need.

Suggest:

• remove PaginationInterface;
• remove from paginator implementation any "read" functional;
• add to paginator implementation new method getFilter(): FilterInterface that contains pagination logic in the form of a filter;
• usage result of getFilter() method in reader.

Example:

$pagination = (new KeysetPaginator($dataReader))->withPageSize(10);

$filter = new All($myFilter1, $myFilter2, $pagination->getFilter());

$reader = (new IterableDataReader($myDataSet))->withFilter($filter);

$result = $reader->read();

Remove travis if it exists

What steps will reproduce the problem?

Currently, it organizes the current page, not the whole items.

What is the expected result?

sort after slice

What do you get instead?

slice after sort

Additional info

If DataReader contains object items then this error is inevitable

Cannot use object of type App\Blog\Entity\Comment as array 
src/Paginator/KeysetPaginator.phpat line 271

The current filter set does not allow us to check for null values; there is no operator between, which is also often used when working with a collection of data.

I think such operators should be present in this basic package. Can you suggest some more filters?

What steps will reproduce the problem?

Yii 2.x pagination base sentence:

• total count of items
• page size
• page number (total count / page size)

Most cloud-based systems (storage, APIs, databases) do not support the total count of items, and only knows where the scanning has finished, so paging can only be done with a next page token.

Example cloud platform services:

• Cloud storages API (Google Cloud Storage, Amazon S3 etc.)
• Scanning based databases (Google Cloud Bigtable, HBase)
• All APIs with an infinite number of records.

What is the expected result?

Pagination support next page token. I do not think it is necessary to implement the previous page token, but it is worth discussing this topic as well.

In this case, the page to be displayed is only Load more.

What do you get instead?

In the absence of a complete batch, ArrayDataProvider requires custom solutions.

Additional info

Google applies the concept of maxResults. An alternative to pageSize, which does not specify a fixed number of elements in the response, but only the maximum number. MaxResults can vary between 0 and maximum on any page.

Items in the response can be limited to any criteria:

• the response is too long
• the response requires too many resources (eg RAM)
• the server is overloaded
• and more

Its application is very diverse:

• frontend/backend/api:
  • returns with fewer items, but much faster. By the time the user scrolls through the response items and scrolls further, the newer page will be loaded.
  • It allows processing huge sets of data (eg logs) in a system and user friendly way. For example, it allows only daily partitions, so the user can see where the search is going on the fly, and if he is no longer interested, he can navigate away from the page or stop the more scanning, saving the useless responses and processes.
  • The page uses the ignore_user_abort() function and if it detects connection_aborted it stops any further running (similar to the console process, the kill signal)
• console:
  • Limits the number of items created in the process
  • does not let the process run into infinity, eg. handles kill signal.
• There are paid services (such as Google Bigquery) that pay for the amount of data processed. It is unfortunate that processes continue to run even when they are no longer needed. (The Bigquery supports a partitioning solution by default.)

I could imagine this feature in KeysetPaginator.

This roughly strains the boundaries of the layers and requires its correct design. You can think about whether the theme of a separate package, or fits into the framework, but one thing is certain: at worst, any external package should be able to handle a similar concept.

Suggest

interface StoppableStrategyDataInterface {
  public function withStoppableStrategy(StobbaleStrategy $strategy)
}

example strategies

• list (group)
  • AnyStrategy
  • ScoredStrategy
• final
  • UserAbortStrategy (connection_aborted, kill signal handling)
  • PartitionStrategy (field and an associated value range)
  • ScanningCountStrategy
  • SystemResourceStrategy
    • SystemMemoryStrategy
    • SystemDiskStrategy
    • SystemCpuStratagy

I have a question about Sort because when i create object Sort as below:

$sort = Sort::only([
  'title' => [
      'asc' => ['id' => 'asc', 'title' => 'desc'],
      'desc' => ['first_name' => 'desc', 'title' => 'asc'],
      'default' => 'desc'
  ],
  'descscription' => [
      'asc' => ['id_2' => 'asc', 'title_2' => 'desc'],
      'desc' => ['first_name_2' => 'desc', 'title_2' => 'asc'],
      'default' => 'desc'
  ],
])->withOrder(['descscription' => 'asc']);

Function getCriteria return:
Array ( [id_2] => asc [title_2] => desc [first_name] => desc [title] => asc )

Why criteria return rest sorts by default option ? I think when i need only description asc it should return only this sort.
In function getCriteria i found this:

foreach ($config as $field => $fieldConfig) {
    $criteria += $fieldConfig[$fieldConfig['default']];
}

Should it work in that way ?
I've checked how it works in yii2 and sort doesn't return rest sorting options by default value.
In yii-cycle it's used getOrder instead of getCriteria.
Could someone explain me how can i create order query in my Doctrine Reader by using this class Sort.
Perhaps in this package is another aproach ?

So as not to confuse them with filters in suggestions:

The KeysetPaginator class needs to be typed and possibly refactored. It is also worth paying attention to tests.

Now we can:

• pass filters to constructor
• use withCriteriaArray() for set filters or criteria array.

Suggest allow pass to constructor mixed array of filters and criteria (as in withCriteriaArray()) and remove withCriteriaArray().

It's more clear name.

Implementations of PaginatorInterface is very different. Some methods need in one implementation, and not need in other implementations. Some methods should named different in different implementations. Seems PaginatorInterface not need.

For example, KeysetPaginator contains methods withNextPageToken(string $token) and withPreviousPageToken(string $token), but in OffsetPaginator instead this methods need one method withCurrentPage(int $page).

Yii Data depended packages:

• yiisoft/data-db,
• yiisoft/yii-cycle,
• yiisoft/app-api,
• yiisoft/demo,
• yiisoft/yii-dataview.

But PaginatorInterface used in yiisoft/yii-dataview only. In class BaseListView user may set paginator via method paginator(PaginatorInterface $value), but into class code used implementations KeysetPaginator and OffsetPaginator, and pass custom implementation of PaginatorInterface throws error or will be work incorrect, for example, here:

private function renderSummary(): string
{
	if ($this->getPaginator() instanceof KeysetPaginator) {
		return '';
	}

	$pageCount = count($this->getDataReader());

	if ($pageCount <= 0) {
		return '';
	}

	/** @var OffsetPaginator $paginator */
	$paginator = $this->getPaginator();
	
	// ...

FilterDataValidationHelper contains common reader methods. assertFilterHandlerIsIterable() used for iterable reader only and should be moved to Iterable namespace.

See: #85 (comment)

Data:

• Alexander
• Alexey
• Boris
• Alena

Filter:

• Ale

Output:

• Alexander
• Alexey
• Alena

In the IterableDataReader class, the default Limit is set to 10 on creation.

I think this is wrong (not expected) behavior.
When i send an array of 20 elements to the IterableDataReader, i expect the read() method to read all the values by default.

The default limit should be set by a paginator.
If the user needs a limit directly in the reader, he will do it manually.

Occurred after the PR #13874.

What steps will reproduce the problem?

We have application with restful API which uses ActiveDataProvider (extends BaseDataProvider) in Index action to support pagination.
Let's consider we have one API resource /users with pagination (see http://www.yiiframework.com/doc-2.0/guide-rest-response-formatting.html#data-serializing) and we create two test methods.

1. Gets all items of /users resource.
2. Gets all items of /users resource from the second page (query parameter per-page=2).

Since both tests run within a single process calling the first one will initialize BaseDataProvider::$counter=0 and increase it by 1. Calling of the second test will use $counter=1 and ActiveDataProvider of the /users resource will have a different ID now. So it won't accept per-page parameter because it's waiting for the dp-1-per-page.
The fact this "dynamic ID" can not be disabled is frustrating.

What is the expected result?

ID of DataProvider remains consistent name during tests running.

What do you get instead?

ID of DataProvider changes.

Additional info

What steps will reproduce the problem?

There may be times when data needs to be scanned as reversed. Currently there is no specific interface defined, but can using with reversed Sort orders.

• RDBMS: supported partially or with sort
• NoSQL:
  • mongoDb: supported by sort
  • Apache HBase: supported by scanner level
  • Google Bigtable: not supported

What is the expected result?

$reader->withReverseScanning()->read();

examples:

• if supported it or partially supported

SELECT * FROM (
    SELECT *
    FROM ...
    WHERE id > :first
    ORDER BY id DESC
) ORDER BY id ASC LIMIT 3  -- this is not real reverse scanning

• if not supported it, but supported sorting
  exmaple in IteratorDataReader
• If not supported it
  Depending on the number of data, use the array_reverse() function, but for example, if the number of data can be hundreds of millions, can decide to throw an exception based on the withLimit(). (typically with noSQL)

What do you get instead?

Additional info

Also consider giving Sort awithInversed() or withReverseSorting() method.

Currently Sort label isn't used. Either should be used or be removed.

IteratorDataReader uses ArraySorter::multisort that uses array_multisort() function. This function resets numeric keys.

This behavior is undesirable.

Syntax could be something like:

$filter = $aaa->where([
  ['and', 'id', '>', 3],
  ['and', ['or', ['temperature', '<', 10], ['temperature', '>', 30]] ],
  ['and', 'name', like', '%agent%'],
];

See https://yiiframework.ru/forum/viewtopic.php?f=39&t=52964

What steps will reproduce the problem?

What is the expected result?

What do you get instead?

Additional info

The idea of keyset pagination is simple. Instead of "get me a page 5" you're telling framework get me 10 records since ID = X. On the 1st page X is 0, on the next page it's last ID from the previous page:

SELECT ...
  FROM ...
 WHERE ...
   AND id < ?last_seen_id
 ORDER BY id DESC
 LIMIT 10

Why

• "Normal" pagination is inefficient on big data sets because OFFSET 100000 LIMIT 10 has to fetch 100010 records.
• count(*) is inefficient. At least when using InnoDB.

Considerations

In many cases there's no need to know exact number of a page you are on, total number of pages and total items count.

Therefore visually it can be what Reddit does:

There's a number of other ways to display data using the same technique:

• Infinite scroll (what Twitter does).
• "Load more" what Facebook does.

Which way to use heavily depends on project type. Social networks are fine with infinite scroll, e-commerce is OK with "Load more.

Still, classic pagination is popular and heavily used so it is a bad idea to remove it from the framework.

Implementation

Implementation spans over several components:

1. Pagination and data providers. Introduce a way to specify a field or fields on which we're seeking. Consider if we can make it part of base data provider. If not, separate it into its own class.
2. Introduce separate data-widgets for keyset pagination but consider making it an option in existing ones (BaseListView mainly).
3. Introduce a widget that displays "Previous/Next" buttons.

References

• https://forum.yiiframework.com/t/pagination-without-offset/123570/7
• https://use-the-index-luke.com/no-offset

What steps will reproduce the problem?

It also causes problems with usage and implementation.

For example, the PaginatorInterface cache is the result itself or not, because the data reader is already doing it.

Repeated reading may require:

• caching: repeated read for get count, items and more.
• not caching: reuse object, and watch change at any time.

What is the expected result?

specfy exactly

What do you get instead?

not specified

What steps will reproduce the problem?

IterableDataReader#data is iterable type, but count() support only array and \Countable.

What is the expected result?

$reader = IterableDataReader(new class implements \Iterator { ... });
var_dump($reader->count());     // throw not supported exception

What do you get instead?

$reader = IterableDataReader(new class implements \Iterator { ... });
var_dump($reader->count());
// PHP Warning:  count(): Parameter must be an array or an object that implements Countable in Command line code on line 1
// int(1)

Additional info

This issue has originally been reported by @az-joss at yiisoft/yii2#16625.
Moved here by @samdark.

What steps will reproduce the problem?

I have configured IndexAction in controller like fallow

    public function actions()
    {
        $actions =  ArrayHelper::merge(
            parent::actions(),
            [
                'index' => [
                    'dataFilter'          => [
                        'class'        => ActiveDataFilter::class,
                        'searchModel'  => function () {
                            return (new \yii\base\DynamicModel([
                                'tags'     => null,
                                'counties' => null,
                            ]))
                                ->addRule('tags', 'each', ['rule' => ['integer']])
                                ->addRule('counties', 'each', ['rule' => ['integer']]);
                        },
                    ]
                ],
            ]
        );

        return $actions;
    }

As you can see, expect to use this attributes like an array of integers
When I use operator in in filter param as fallow:

http://somehost.com/endpoint?filter[tags][in][]=5&filter[tags][in][]=6

Validation of DynamicModel fails because-of passing value not as an array and as each element of an array.
If change DynamicModel rules definision as fallow:

...
     ->addRule('tags', 'integer')
     ->addRule('counties', 'integer');
...

It works fine for IN condition but not for the simple conditions
I've traced it and found this, where loop over an array goes and validator recives element and not all list as expected..

Additional info