[Bug] [ml] Incorrect behavior of Topic Retention Policy about pulsar

[dao-jun]

@lhotari PTAL

[lhotari]

Good description @dao-jun . This makes sense.

We need to ask the original developer of the solution for clarification whether this is a bug or not. I guess
it depends a lot on how the "retention" settings are defined.

[dao-jun]

@lhotari According to https://pulsar.apache.org/docs/3.2.x/cookbooks-retention-expiry/#retention-policies and ManagedLedgerImpl#internalTrimLedgers, delete a ledger or not should be based on Total size of consumed ledgers, base on Total size of all the ledgers will lead to ManagedLedgerImpl#isLedgerRetentionOverSizeQuota always true(in some cases). This is not in line with its intended meaning

[dao-jun]

/cc @merlimat @hangc0276 @codelipenghui @poorbarcode @mattisonchao @BewareMyPower @Technoboy-

[lhotari]

@lhotari According to https://pulsar.apache.org/docs/3.2.x/cookbooks-retention-expiry/#retention-policies and ManagedLedgerImpl#internalTrimLedgers, delete a ledger or not should be based on Total size of consumed ledgers, base on Total size of all the ledgers will lead to ManagedLedgerImpl#isLedgerRetentionOverSizeQuota always true(in some cases). This is not in line with its intended meaning

+1 @dao-jun the reported issue seems to be a bug based on this information.

The definition of the retention settings was explicitly mentioned.

The retention settings apply to all messages on topics that do not have any subscriptions, or to messages that have been acknowledged by all subscriptions. The retention policy settings do not affect unacknowledged messages on topics with subscriptions. The unacknowledged messages are controlled by the backlog quota.

[dao-jun]

See:
https://github.com/apache/pulsar/pull/2206/files#diff-63f327b2f634d152b3ab8528d0d4ba49eb39276482ae60eff3f182ea2e1037d6

#14546

[dao-jun]

In the first version doc, it says: messages in all topics in the namespace, even acknowledged messages, will be retained.

[dao-jun]

This is a doc issue, since https://github.com/apache/pulsar/pull/5482/files