Deduplicate items across sources about selfoss

[jtojnar]

Thanks, that is interesting idea. I wonder if we could make it always enabled and have the item in multiple sources.

We would probably need to replace the source column in the items table with an m:n association table. Will need to check the performance implications.

[davidoskky]

This is a very nice idea, what are you using as identifier to deduplicate? The url?
What if the two feeds return a different content? Should not be an issue if you're using the full text recovery though.

[jtojnar]

what are you using as identifier to deduplicate? The url?

The UID. Most commonly, this is the post URL but it is not required. For example blogger.com will use something like tag:blogger.com,1999:blog-6112936277054198647.post-403878284366003238.

What if the two feeds return a different content? Should not be an issue if you're using the full text recovery though.

We could have findAll return the source id in addition to item id and check whether the content and url matches when the source id does not, and only deduplicate it then.

That would also probably resolve the uid collisions.

The issue that items will be missing from some of the sources will still remain, though, which is why I would like to test the performance impact of having sources table in m:n relation to items.