Question about the title about trafilatura

[adbar]

I may use different heuristics but this is open for debate.

[pieterhartel]

Well, I tried to summarise what trafilatura is currently doing, could you confirm or correct please?

[adbar]

My bad, I answered question 2.

Title extraction roughly works as you described in your list, with two additional steps:

• First and foremost HTML meta or JSON information is used when available
• Tag attributes are used in some cases, not only tags themselves, e.g. @class="entry-title"

[pieterhartel]

I have a dataset with over 10M home pages, of which 82K (less than 1%) contain a @class="entry-title". WP is not popular in this dataset.
75K pages contain a <h1 class="entry-title">, 56K a h2, 18K a h3.
For my dataset h1 and h2 are sufficient.

I'll figure out next how many pages contain a <meta name="description" content="...">

[adbar]

That's interesting, feel free to share results if you find anything else, I patched together a series of heuristics but I lack the time to evaluate them properly. Besides, the quick searches I run on my data may also be biased so it's useful to try them on yours.

The manually curated list of X-Path expressions to extract titles is here:

trafilatura/trafilatura/metaxpaths.py

Line 54 in c6f4559

title_xpaths = [

    '//*[(self::h1 or self::h2)][contains(@class, "post-title") or contains(@class, "entry-title") or contains(@class, "headline") or contains(@id, "headline") or contains(@itemprop, "headline") or contains(@class, "post__title") or contains(@class, "article-title")]',
    '//*[@class="entry-title" or @class="post-title"]',
    '//*[(self::h1 or self::h2 or self::h3)][contains(@class, "title") or contains(@id, "title")]',