metadata extraction problem about trafilatura

[felipehertzer]

Hey @fortyfourforty, I can't replicate this issue with the given URL, also they aren't using json-ld. Can you provide more details as Trafilatura version?

[fortyfourforty]

Hi @felipehertzer, I just tested with a clean install the issue is gone. Guess it has to do with my own code. Thanks for your time testing!

[adbar]

Thanks @fortyfourforty & @felipehertzer for your feedback.

[fortyfourforty]

Another example of the same problem.

I've tested with a clean pip install, uses trafilatura's built-in fetch and can reproduce the same error. @felipehertzer @adbar

Error url : https://www.safetyinsurance.com/

Error: /usr/local/lib/python3.10/dist-packages/trafilatura/json_metadata.py in extract_json(schema, metadata)
63 if metadata.sitename is None or (len(metadata.sitename) < len(candidate) and content_type != "webpage"):
64 metadata.sitename = candidate
---> 65 if metadata.sitename is not None and metadata.sitename.startswith('http') and not candidate.startswith('http'):
66 metadata.sitename = candidate
67

AttributeError: 'list' object has no attribute 'startswith'

Function:
downloaded = fetch_url(url)
res = trafilatura.extract(downloaded, output_format="xml", include_comments=False)

output_format="json" also produces the same error.

The workaround is by commenting out the corresponding metadata.sitename check. But I'm also looking for an official fix.

[adbar]

The webpage is currently offline, I tried with an archived version (https://web.archive.org/web/20230607164633/https://www.safetyinsurance.com/) and I cannot reproduce the problem. Are you using the last Trafilatura version?

[fortyfourforty]

Yes, last Trafilatura version. @adbar
Pip install in colab with a new notebook dedicated for testing.

I tested again with a notebook, you can have a look.
The site is not offline as I just tested and you can also see the source in the notebook.

https://colab.research.google.com/drive/1Bm5rJpt40_veEAGREOwIw-EcMelXHiCr?usp=sharing

[adbar]

Thanks @fortyfourforty, I can reproduce the bug using the HTML in the notebook.

@felipehertzer Do you have an idea?

[felipehertzer]

Hey @adbar,

The problem is that this website is using alternateName as list instead of text. I added a extra to avoid that bug.

https://schema.org/Organization