geminidsystems / googlenewsscraper Goto Github PK

@abnoviello23

For stability reasons, we want to replace the use of find_elements_by_class_name or div[contains(@class)], google changes the class names regularly and it breaks our script

We should be able to select what we need using one of the following

• select by id (preferred method as this is unlikely to change)
• select by tag name (example <img/> for the image_url and <a/> for the url for sure can be used)
• select by tag position (for example we know the text content we want is under a > div > div > [div,div,div] (the 3 divs each contain the source, title, and description we need)

So I tried the example but I got the following error:

Message: javascript error: Cannot read properties of null (reading 'classList')
  (Session info: headless chrome=103.0.5060.114)

What can I do to fix this?

Debug why the results are empty
Report on the ticket what is going on
Try to fix it

Hi there,
Thanks for this great script. I was able to get the output text in the terminal, however I am unable to figure out how to save the results in a spreadsheet format — my python skills are obviously limited. Could you expand your example code to include storing the results in csv? thanks.

@abnoviello23

Our scraper current allows for the caller to use their own driver or use the static driver

We want to replace the static driver to use ChromeDriverManager (see working in example/app.py)

The user must still be allowed to pass their own driver in as shown below

GoogleNewsScraper(my_driver)

If there is no driver passed in, we want to use ChromeDriverManager to install the latest version of the chrome driver

I am working on the following.

This method of selecting HTML elements fails to work.

'//div[@id="rso"]/div/div/div/a/div/div[2]/div[4]/p/span'

Additionally, these HTML selectors need to be updated.

driver.execute_script("""
        const menu = document.getElementById('hdtbMenus');
        menu.classList.remove('p4DDCd');
        menu.classList.add('yyoM4d');
""")