Google sometimes serves a legacy page meant for bots and/or old browsers - this breaks the search controller and results in a TimeoutException. about ad_clicker

[pmnlla]

I figured I should attach my chrome options.

    chrome_options = ChromeOptions()
    chrome_options.add_argument("--disable-gpu")
    chrome_options.add_argument("--disable-dev-shm-usage")
    chrome_options.add_argument("--disable-infobars")
    chrome_options.add_argument("--disable-popup-blocking")
    chrome_options.add_argument("--disable-notifications")
    chrome_options.add_argument("--ignore-ssl-errors")
    chrome_options.add_argument("--start-maximized")
    chrome_options.add_argument(f"--user-agent={user_agent_str}")
    chrome_options.add_argument("--user-data-dir=/tmp/chrome")
    chrome_options.add_argument("--disable-setuid-sandbox")
    chrome_options.add_argument("--disable-application-cache")
    chrome_options.add_argument("--disable-extensions")
    chrome_options.add_argument("--enable-javascript")
    chrome_options.add_argument("--disable-blink-features=AutomationControlled")
    
#if headless
    chrome_options.add_argument("--no-sandbox")
    chrome_options.add_argument("--window-size=1920,1080")

[pmnlla]

Update on this: the legacy page does not serve ads any more, as far as I'm aware. This means we need to figure out what triggers it, which I found out is some obscured fingerprinting code. Unless we can bypass it, mass deployment of this is likely toast after a while on instances dedicated to it.

I'm waiting for a friend to spin up some VMs for me on her home lab so I can test this using non-oracle IP ranges.

[coskundeniz]

It is related to selected user agent string. I updated it to take user agent only from the predefined list.

[pmnlla]

Got it. Pulled the latest commit, everything seems to work now. Thanks a lot!