Message: invalid cookie domain: Cookie 'domain' mismatch about requestium

[lordjabez]

Hi @m1ch4elx can you post example code that's causing this error?

[m1ch4elx]

cooks = open('cookies/YT_cookie96828943.txt', encoding='utf-8').read().splitlines()
lastcooks = []
for cook in cooks:
  domain, flag, path, secure, expiration, name, value = cook.split('	')
  data = {
    'domain': domain,
    'flag': bool(flag),
    'path': path,
    'secure': bool(secure), 
    'expiration': expiration, 
    'name': name if len(name) >=1 else 'None', 
    'value': value
  }
  lastcooks.append(data)

url = 'https://mail.google.com'

s = Session(webdriver_path='chromedriver.exe',
            browser='chrome',
            default_timeout=15,)

for cook in lastcooks:
    s.driver.ensure_add_cookie(cook, override_domain='')

s.driver.get(url)```

[m1ch4elx]

Hi @m1ch4elx can you post example code that's causing this error?

Seems like dict of cookies is correct

[m1ch4elx]

Hi @m1ch4elx can you post example code that's causing this error?

It goes wrong when trying to add cookie to google.com.ec even when webdriver on this page

[lordjabez]

Can you tell me a little bit more about what you're trying to accomplish? It seems you want to load some cookies so that you can scrape data out of Gmail? When mail.google.com loads, are you authenticated or does it redirect to account.google.com expecting you to login there?

[m1ch4elx]

It's just example of cookies, but mail.google.com is redirecting to account.google.com too. The main problem is webdriver trying to apply cookies to google.com.es, cuz of ensure_add_cookies it goes to that page and after that giving domain mismatch error

[m1ch4elx]

Can you tell me a little bit more about what you're trying to accomplish? It seems you want to load some cookies so that you can scrape data out of Gmail? When mail.google.com loads, are you authenticated or does it redirect to account.google.com expecting you to login there?

And even with mail.google.com it's not working too, cuz when ensure_add_cookies trying to get to mail.google.com it redirects to accounts.google.com and domain mismatch shows

[lordjabez]

All ensure_add_cookie does is visit the page of the cookie it's trying to load. If that page redirects (which is outside of the function's control since the webdriver behaves exactly like a browser), the function won't have an opportunity to load the cookie until it's too late.

The above situation is exactly what the override_domain parameter is for. So if I'm understanding your use case right, for your cookies you need to set override_domain to account.google.com, and then each cookie will be added there.

[m1ch4elx]

Ok, will try it now and get u know

[m1ch4elx]

Got this error

[m1ch4elx]

Im trying to add netscape cookies but formatted to dict

[victorccaldas]

Same InvalidCookieDomainException happening here. And the driver loads the page unauthenticated

[lordjabez]

Hey all, I'm wanting to dig back into this issue, but need a clear and complete example of steps to reproduce. @m1ch4elx your code is the closest to that, but it references a cookies file and I'd need to know how you created that so I can try to make a similar one.

[m1ch4elx]

Hey all, I'm wanting to dig back into this issue, but need a clear and complete example of steps to reproduce. @m1ch4elx your code is the closest to that, but it references a cookies file and I'd need to know how you created that so I can try to make a similar one.

It's just a simple file with cookies in netscape, Firefox has extension called cookie quick manager that can save cookies in that format

[lordjabez]

So you log into Google account manually in Firefox, save the cookies off, and then you want to use requestium to automate interacting with your gmail, am I tracking that right?

[m1ch4elx]

So you log into Google account manually in Firefox, save the cookies off, and then you want to use requestium to automate interacting with your gmail, am I tracking that right?

Well, I can say yes, I guess

[lordjabez]

I just published a new version that is a bit more robust in its loading of cookies, but ultimately the issue here wasn't requestium itself but some limitations in Selenium combined with quirks in how sites like google do authentication. For example, the order you load the cookies matters.

Here's code I tested that worked with the latest version:

# Only include cookies relevant to the google domain
lastcooks = [c for c in lastcooks if 'google.com' in c['domain']]
# Sort by domain (effectively ensuring the google.com cookies are loaded ahead of mail.google.com)
lastcooks.sort(key=lambda c: c['domain'])

# A handful of cookies still don't load, so make sure to keep going in those cases
for cook in lastcooks:
    try:
        s.driver.ensure_add_cookie(cook, override_domain='google.com')  # Note the domain override here
    except Exception as error:
        pass

url = 'https://mail.google.com'
s.driver.get(url)

@m1ch4elx give this a try and let me know how it goes for you.

[m1ch4elx]

I just published a new version that is a bit more robust in its loading of cookies, but ultimately the issue here wasn't requestium itself but some limitations in Selenium combined with quirks in how sites like google do authentication. For example, the order you load the cookies matters.

Here's code I tested that worked with the latest version:

# Only include cookies relevant to the google domain
lastcooks = [c for c in lastcooks if 'google.com' in c['domain']]
# Sort by domain (effectively ensuring the google.com cookies are loaded ahead of mail.google.com)
lastcooks.sort(key=lambda c: c['domain'])

# A handful of cookies still don't load, so make sure to keep going in those cases
for cook in lastcooks:
    try:
        s.driver.ensure_add_cookie(cook, override_domain='google.com')  # Note the domain override here
    except Exception as error:
        pass

url = 'https://mail.google.com'
s.driver.get(url)

@m1ch4elx give this a try and let me know how it goes for you.

Okay, I'll try it

[m1ch4elx]

Not working for me :(

[lordjabez]

I saw similar behavior until I played around with 1) which cookies I tried to load, and 2) the order that I loaded them in. Do some experimentation.

[m1ch4elx]

I saw similar behavior until I played around with 1) which cookies I tried to load, and 2) the order that I loaded them in. Do some experimentation.

Okay, thanks anyway

[lordjabez]

Hey @m1ch4elx I realize in re-reading my previous post I may have come across as dismissive. I apologize for that. It's just that there isn't much more that requestium as a library can do to help in this situation. It can't possibly know the particularities of every website's use of cookies, how they redirect to other domains when cookies aren't present, etc.

What I can say is that when I exported all my cookies using the Firefox plugin you described, then filtered the list by ones whose domain contained google.com, and loaded them into requestium in sorted order, I was able to open and interact with mail.google.com successfully.

With some persistence and trial and error I bet you can figure out what works for your situation also. But for now I'm closing this ticket.