Comments (22)
Hi @m1ch4elx can you post example code that's causing this error?
from requestium.
cooks = open('cookies/YT_cookie96828943.txt', encoding='utf-8').read().splitlines()
lastcooks = []
for cook in cooks:
domain, flag, path, secure, expiration, name, value = cook.split(' ')
data = {
'domain': domain,
'flag': bool(flag),
'path': path,
'secure': bool(secure),
'expiration': expiration,
'name': name if len(name) >=1 else 'None',
'value': value
}
lastcooks.append(data)
url = 'https://mail.google.com'
s = Session(webdriver_path='chromedriver.exe',
browser='chrome',
default_timeout=15,)
for cook in lastcooks:
s.driver.ensure_add_cookie(cook, override_domain='')
s.driver.get(url)```
from requestium.
Hi @m1ch4elx can you post example code that's causing this error?
Seems like dict of cookies is correct
from requestium.
Hi @m1ch4elx can you post example code that's causing this error?
It goes wrong when trying to add cookie to google.com.ec even when webdriver on this page
from requestium.
Can you tell me a little bit more about what you're trying to accomplish? It seems you want to load some cookies so that you can scrape data out of Gmail? When mail.google.com
loads, are you authenticated or does it redirect to account.google.com
expecting you to login there?
from requestium.
It's just example of cookies, but mail.google.com is redirecting to account.google.com too. The main problem is webdriver trying to apply cookies to google.com.es, cuz of ensure_add_cookies it goes to that page and after that giving domain mismatch error
from requestium.
Can you tell me a little bit more about what you're trying to accomplish? It seems you want to load some cookies so that you can scrape data out of Gmail? When
mail.google.com
loads, are you authenticated or does it redirect toaccount.google.com
expecting you to login there?
And even with mail.google.com it's not working too, cuz when ensure_add_cookies trying to get to mail.google.com it redirects to accounts.google.com and domain mismatch shows
from requestium.
All ensure_add_cookie
does is visit the page of the cookie it's trying to load. If that page redirects (which is outside of the function's control since the webdriver behaves exactly like a browser), the function won't have an opportunity to load the cookie until it's too late.
The above situation is exactly what the override_domain
parameter is for. So if I'm understanding your use case right, for your cookies you need to set override_domain
to account.google.com
, and then each cookie will be added there.
from requestium.
Ok, will try it now and get u know
from requestium.
from requestium.
Im trying to add netscape cookies but formatted to dict
from requestium.
Same InvalidCookieDomainException happening here. And the driver loads the page unauthenticated
from requestium.
Hey all, I'm wanting to dig back into this issue, but need a clear and complete example of steps to reproduce. @m1ch4elx your code is the closest to that, but it references a cookies file and I'd need to know how you created that so I can try to make a similar one.
from requestium.
Hey all, I'm wanting to dig back into this issue, but need a clear and complete example of steps to reproduce. @m1ch4elx your code is the closest to that, but it references a cookies file and I'd need to know how you created that so I can try to make a similar one.
It's just a simple file with cookies in netscape, Firefox has extension called cookie quick manager that can save cookies in that format
from requestium.
So you log into Google account manually in Firefox, save the cookies off, and then you want to use requestium to automate interacting with your gmail, am I tracking that right?
from requestium.
So you log into Google account manually in Firefox, save the cookies off, and then you want to use requestium to automate interacting with your gmail, am I tracking that right?
Well, I can say yes, I guess
from requestium.
I just published a new version that is a bit more robust in its loading of cookies, but ultimately the issue here wasn't requestium
itself but some limitations in Selenium combined with quirks in how sites like google do authentication. For example, the order you load the cookies matters.
Here's code I tested that worked with the latest version:
# Only include cookies relevant to the google domain
lastcooks = [c for c in lastcooks if 'google.com' in c['domain']]
# Sort by domain (effectively ensuring the google.com cookies are loaded ahead of mail.google.com)
lastcooks.sort(key=lambda c: c['domain'])
# A handful of cookies still don't load, so make sure to keep going in those cases
for cook in lastcooks:
try:
s.driver.ensure_add_cookie(cook, override_domain='google.com') # Note the domain override here
except Exception as error:
pass
url = 'https://mail.google.com'
s.driver.get(url)
@m1ch4elx give this a try and let me know how it goes for you.
from requestium.
I just published a new version that is a bit more robust in its loading of cookies, but ultimately the issue here wasn't
requestium
itself but some limitations in Selenium combined with quirks in how sites like google do authentication. For example, the order you load the cookies matters.Here's code I tested that worked with the latest version:
# Only include cookies relevant to the google domain lastcooks = [c for c in lastcooks if 'google.com' in c['domain']] # Sort by domain (effectively ensuring the google.com cookies are loaded ahead of mail.google.com) lastcooks.sort(key=lambda c: c['domain']) # A handful of cookies still don't load, so make sure to keep going in those cases for cook in lastcooks: try: s.driver.ensure_add_cookie(cook, override_domain='google.com') # Note the domain override here except Exception as error: pass url = 'https://mail.google.com' s.driver.get(url)@m1ch4elx give this a try and let me know how it goes for you.
Okay, I'll try it
from requestium.
Not working for me :(
from requestium.
I saw similar behavior until I played around with 1) which cookies I tried to load, and 2) the order that I loaded them in. Do some experimentation.
from requestium.
I saw similar behavior until I played around with 1) which cookies I tried to load, and 2) the order that I loaded them in. Do some experimentation.
Okay, thanks anyway
from requestium.
Hey @m1ch4elx I realize in re-reading my previous post I may have come across as dismissive. I apologize for that. It's just that there isn't much more that requestium
as a library can do to help in this situation. It can't possibly know the particularities of every website's use of cookies, how they redirect to other domains when cookies aren't present, etc.
What I can say is that when I exported all my cookies using the Firefox plugin you described, then filtered the list by ones whose domain contained google.com
, and loaded them into requestium
in sorted order, I was able to open and interact with mail.google.com
successfully.
With some persistence and trial and error I bet you can figure out what works for your situation also. But for now I'm closing this ticket.
from requestium.
Related Issues (20)
- Empty cookies <RequestsCookieJar[]> HOT 1
- How to support IE browser HOT 1
- How to use with Firefox? HOT 2
- PhantomJS critical error with Selenium v4 HOT 10
- selenium.common.exceptions.WebDriverException: Message: 'chromedriver' executable needs to be in PATH HOT 3
- pip has wrong dependencies, rendering requestium useless HOT 2
- Why can't I use request features also in selenium? HOT 1
- External webdriver outside Requestium and Selenium Wire HOT 6
- 👋 From the Selenium project!
- avoid GET when using transfer_session_cookies_to_driver() HOT 2
- Browser opens, but does not go to desired page HOT 1
- Use undetected_chromedriver in driver HOT 2
- About requestium with 2 step authentication HOT 1
- requestium 0.2.0 requires selenium<4.0.0,>=3.7.0, but you have selenium 4.8.3 which is incompatible. HOT 6
- The version dependency of selenium has not been updated HOT 1
- socks5 HOT 1
- Out of Date Requirements HOT 5
- find_element / ensure_element HOT 3
- Release tags
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from requestium.