Server returned a status code of 403 while downloading https://poe. com about poe-api

[mak448a]

Please stop bombing this thread unless if you actually have more information. It'll make it easier on the developers. Also, if you have the same issue, place a thumbs up on the first comment.

[Nox998464]

https://github.com/snowby666/poe-api-wrapper is working for me

[ading2210]

I've fixed this in the latest commit, but the library is still broken due to changes in the __NEXT_DATA__ format.

[ading2210]

My current understanding of this bug is that Poe has started enforcing some sort of cloudflare-based verification which requires a full web browser.

[canxin121]

not the same promblem, i can successfuly download the poe.com, but i can't send query successfullt.
@canxin121

Are you able to port those changes from your async client to this library?

But my library don't have this downloading problem. I don't know why, perhaps because the headers? or aiohttp?I will try to find the reason.

[ading2210]

Sorry for the lack of an update, but I'm currently busy right now and won't be able to work on this for at least a week. Although if someone submits a PR that fixes this issue then I'll merge it and publish a new release.

[johnd0e]

https://github.com/x404xx/POE-Api seems ok

[Hafi-Ansari]

"After investigating, I found that accessing https://poe.com/ using GET method prompts a 403 error, 'Enable JavaScript and cookies to continue.' I can successfully retrieve data by using the Selenium library to access https://poe.com/, but it requires writing the cookies: p-b, __cf_bm, and cf_clearance in advance."

is what @31702160136 is saying, but what exactly does this mean?

[bolavefasfas]

Hi Guys,

Started getting this error on Saturday, I sent an email to Poe and they said that the specific IP has been blocked for 7 days. I have tried using my same token from another IP, I tried getting another account with a different token and all have come up with the same 403 error. I think that they must have some antiscraping software running which is flagging the API requests. Has anyone tried using sock5 proxies ? Any better ?

Did you try new account with new ip ?
And if they are banning or they have some mechanism in place and even if new ip with new account works it will be useless as it will be limited again right ?

I have tried with diffrent ip and diffrent account key still the error

[canxin121]

async poe client didn't have this problem, but it could't extarct formkey, "document is not defined", it requires a browser environment to run the script? I could't find a way to do that except for playwright. So i set formkey a must currently.But all other fuctions run well.

[DogManTC]

I've fixed this in the latest commit, but the library is still broken due to changes in the __NEXT_DATA__ format.

So does it work now or not?

[Toaster496]

same here

[cmldyu]

same problem

[445022889]

一样的遭遇。

[media1demo]

yeah same here RuntimeError: Failed to download https://poe.com too many times.

[bolavefasfas]

Its hapenning on older versions as well i was in v0.4.x
Its sad i had lots of texts to summerize

[freQuensy23-coder]

Same problem(

same here!

[Milou4Dev]

SAME

[chaney1030]

same

[Hafi-Ansari]

Unfortunately same :(

[double-zinc]

same here :(

[31702160136]

我经过排查后发现使用GET访问https://poe.com提示403，Enable JavaScript and cookies to continue。我使用selenium库去访问https://poe.com可以成功获取到数据，但需要提前写入cookie：p-b、__cf_bm、cf_clearance才可以

[31702160136]

"After investigating, I found that accessing https://poe.com/ using GET method prompts a 403 error, 'Enable JavaScript and cookies to continue.' I can successfully retrieve data by using the Selenium library to access https://poe.com/, but it requires writing the cookies: p-b, __cf_bm, and cf_clearance in advance."

is what @31702160136 is saying, but what exactly does this mean?

我对这方面不太熟悉，我认为是需要加载JavaScript后才可以获得到数据，我们通常使用代码进行GET请求页面是无法加载JavaScript的，但是使用浏览器就可以加载JavaScript

[Hafi-Ansari]

"After investigating, I found that accessing https://poe.com/ using GET method prompts a 403 error, 'Enable JavaScript and cookies to continue.' I can successfully retrieve data by using the Selenium library to access https://poe.com/, but it requires writing the cookies: p-b, __cf_bm, and cf_clearance in advance."
is what @31702160136 is saying, but what exactly does this mean?

我对这方面不太熟悉，我认为是需要加载JavaScript后才可以获得到数据，我们通常使用代码进行GET请求页面是无法加载JavaScript的，但是使用浏览器就可以加载JavaScript

Have you confirmed that this method works?

[445022889]

我经过排查后发现使用GET访问https://poe.com提示403，Enable JavaScript and cookies to continue。我使用selenium库去访问https://poe.com可以成功获取到数据，但需要提前写入cookie：p-b、__cf_bm、cf_clearance才可以

昨天用golang的chromedp访问好像也没问题，只需要加入p-b就行

[31702160136]

我经过排查后发现使用GET访问https://poe.com提示403，启用JavaScript和cookie才能继续。我使用selenium去库访问https://poe.com可以成功获取到数据，但需要提前写入cookie ：pb、__cf_bm、cf_clearance才可以

昨天用golang的chromedp访问没什么问题，只需要加入pb就行

我试了一下，确实只需要pb，之前我需要加入__cf_bm和cf_clearance是因为浏览器有缓存

[31702160136]

“经过调查，我发现使用 GET 方法访问https://poe.com/会提示 403 错误，'启用 JavaScript 和 cookies 即可继续'。我可以使用 Selenium 库访问https://poe.com/成功检索数据，但需要提前写入cookie：pb、__cf_bm 和 cf_clearance。”
是什么@31702160136是这么说的，但这到底是什么意思呢？

我对这方面还不太熟悉，我认为是需要加载JavaScript后才能获得数据，通常使用我们的代码进行GET请求页面是无法加载JavaScript的，但是使用浏览器就可以加载JavaScript

你确认这个方法有效吗？

确实能有效获得到数据，但是又遇到了新的问题，提示’document‘ is not defined，可能是formkey算法变了

[tejas36026]

me too getting the same

[mahmuod711]

Same Problem

[YUFOHON]

same

[sycapple]

same

[RamyGomaa]

same on an older version too, looks like it's a general problem, not a version specific one.

[asce0]

does anyone get any idea about how to solve this error?

[kraravind]

getting the same issue, i went to poe, got the token value through inspect and ran the poe.Client and passed the token. This is the error below, please help:
File ~\anaconda3\Lib\site-packages\poe_init_.py:156, in Client.init(self, token, proxy, headers, device_id, client_identifier, formkey)
153 self.formkey_salt = None
154 self.formkey = formkey
--> 156 self.connect_ws()

File ~\anaconda3\Lib\site-packages\poe_init_.py:446, in Client.connect_ws(self, timeout)
443 self.ws_connected = False
445 if self.connect_count % 5 == 0:
--> 446 self.setup_connection()
448 self.connect_count += 1
450 ws = websocket.WebSocketApp(
451 self.get_websocket_url(),
452 header={"User-Agent": user_agent},
(...)
456 on_close=self.on_ws_close
457 )

File ~\anaconda3\Lib\site-packages\poe_init_.py:182, in Client.setup_connection(self)
179 self.setup_count += 1
181 self.ws_domain = f"tch{random.randint(1, 1e6)}"
--> 182 self.next_data = self.get_next_data(overwrite_vars=True)
183 self.channel = self.get_channel_data()
185 self.gql_headers = {
186 "poe-formkey": self.formkey,
187 "poe-tchannel": self.channel["channel"],
188 }

File ~\anaconda3\Lib\site-packages\poe_init_.py:244, in Client.get_next_data(self, overwrite_vars)
241 def get_next_data(self, overwrite_vars=False):
242 logger.info("Downloading next_data...")
--> 244 r = request_with_retries(self.session.get, self.home_url)
245 json_regex = r'<script id="__NEXT_DATA__" type="application\/json">(.+?)</script>'
246 json_text = re.search(json_regex, r.text).group(1)

File ~\anaconda3\Lib\site-packages\poe_init_.py:84, in request_with_retries(method, *args, **kwargs)
81 raise RuntimeError("Invalid or missing token.")
82 logger.warn(f"Server returned a status code of {r.status_code} while downloading {url}. Retrying ({i+1}/{attempts})...")
---> 84 raise RuntimeError(f"Failed to download {url} too many times.")

RuntimeError: Failed to download https://poe.com/ too many times.

[teclectic]

Hi Guys,

Started getting this error on Saturday, I sent an email to Poe and they said that the specific IP has been blocked for 7 days. I have tried using my same token from another IP, I tried getting another account with a different token and all have come up with the same 403 error. I think that they must have some antiscraping software running which is flagging the API requests. Has anyone tried using sock5 proxies ? Any better ?

[4781121]

作者也不出来说两句, 也不知道是能解决还是不能解决

[akl7777777]

解决应该是能解决,但是一解决就会被盯上,估计也是难

[teclectic]

Hi Guys,
Started getting this error on Saturday, I sent an email to Poe and they said that the specific IP has been blocked for 7 days. I have tried using my same token from another IP, I tried getting another account with a different token and all have come up with the same 403 error. I think that they must have some antiscraping software running which is flagging the API requests. Has anyone tried using sock5 proxies ? Any better ?

Did you try new account with new ip ? And if they are banning or they have some mechanism in place and even if new ip with new account works it will be useless as it will be limited again right ?

I have tried with diffrent ip and diffrent account key still the error

Yes still get the same error. Any luck with proxies or selenium?

Hi Guys,
Started getting this error on Saturday, I sent an email to Poe and they said that the specific IP has been blocked for 7 days. I have tried using my same token from another IP, I tried getting another account with a different token and all have come up with the same 403 error. I think that they must have some antiscraping software running which is flagging the API requests. Has anyone tried using sock5 proxies ? Any better ?

Did you try new account with new ip ? And if they are banning or they have some mechanism in place and even if new ip with new account works it will be useless as it will be limited again right ?
I have tried with diffrent ip and diffrent account key still the error

Yes still get the same error. Any luck with proxies or selenium?

have you tried to set a Custom Device ID with new IP and account ?
from the README:

Setting a Custom Device ID:

If you want to change the device ID that is being spoofed, you can use the poe.set_device_id, which accepts the following arguments:

user_id - The user ID of the account you want to change the device ID for. The user ID can be found at client.viewer["poeUser"]["id"].
device_id - The new device ID. This is a 32 character UUID string in the following format: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

poe.set_device_id("UGMlVXqlcLYyMOATMDsKNTMz", "6d659b04-043a-41f8-97c7-fb7d7fe9ad34")

The device IDs are saved to ~/.config/poe-api/device_id.json on Unix-like systems, and C:\Users<user>\AppData\Roaming\poe-api\device_id.json on Windows.

Additionally, the poe.get_device_id function or client.device_id can be used to retrieve the saved device ID.

poe.get_device_id("UGMlVXqlcLYyMOATMDsKNTMz")
#6d659b04-043a-41f8-97c7-fb7d7fe9ad34

client.device_id
#6d659b04-043a-41f8-97c7-fb7d7fe9ad34"

[pythonhook]

How to resolve this, guys

[wangyu1997]

感觉解决不了 放弃了一个月了 作者好像没有要解决的意思 换个库吧兄弟

[pythonhook]

有什么好的库推荐下呗

…

---Original--- From: "Wang ***@***.***&gt; Date: Wed, Aug 9, 2023 15:20 PM To: ***@***.***&gt;; Cc: ***@***.******@***.***&gt;; Subject: Re: [ading2210/poe-api] Server returned a status code of 403 whiledownloading https://poe. com (Issue #218) 感觉解决不了 放弃了一个月了 作者好像没有要解决的意思 换个库吧兄弟 — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: ***@***.***&gt;

[445022889]

目前自己golang用的无头浏览器，劫持ws消息，勉强能用吧，不用一堆js，希望作者早日修复bug

有什么好的库推荐下呗
…
---Original--- From: "Wang @.> Date: Wed, Aug 9, 2023 15:20 PM To: @.>; Cc: @.@.>; Subject: Re: [ading2210/poe-api] Server returned a status code of 403 whiledownloading https://poe. com (Issue #218) 感觉解决不了 放弃了一个月了 作者好像没有要解决的意思 换个库吧兄弟 — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>

[teclectic]

Is it worth building the headless selenium solution in your opinion into poe-api or otherwise? Did you also use proxies?

…

On Wed, 9 Aug 2023, 09:18 Djing, ***@***.***> wrote: 目前自己golang用的无头浏览器，劫持ws消息，勉强能用吧，不用一堆js，希望作者早日修复bug 有什么好的库推荐下呗 … <#m_-2824118766624701009_> ---Original--- From: "Wang *@*.*> Date: Wed, Aug 9, 2023 15:20 PM To: @.*>; Cc: *@***@***.****>; Subject: Re: [ading2210/poe-api] Server returned a status code of 403 whiledownloading https://poe. com (Issue #218 <#218>) 感觉解决不了 放弃了一个月了 作者好像没有要解决的意思 换个库吧兄弟 — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: *@* .***> — Reply to this email directly, view it on GitHub <#218 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHHFWJ22NPLATJYFQPWGZZ3XUNBUJANCNFSM6AAAAAA3ELAPUI> . You are receiving this because you commented.Message ID: ***@***.***>

[ongspxm]

curl 'https://poe.com/' \
  -H 'authority: poe.com' \
  -H 'cookie: p-b=<token>' \
  -H 'user-agent: <copy user-agent>' --http2

after testing with curl

• found that both "authority" and "user-agent" header required
• http1.1 doesnt work (tested with --http1.1 on curl and it gave the error), http2 does

but...

• requests is a 1.1 only library, maybe httpx might work?

can anybody confirm this?

[claell]

@ading2210 You added the help wanted label. Is there anything you need right now? Or are you just busy with other stuff right now?

[huyremy]

@ading2210 You added the help wanted label. Is there anything you need right now? Or are you just busy with other stuff right now?

and...I feel something like dead project.

[fanxyu]

希望作者能注意到这个问题，解决一下

[claell]

Alright, thanks for the update! If your understanding is correct, that would be rather sad to see.

@canxin121 I assume you experience the same problem with your async client?

[canxin121]

not the same promblem, i can successfuly download the poe.com, but i can't send query successfullt.

[mak448a]

My current understanding of this bug is that Poe has started enforcing some sort of cloudflare-based verification which requires a full web browser.

Maybe selenium would work for this project?

[claell]

@mak448a see #195

[claell]

@canxin121 That's interesting findings. Apparently, a web browser might only be required to retrieve the formkey, then (if even that)? That might leave some room for a hybrid solution where the formkey can get extracted through browser automation and following that the API can still be used.

[4781121]

@ading2210 可以参考这个项目, 他解决了目前的问题, https://github.com/canxin121/Async-Poe-Client

[ading2210]

not the same promblem, i can successfuly download the poe.com, but i can't send query successfullt.
@canxin121

Are you able to port those changes from your async client to this library?

[GoAwayNow]

@ongspxm
I got a 200 returned with a full set of browser cookies and the user-agent. I didn't use the authority header and made the request with http1.1. I don't know if it's the same for everyone, but it seems to be entirely a Cloudflare issue for me.

[tic-top]

寄！

[Iamprashanth-1]

try g4f pypi