requests.exceptions.HTTPError: 422 Client Error: Unprocessable Entity for url: about pleroma-bot

[robertoszek]

I keep getting HTTP 422 for https://respublicae.eu/@EP_Defence possibly due to https://nitter.eu/EP_Defence/status/1513526853532897283#m . Could it be because there are 4+2 images there considering the quoted tweet?

@nemobis Right! Nice catch, Mastodon only allows 4 images per post, possibly that's the cause. (Misskey and Pleroma allow 16).

I've added the upper limit for Mastodon to 1.0.3rc13, feel free to try it out and see if you still get an HTTP 422 for the same tweet:

pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple pleroma-bot==1.0.3rc13

[vserbu]

It looks like it's working now!

[vserbu]

Thank you for quick response :)

[vserbu]

It's working, you were absolutely - right! tnx!

[robertoszek]

When truncating a toot, it might be worth truncating before the signature, to avoid breaking links: see https://respublicae.eu/@Europarl_DE/108068732034451913 for an example.

@nemobis
Oh, right! I missed how this would happen when a signature and original date are both enabled.
it should be fixed on 1.0.3rc8, feel free to give it a spin and see if the changes solved it:

pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple pleroma-bot==1.0.3rc8

[nemobis]

Il 03/04/22 20:23, robertoszek ha scritto:

it should be fixed on 1.0.3rc8, feel free to give it a spin and see if the changes solved it:

Yes, thank you! All good on 1.0.3rc9.

[robertoszek]

Hi @vserbu !
What version of pleroma-bot are you using?

[vserbu]

I'm still on the 1.0.1 version. How do I update on newer version?

[robertoszek]

If you installed it using pip, you can upgrade to the latest version using this command:

pip install pleroma-bot --upgrade

[vserbu]

I did it, but the error is the same. It starts to post tweets and then blocks on one (in that example it was on second one) and that is it. Is there any cache to clear or something? I tried to limit the size of the tweet because I taught maybe it could be some big video or image, but it is not.

[robertoszek]

Would you mind sharing the output of your second try?

And maybe also your current config file (with the tokens or any sensitive info removed).

[vserbu]

Yes, I'm seeing it is somewhat different:

...
ℹ 2022-02-23 08:53:03,014 - pleroma_bot - INFO - Processing user: 2139
ℹ 2022-02-23 08:53:04,529 - pleroma_bot - INFO - tweets gathered: 4
ℹ 2022-02-23 08:53:05,787 - pleroma_bot - INFO - tweets to post: 4
ℹ 2022-02-23 08:53:05,787 - pleroma_bot - INFO - (1/4)
✖ 2022-02-23 08:53:05,930 - pleroma_bot - ERROR - Exception occurred
Unprocessable Entity
Validation failed: File content type is invalid, File is invalid
File: /home/mastodon/tweets/1496229640171630596/0.mp4 (_pleroma.py:127)
⚠ 2022-02-23 08:53:05,931 - pleroma_bot - WARNING - Error uploading media: 422 (_pleroma.py:134)
✖ 2022-02-23 08:53:05,984 - pleroma_bot - ERROR - Exception occurred (cli.py:536)
Traceback (most recent call last):
File "/home/mastodon/.local/lib/python3.8/site-packages/pleroma_bot/cli.py", line 513, in main
post_id = user.post_pleroma(
File "/home/mastodon/.local/lib/python3.8/site-packages/pleroma_bot/_pleroma.py", line 167, in post_pleroma
response.raise_for_status()
File "/home/mastodon/.local/lib/python3.8/site-packages/requests/models.py", line 960, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 422 Client Error: Unprocessable Entity for url: https://salocha.online/api/v1/statuses
...

[robertoszek]

It looks like the Mastodon instance is probably reporting the wrong type for this attachment:
https://video.twimg.com/tweet_video/FMOs2sNXMAEWLxe.mp4

$ file 0.mp4 
0.mp4: ISO Media

Hence, failing the validation. The instance wrongly expects an ISO (.iso) file (or something else than an .mp4) to be uploaded based on running the file utility.

pleroma-bot can't do much here, the issue lies on the instance's file command reporting the wrong type. It could be that it's an old version and updating to a more recent version would fix it.

This shouldn't block you from using the bot normally, for these cases we skip the upload of the problematic media and try to publish the tweet's text anyway.
But in this case, the tweet's text is empty and it only has the attachment, so it also fails to post an empty status.

[robertoszek]

I've published a release candidate version which won't fail due to the empty status (it will skip posting it), so at least it's not a fatal error and you could continue using the bot for publishing the rest of the tweets if you so choose.
If you want to try it you can install it like so:

pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple pleroma-bot==1.0.3rc2

However, the file issue will still occur, the problematic attachment won't be uploaded because the Mastodon instance won't recognize it as an mp4 media file, there's not much the bot can do about that.

[vserbu]

I'll give it a try, tnx :)

[vserbu]

It's working, but I have a new problem. Visibility is suddenly not public even dough I stated visibility: "public" already. Now the new tweets are not in LOCAL timeline, only in my HOME timeline.

[robertoszek]

Nice catch!
I somehow missed an edge-case where the visibility is defined as a global value, sorry about that!
Feel free to give it a try with 1.0.3rc3:

pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple pleroma-bot==1.0.3rc3

[vserbu]

Still the same problem :(

[robertoszek]

Really? That's odd, I thought I had found the issue.
Could you share your current config? (without the tokens or sensitive info)

[vserbu]

pleroma_base_url: https://salocha.online
max_tweets: 40
twitter_token: xxxxx
users:
- twitter_username: TheSpaceshipper
  pleroma_username: 1771
  pleroma_token: xxx
- twitter_username: TrekNewsNet
  pleroma_username: 1792
  pleroma_token: xxx
- twitter_username: retroscifiart
  pleroma_username: 1808
  pleroma_token: xxx
- twitter_username: 70sscifi
  pleroma_username: 1812
  pleroma_token: xxx
- twitter_username: tosgifs
  pleroma_username: 1833
  pleroma_token: xxx
- twitter_username: ultrakillblast
  pleroma_username: 2139
  pleroma_token: xxx
visibility: "public"
include_replies: false
file_max_size: 5MB
include_rts: false

[robertoszek]

Great!
I was about to say that I could not seem to be able to reproduce the visibility issue on my side with your config on 1.0.3rc3 haha

Maybe I should've clarified that it only would apply to new posts published after upgrading to 1.0.3rc3 (the ones published already would've kept the wrong visibility).

I'll include these fixes on the next stable release as well.
Have a nice day!

[robertoszek]

And thank you for finding these issues and reporting them as well!

[vserbu]

I think we have another Issue regarding this.
The script stuck at:

ℹ 2022-02-23 20:03:35,112 - pleroma_bot - INFO - config path: /home/mastodon/config.yml 
ℹ 2022-02-23 20:03:35,113 - pleroma_bot - INFO - tweets temp folder: /home/mastodon/tweets 
ℹ 2022-02-23 20:03:35,119 - pleroma_bot - INFO - ====================================== 
ℹ 2022-02-23 20:03:35,120 - pleroma_bot - INFO - Processing user:       1771 
ℹ 2022-02-23 20:03:38,831 - pleroma_bot - INFO - tweets gathered:        18 
Processing tweets...  |

And that is it....stuck.

When I removed that account (#) everything else went ok.

[robertoszek]

I may have an idea of what's happening there.
Does running it with this version make any difference?

pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple pleroma-bot==1.0.3rc5

[vserbu]

It is not stucked any more but we are back with the similar error as on the beggining.

ℹ 2022-02-24 10:01:19,351 - pleroma_bot - INFO - config path: /home/mastodon/config.yml 
ℹ 2022-02-24 10:01:19,351 - pleroma_bot - INFO - tweets temp folder: /home/mastodon/tweets 
ℹ 2022-02-24 10:01:19,364 - pleroma_bot - INFO - ====================================== 
ℹ 2022-02-24 10:01:19,364 - pleroma_bot - INFO - Processing user:       1771 
ℹ 2022-02-24 10:01:21,926 - pleroma_bot - INFO - tweets gathered:        34 
ℹ 2022-02-24 10:01:24,416 - pleroma_bot - INFO - tweets to post:         7 
ℹ 2022-02-24 10:01:24,417 - pleroma_bot - INFO - (1/7) 
✖ 2022-02-24 10:01:24,871 - pleroma_bot - ERROR - Exception occurred (cli.py:537) 
Traceback (most recent call last):
  File "/home/mastodon/.local/lib/python3.8/site-packages/pleroma_bot/cli.py", line 514, in main
    post_id = user.post_pleroma(
  File "/home/mastodon/.local/lib/python3.8/site-packages/pleroma_bot/_pleroma.py", line 170, in post_pleroma
    response.raise_for_status()
  File "/home/mastodon/.local/lib/python3.8/site-packages/requests/models.py", line 960, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 422 Client Error: Unprocessable Entity for url: https://salocha.online/api/v1/statuses

[robertoszek]

Hmm... I'm wondering if you're hitting Mastodon's character limit (500 per toot) with a quoted tweet.

Would you mind testing out my theory by running this version?:

pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple pleroma-bot==1.0.3rc6

[robertoszek]

Nice!
I should be the one thanking you :)
Thanks to your reporting we found these issues and the fixes will be included on the next stable release.
Cheers!

[nemobis]

Hmm... I'm wondering if you're hitting Mastodon's character limit (500 per toot) with a quoted tweet.

When truncating a toot, it might be worth truncating before the signature, to avoid breaking links: see https://respublicae.eu/@Europarl_DE/108068732034451913 for an example.

I see that's actually what you were trying to do, not sure what went wrong:
8f44c53#diff-46774f4895c169807f7a3d85cc4ebc34a18ac37e7f4b2ed7306d5ef6977e59adR156

[nemobis]

I'm still seeing some HTTP 422. Consistently for this one account:

ℹ 2022-04-10 09:29:33,942 - pleroma_bot - INFO - Processing user:       108070048904140658
ℹ 2022-04-10 09:29:36,909 - pleroma_bot - INFO - tweets gathered:        49
ℹ 2022-04-10 09:29:44,172 - pleroma_bot - INFO - tweets to post:         49
ℹ 2022-04-10 09:29:44,173 - pleroma_bot - INFO - (1/49)
✖ 2022-04-10 09:29:56,582 - pleroma_bot - ERROR - Exception occurred for user, skipping... (cli.py:556) 
Traceback (most recent call last):
  File "/home/7/federico/mastodon/lib/python3.7/site-packages/pleroma_bot/cli.py", line 536, in main
    tweet["possibly_sensitive"],
  File "/home/7/federico/mastodon/lib/python3.7/site-packages/pleroma_bot/_utils.py", line 605, in post
    post_id = self.post_pleroma(tweet, poll, sensitive)
  File "/home/7/federico/mastodon/lib/python3.7/site-packages/pleroma_bot/_pleroma.py", line 170, in post_pleroma
    response.raise_for_status()
  File "/home/7/federico/mastodon/lib/python3.7/site-packages/requests/models.py", line 960, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 422 Client Error: Unprocessable Entity for url: https://respublicae.eu/api/v1/statuses

I think it might be one of these tweets (cw: controversial political content):

• https://nitter.eu/vox_es/status/1512018178221690886#m
• https://nitter.eu/VOX_Europa_/status/1512022775166345223#m

[robertoszek]

@nemobis
For those specific 2 tweets you provided, I've not been able to reproduce the issue, they posted fine on a Mastodon instance on my testing (with or without signature/original date too).

Maybe the culprit is some other tweet.
You can try running the bot in verbose mode and see if it helps to narrow it down.

[nemobis]

Il 10/04/22 23:40, robertoszek ha scritto:

Maybe the culprit is some other tweet.

Yeah I don't know, I'll add some logging locally to find out.

[nemobis]

I keep getting HTTP 422 for https://respublicae.eu/@EP_Defence possibly due to https://nitter.eu/EP_Defence/status/1513526853532897283#m . Could it be because there are 4+2 images there considering the quoted tweet?

[nemobis]

Some quote-tweets with 5+ photos are still getting an HTTP 422 on the most recent release, though I'm not quite sure why (I even hacked the Mastodon code to raise the limit to 8).

Culprit seems to be https://nitter.eu/andreykovatchev/status/1534110522722996224#m

ℹ 2022-07-30 01:02:44,385 - pleroma_bot - INFO - Processing user:       108069143892397143
ℹ 2022-07-30 01:02:46,464 - pleroma_bot - INFO - tweets gathered:        7
ℹ 2022-07-30 01:02:50,080 - pleroma_bot - INFO - tweets to post:         7
ℹ 2022-07-30 01:02:50,081 - pleroma_bot - INFO - (1/7)
✖ 2022-07-30 01:02:57,280 - pleroma_bot - ERROR - Exception occurred for user, skipping... (cli.py:574) 
Traceback (most recent call last):
  File "/home/7/federico/mastodon/lib/python3.7/site-packages/pleroma_bot/cli.py", line 554, in main
    tweets_to_post["media_processed"]
  File "/home/7/federico/mastodon/lib/python3.7/site-packages/pleroma_bot/_utils.py", line 620, in post
    post_id = self.post_pleroma(tweet, poll, sensitive, media)
  File "/home/7/federico/mastodon/lib/python3.7/site-packages/pleroma_bot/_pleroma.py", line 192, in post_pleroma
    response.raise_for_status()
  File "/home/7/federico/mastodon/lib/python3.7/site-packages/requests/models.py", line 960, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 422 Client Error: Unprocessable Entity for url: https://respublicae.eu/api/v1/statuses

[nemobis]

On the other hand a bunch of posts which previously failed are now succeeding, like https://respublicae.eu/@Cedefop/108732982556276877 from https://nitter.eu/Cedefop/status/1542082206641840129 . 🤷

[nemobis]

I'm getting an HTTP 422 while trying to post (I think) this https://nitter.it/NatalieAmiri/status/1586320608538902528#m , a tweet with a video quoting another tweet with a video.

It would be easier to debug if the exception printed the original tweet ID too.

[nemobis]

A RT https://nitter.it/EmmanuelMacron/status/1591869166046380035#m failed too, maybe any quote tweet with video fails?

[robertoszek]

Ah, for the first tweet you mentioned it's probably because I don't think Mastodon supports multiple videos on the same post:
https://user-images.githubusercontent.com/1080963/202745994-93c67a8f-417e-465c-9f0a-0f69c61e7085.png

Hmm I need to think what's the best way to handle that. I assume it's better to drop the video on the quoted/retweeted tweet instead of the one attached to the user you're mirroring. You'll lose some context either way, which is a shame.

Ah, I see it's the same problem for the other tweet too (multiple videos):
https://user-images.githubusercontent.com/1080963/202748467-169040d2-e560-4b2d-9a31-ff607e7035a7.png

I'll open a new issue to keep track of it, as the "422 Unprocessable Entity" error is too generic and could be multiple things causing it.

[nemobis]

Il 18/11/22 18:06, robertoszek ha scritto:

I'll open a new issue to keep track of it, as the "422 Unprocessable Entity" error is too generic and could be multiple things causing it.

Thanks!