mcbe_getzip: wget read error in headers about mcscripts

[TapeWerm]

On another note mcbe_backup and mcbe_log seem to work with the new update.

[TapeWerm]

May 21 10:18:12 majima systemd[1]: Starting Automatically download Minecraft Bedrock Edition server ZIP...
May 21 10:18:13 majima bash[80484]: 2021-05-21 10:18:13 URL:https://www.minecraft.net/en-us/download/server/bedrock [309070] -> "-" [1]
May 21 10:18:13 majima systemd[1]: mcbe-getzip.service: Succeeded.
May 21 10:18:13 majima systemd[1]: Finished Automatically download Minecraft Bedrock Edition server ZIP.
May 21 11:18:18 majima systemd[1]: Starting Automatically download Minecraft Bedrock Edition server ZIP...
May 21 11:30:10 majima bash[80792]: Read error (Connection reset by peer) in headers.
May 21 11:42:05 majima bash[80792]: Read error (Connection reset by peer) in headers.
May 21 11:54:00 majima bash[80792]: Read error (Connection reset by peer) in headers.
May 21 12:05:57 majima bash[80792]: Read error (Connection reset by peer) in headers.
May 21 12:18:00 majima bash[80792]: Read error (Connection reset by peer) in headers.
May 21 12:29:59 majima bash[80792]: Read error (Connection reset by peer) in headers.
May 21 12:41:57 majima bash[80792]: Read error (Connection reset by peer) in headers.
May 21 12:53:58 majima bash[80792]: Read error (Connection reset by peer) in headers.

Heh, looks like it may have been this way for a while. User agent spoofing may work around this, getjar script is also affected.

[TapeWerm]

wget --user-agent "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0" -nv https://www.minecraft.net/en-us/download/server/bedrock -O -
My face when that succeeded
So I don't really like the idea of spoofing user agents to get around this issue. However, it is clear that if I want this to continue working I need to do this. It's possible this is a Microsoft policy, wget https://www.microsoft.com/en-us/ is 403. It's also possible Mojang thought me and other people hit the server too hard at 10 AM every day. I would like to know what anyone else thinks. Should I spoof the user agent? Should I make a unique user agent for MCscripts, go for maximum meme value with IE11, or spoof a popular browser? User agent hello world even works so it's possible to be honest with them and work around this at the same time. I just have a feeling it may be targeted at the MCscripts project itself and not web scraping in general. Maybe I could add a random multi-minute wait and reduce the mcbe-getzip timer to once a day.

[TapeWerm]

I can download https://minecraft.azureedge.net/bin-linux/bedrock-server-1.17.0.03.zip no user agent so this was likely not targeted at MCscripts. I will probably reduce the frequency of the timer to twice a day and rely on a better offset for it than boot time to something far more random. Since the desired functionality is no delay the delay will move from the timer to a flag for random delay. Any ideas?

[TapeWerm]

From Ubuntu:

When the apt job starts, it will sleep for a random period between 0 and APT::Periodic::RandomSleep seconds. The default value is "1800" so that the script will stall for up to 30 minutes (1800 seconds) so that the mirror servers are not crushed by everyone running their updates all at the same time.

A sleep $((RANDOM % 30)) would mimic this, $RANDOM is not crypto grade random but it's good enough for load relief. At the timer level an option for this exists: https://www.freedesktop.org/software/systemd/man/systemd.timer.html#RandomizedDelaySec=

[TapeWerm]

TheRemote/MinecraftBedrockServer#87 (comment) blames the CDN for this change. That makes the most sense, and possibly something to keep an eye on going forward: Who their CDN is and what the CDN's policies are.
https://community.akamai.com/customers/s/question/0D54R00007GjCANSA3/why-does-akamai-edge-services-sometime-just-not-send-any-response-leaving-the-connection-to-timeout?language=en_US sounds like this is a common issue.