Regression in v4.2+ about git-release

[anton-yurchenko]

This should be fixed at v4.2.2, but as you understand it is impossible to simulate during the development process, so please re-open the ticket if you still see this problem.

In case not, it would be very helpful to see a log of git-release recovery from a network issue.

Thanks @kongsgard, @rgriebl for raising this. 👑

[rgriebl]

Just FYI: I just created a new release and it worked perfectly fine this time:
https://github.com/rgriebl/brickstore/runs/5118820510?check_suite_focus=true#step:7:23

[anton-yurchenko]

Thank you @kongsgard 👑

That is strange as v4.2.1 was released because of transient network errors like connection reset by peer. Its retry mechanism supposes to work around such problems.

Are you on GitHub Enterprise?

Is that step running on a custom runner or GitHub managed?

Please provide a full execution log, masking the bare minimum like renaming project names and assets filenames.

[kongsgard]

I'm on GitHub Team, I believe. No custom runner involved.

Run docker://antonyurchenko/git-release:v4
/usr/bin/docker run --name antonyurchenkogitreleasev4_16ca75 --label 9916a7 --workdir /github/workspace --rm -e GITHUB_TOKEN -e PRE_RELEASE -e CHANGELOG_FILE -e ALLOW_EMPTY_CHANGELOG -e RELEASE_NAME_PREFIX -e INPUT_ARGS -e HOME -e GITHUB_JOB -e GITHUB_REF -e GITHUB_SHA -e GITHUB_REPOSITORY -e GITHUB_REPOSITORY_OWNER -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RETENTION_DAYS -e GITHUB_RUN_ATTEMPT -e GITHUB_ACTOR -e GITHUB_WORKFLOW -e GITHUB_HEAD_REF -e GITHUB_BASE_REF -e GITHUB_EVENT_NAME -e GITHUB_SERVER_URL -e GITHUB_API_URL -e GITHUB_GRAPHQL_URL -e GITHUB_REF_NAME -e GITHUB_REF_PROTECTED -e GITHUB_REF_TYPE -e GITHUB_WORKSPACE -e GITHUB_ACTION -e GITHUB_EVENT_PATH -e GITHUB_ACTION_REPOSITORY -e GITHUB_ACTION_REF -e GITHUB_PATH -e GITHUB_ENV -e RUNNER_OS -e RUNNER_ARCH -e RUNNER_NAME -e RUNNER_TOOL_CACHE -e RUNNER_TEMP -e RUNNER_WORKSPACE -e ACTIONS_RUNTIME_URL -e ACTIONS_RUNTIME_TOKEN -e ACTIONS_CACHE_URL -e GITHUB_ACTIONS=true -e CI=true -v "/var/run/docker.sock":"/var/run/docker.sock" -v "/home/runner/work/_temp/_github_home":"/github/home" -v "/home/runner/work/_temp/_github_workflow":"/github/workflow" -v "/home/runner/work/_temp/_runner_file_commands":"/github/file_commands" -v "/home/runner/work/app-name/app-name":"/github/workspace" antonyurchenko/git-release:v4 ./build/*.zip
DEBUG git-release v4.2.1                           
INFO creating Release: 2022.0.0 release           
INFO release created successfully 🎉               
INFO uploading asset                               asset=asset.zip
WARNING error uploading asset: Post "https://uploads.github.com/repos/org-name/app-name/releases/57557111/assets?name=asset.zip": write tcp 172.17.0.2:56898->140.82.112.13:443: write: connection reset by peer  asset=asset.zip
INFO retrying (2/4) uploading asset in 9 seconds   asset=asset.zip
WARNING error uploading asset: Post "https://uploads.github.com/repos/org-name/app-name/releases/57557111/assets?name=asset.zip": http2: Transport: cannot retry err [stream error: stream ID 1; REFUSED_STREAM; received from peer] after Request.Body was written; define Request.GetBody to avoid this error  asset=asset.zip
INFO retrying (3/4) uploading asset in 27 seconds  asset=asset.zip
WARNING error uploading asset: POST https://uploads.github.com/repos/org-name/app-name/releases/57557111/assets?name=asset.zip: 422 Validation Failed [{Resource:ReleaseAsset Field:name Code:already_exists Message:}]  asset=asset.zip
INFO retrying (4/4) uploading asset in 81 seconds  asset=asset.zip
WARNING error uploading asset: POST https://uploads.github.com/repos/org-name/app-name/releases/57557111/assets?name=asset.zip: 422 Validation Failed [{Resource:ReleaseAsset Field:name Code:already_exists Message:}]  asset=asset.zip
ERROR maximum attempts reached uploading asset: asset.zip 
FATAL error uploading assets

[rgriebl]

I have a similar problem. The log of the failed release is here:

• 1st attempt: https://github.com/rgriebl/brickstore/runs/4939289547?check_suite_focus=true
• 2nd attempt: https://github.com/rgriebl/brickstore/runs/4939603547?check_suite_focus=true

I first had this problem back in October and now it's happening again.

I'm no Go expert, but I think the code tries to upload all assets in parallel: could it be that having 7 assets in tens-of-MB range is too much to upload in parallel?

[kongsgard]

That is a good point to mention too - the assets I'm uploading are about 200-300 MB in size.

[anton-yurchenko]

I was able to recreate the initial error connection reset by peer by writing an API server that mimics GitHub API and simulating its failures.

The read seems to be happening when underlying storage fails, while write seems to be network-related as it is directly affected by an HTTP request itself.

Here you may see an example of the API server failover simulation because of a storage failure. First there is an error, then the server is unavailable and finally, it's up again after a reboot:

INFO uploading asset                               asset=archive.zip
WARNING error uploading asset: Post "http://localhost:8080/repos/owner/repo/releases/1/assets?name=archive.zip": read tcp [::1]:62946->[::1]:8080: read: connection reset by peer  asset=archive.zip
INFO retrying (2/4) uploading asset in 9 seconds   asset=archive.zip
WARNING error uploading asset: Post "http://localhost:8080/repos/owner/repo/releases/1/assets?name=archive.zip": dial tcp [::1]:8080: connect: connection refused  asset=archive.zip
INFO retrying (3/4) uploading asset in 27 seconds  asset=archive.zip
INFO assets uploaded successfully 🎉 

However, in my implementation, the asset was successfully uploaded on its third attempt, while the actual API throws the following error: 422 Validation Failed [{Resource:ReleaseAsset Field:name Code:already_exists Message:}].

Gladly, I found the following notice on GitHub Docs:

When an upstream failure occurs, you will receive a 502 Bad Gateway status. This may leave an empty asset with a state of starter. It can be safely deleted.

(It might be helpful to inspect assets of an incomplete release)

This makes me think that the retry mechanism is working correctly and what we observe here are transient network issues. Nevertheless, git-release might recover from the already_exists error. It will have to do something like (here):

1. Catch and validate the API response
2. Delete partially uploaded asset
3. Retry asset upload

As per your question regarding the size of the assets (GitHub Docs):

Each file included in a release must be under 2 GB. There is no limit on the total size of a release, nor bandwidth usage.

[anton-yurchenko]

There are two HTTP codes that should result in an asset deletion attempt and retry:

• 422/UnprocessableEntity
• 502/BadGateway