Deadlock reading from R2 and writing to a ZIP writer about workers

[syumai]

I've confirmed the problem happens when writing large file to zip. I'll do more investigation.

[syumai]

@eliezedeck I found that response streaming based on io.Pipe (in handler.go) was causing the deadlock.
If you use a buffer for zip writing and stream the response from there, you won't have any problems.
see:
https://github.com/syumai/workers-playground/blob/3877c5bac5c6e747b7b5ff258c18d71ff3ace5a9/syumai-workers-repro-issue-103/main.go#L33-L37

However, this is not a complete solution as there may be very large files and require a large amount of memory.
Therefore, I'll continue to investigate.

[syumai]

@eliezedeck Perhaps https://github.com/syumai/workers/releases/tag/v0.25.0 fixed the problem.
Please check it!

Now, my reproduction code is working.

[eliezedeck]

Wonderful @syumai!

It works flawlessly now. You made the library even better by fixing streaming I see. Just golden!

[syumai]

@eliezedeck Thank you for filing an issue!
I'm sorry, but I don't know what causes this issue. It's probably correct to say that reading from R2 is the cause. This could either be due to file size or stream conversion. I will investigate.
If you could share the repository for reproduction (including dependencies, go.mod, etc.), it would be helpful for investigation.

[eliezedeck]

I'm not allowed to directly share the repo. However, I'll do my best to give the max info I can to help.

go.mod:

module workersgo

go 1.22.2

require (
	github.com/labstack/echo/v4 v4.11.4
	github.com/syumai/workers v0.23.3
)

require (
	github.com/gabriel-vasile/mimetype v1.4.3 // indirect
	github.com/go-playground/locales v0.14.1 // indirect
	github.com/go-playground/mold/v4 v4.5.0 // indirect
	github.com/go-playground/universal-translator v0.18.1 // indirect
	github.com/go-playground/validator/v10 v10.19.0 // indirect
	github.com/gosimple/slug v1.13.1 // indirect
	github.com/gosimple/unidecode v1.0.1 // indirect
	github.com/klauspost/compress v1.17.7 // indirect
	github.com/labstack/gommon v0.4.2 // indirect
	github.com/leodido/go-urn v1.4.0 // indirect
	github.com/mattn/go-colorable v0.1.13 // indirect
	github.com/mattn/go-isatty v0.0.20 // indirect
	github.com/segmentio/go-camelcase v0.0.0-20160726192923-7085f1e3c734 // indirect
	github.com/segmentio/go-snakecase v1.2.0 // indirect
	github.com/tidwall/gjson v1.14.2 // indirect
	github.com/tidwall/match v1.1.1 // indirect
	github.com/tidwall/pretty v1.2.0 // indirect
	github.com/tidwall/sjson v1.2.5 // indirect
	github.com/valyala/bytebufferpool v1.0.0 // indirect
	github.com/valyala/fasttemplate v1.2.2 // indirect
	golang.org/x/crypto v0.22.0 // indirect
	golang.org/x/net v0.21.0 // indirect
	golang.org/x/sys v0.19.0 // indirect
	golang.org/x/text v0.14.0 // indirect
)

wrangler.toml:

name = "go-services"
main = "./build/worker.mjs"
compatibility_date = "2024-04-05"
workers_dev = false
send_metrics = true

account_id = "xxxxxxxxxxxxxxxx"

[build]
command = "make build"

[env.dev]
name = "go-services"
vars = { ENV = "dev" }
r2_buckets = [
    { binding = "EXPENSE_DOCUMENTS", bucket_name = "expense-documents", preview_bucket_name = "expense-documents-preview" },
    { binding = "PROFILE_PHOTOS", bucket_name = "profile-photos", preview_bucket_name = "profile-photos-preview" }
]

[env.staging]
name = "staging-go-services"
vars = { ENV = "staging" }
r2_buckets = [
    { binding = "EXPENSE_DOCUMENTS", bucket_name = "staging-expense-documents", preview_bucket_name = "staging-expense-documents-preview" },
    { binding = "PROFILE_PHOTOS", bucket_name = "staging-profile-photos", preview_bucket_name = "staging-profile-photos-preview" }
]

[env.production]
name = "go-services"
vars = { ENV = "production" }
r2_buckets = [
    { binding = "EXPENSE_DOCUMENTS", bucket_name = "expense-documents", preview_bucket_name = "expense-documents-preview" },
    { binding = "PROFILE_PHOTOS", bucket_name = "profile-photos", preview_bucket_name = "profile-photos-preview" }
]

Makefile:

.PHONY: dev
dev:
	wrangler dev --port 9899 --inspector-port 9232

.PHONY: build
build:
	go run github.com/syumai/workers/cmd/[email protected] -mode=go
	GOOS=js GOARCH=wasm go build -o ./build/app.wasm .

.PHONY: deploy
deploy:
	wrangler deploy --env staging --minify --keep-vars

The script is simple enough, with this section as start:

package main

import (
	"compress/flate"
	"context"
	"errors"
	"fmt"
	"io"
	"math/rand"
	"net/http"
	"time"

	"github.com/klauspost/compress/zip"
	"github.com/labstack/echo/v4"
	"github.com/syumai/workers"
	"github.com/syumai/workers/cloudflare"
)

func main() {
	e := echo.New()

	e.GET("/", func(c echo.Context) error {
		return c.String(http.StatusOK, "Hello, World!")
	})

For reference, here is the test file that is generating garbage ZIP (but still accurate and correct), this is the one that does not crash:

	e.GET("/services/test/generate-zip", func(c echo.Context) error {
		// Prepare to stream a ZIP file
		writer := zip.NewWriter(c.Response())
		writer.RegisterCompressor(zip.Deflate, func(out io.Writer) (io.WriteCloser, error) {
			return flate.NewWriter(out, flate.NoCompression)
		})
		defer func(writer *zip.Writer) {
			_ = writer.Close()
		}(writer)

		// Start streaming
		c.Response().Header().Set(echo.HeaderContentType, "application/zip")
		c.Response().Header().Set(echo.HeaderContentDisposition, "attachment; filename=dummy.zip")
		c.Response().WriteHeader(http.StatusOK)

		// Seed the random number generator
		rand.Seed(time.Now().UnixNano())

		// Generate dummy files
		for i := 0; i < 20; i++ {
			// Generate a random string of a random length
			length := rand.Intn(100) // Change this value to adjust the maximum length
			randomString := make([]byte, length)
			for i := range randomString {
				randomString[i] = byte(rand.Intn(26) + 97) // Generate a random lowercase letter
			}

			// Add the file to the ZIP
			fileWriter, err := writer.Create(fmt.Sprintf("file%d.txt", i))
			if err != nil {
				c.Logger().Errorf("failed to create file 'file%d.txt' in ZIP: %v", i, err)
				return err
			}

			// Write the random string to the file
			if _, err := fileWriter.Write(randomString); err != nil {
				c.Logger().Errorf("failed to write to file 'file%d.txt' in ZIP: %v", i, err)
				return err
			}
		}

		return nil
	})

Hope that helps.

[syumai]

@eliezedeck Thank you! I'll check this later.

[syumai]

works

* R2Object<ReadableStream> -> zip.Writer(bytes.Buffer)
* bytes.Buffer -> ResponseWriter<ReadableStream>

not works

* R2Object<ReadableStream> -> zip.Writer(ResponseWriter<ReadableStream>)

Probably, a direct connection between ReadableStreams on the JS side is considered blocked on Go runtime if both sides of the stream are waiting on a Promise.