googleapis / gax-go

Google API Extensions for Go.

Google API Extensions for Go

Google API Extensions for Go (gax-go) is a set of modules which aids the development of APIs for clients and servers based on gRPC and Google API conventions.

To install the API extensions, use:

go get -u github.com/googleapis/gax-go/v2

Note: Application code will rarely need to use this library directly, but the code generated automatically from API definition files can use it to simplify code generation and to provide more convenient and idiomatic API surface.

Go Versions

This library requires Go 1.6 or above.

License

BSD - please see LICENSE for more information.

gax-go's Issues

Warning: a recent release failed

The following release PRs may have failed:

• #261 - The release job is 'autorelease: tagged', but expected 'autorelease: published'.

introduce a CallOption to supply a timeout for the context used in Invoke

The main work of googleapis/gapic-generator-go#1206 is to introduce a CallSettings property for the timeout to be used in Invoke, configured via a CallOption like WithTimeout.

support gRPC Wait For Ready

There should be a gax.CallOption called WaitForReady, such that WaitForReady(true) is grpc.FailFast(false), and vice versa.

See https://github.com/grpc/grpc/blob/master/doc/wait-for-ready.md.

Allow invoke to preserve call & context errors for retry until timeout

Currently if Invoke is called and the request is retried until a timeout, it will return a context error only. The underlying error from the service will be lost.

We can use error wrapping to preserve both errors. See googleapis/google-cloud-go#4797 for an example.

Support insecure connections

In order to connect to a local fake server for testing, I'd like to use the grpc.WithInsecure() dial option. This isn't currently possible with gax.

wrong module version in go.mod file makes "go get ..." of the module fail

In a recent commit a bot changed the version of a required module such that it does no longer conform to the go.mod specification. This leads to the following error

go: downloading github.com/googleapis/gax-go/v2 v2.5.0
go: github.com/googleapis/gax-go/[email protected]: parsing go.mod: go.mod:8:2: require google.golang.org/genproto: version "b98a9ff5e252" invalid: must be of the form v1.2.3

Environment details

• Programming language: Go
• OS: Linux
• Language runtime version: go1.18.2
• Package version: googleapis/gax-go/v2

Steps to reproduce

1. go get github.com/googleapis/gax-go/v2

Extensibility points for invoke

I'm starting to look at using gax/invoke for retries in a cloud veneer client. There are some places where I think allowing callers to customize the behavior would be useful.

• I need to inspect an outgoing request to know if it is idempotent and therefore retryable. I don't want to pay the cost of this upfront, and would rather delay until after an error. Perhaps a retryable predicate as a CallSetting that gets used in addition to looking at the RetryCodes codes?
• There is no logging of an error before retrying so the underlying error gets lost if the deadline is eventually exceeded. Logging would be helpful, but it might be nice to have another CallSetting that lets me inspect the error to log, increment a failure counter, or otherwise allow some instrumentation of the code.

panic: proto: file "error.proto" is already registered

Thanks for stopping by to let us know something could be better!

PLEASE READ: If you have a support contract with Google, please create an issue in the support console instead of filing on GitHub. This will ensure a timely response.

Is this a client library issue or a product issue? We will only be able to assist with issues that pertain to the behaviors of this library. If the issue you're experiencing is due to the behavior of the product itself, please visit the Support page to reach the most relevant engineers.

If the support paths suggested above still do not result in a resolution, please provide the following details.

Environment details

• Programming language: Golang
• OS: Ubuntu 20.04
• Language runtime version: go version go1.17.7 linux/amd64
• Package version: v.2.2.0

Steps to reproduce

1. Create new project
2. Add github.com/rookie-ninja/rk-grpc/v2/ dependencies
3. Add cloud.google.com/go/pubsub dependencies
4. Create simple application that utilize pubsub and rk-grpc
5. Got

panic: proto: file "error.proto" is already registered
       previously from: "github.com/rookie-ninja/rk-grpc/v2/boot/error/gen"
       currently from:  "github.com/googleapis/gax-go/v2/apierror/internal/proto"
See https://developers.google.com/protocol-buffers/docs/reference/go/faq#namespace-conflict

Making sure to follow these steps will guarantee the quickest resolution possible.

Thanks!

Remove AppName and AppVersion client options

The AppName and AppVersion options should become a public method on the client, SetGoogleClientInfo(name, version string). Its docstring should say that it is intended for use by Google-written clients.

This is in preparation for moving to a new, shared option package. See #29.

add go mod tidy check to CI

A recent renovate PR, #214, failed and put an improper version into our go.mod file. We should add a similar check that we have in other repos to detect if there are diffs when running "go mod tidy" over the changes.

func Invoke() in invoke.go assumes opts is non-nil

Invoke() in invoke.go iterates opts... but does not check if it is nil. This means the user must supply a CallOption, which doesn't make sense since naturally if you don't want any options you pass nil. This is what I do in my code since I could not find any default CallOption in the documentation.

func Invoke(ctx context.Context, call APICall, opts ...CallOption) error {
	var settings CallSettings
	for _, opt := range opts {
-->        opt.Resolve(&settings)
	}
	return invoke(ctx, call, settings, Sleep)
}

My code that eventually hits Invoke() method in googleapis/gax-go/v2/invoke.go

   operation, err := client.AsyncBatchAnnotateFiles(ctx, request)
   pollResp, err := operation.Poll(ctx, nil) // passing nil causes nil ptr dereference, panic :( 

Above code gives the following error:

net/http.(*conn).serve.func1(0xc00031c6e0)
	/usr/local/go/src/net/http/server.go:1801 +0x147
panic(0x16e36e0, 0x1c9bbf0)
	/usr/local/go/src/runtime/panic.go:975 +0x47a
github.com/googleapis/gax-go/v2.Invoke(0x18a4be0, 0xc000612600, 0xc000449588, 0xc0000d8700, 0x2, 0x2, 0x1, 0x2)
	/Users/ridhwaan/go/pkg/mod/github.com/googleapis/gax-go/[email protected]/invoke.go:46 +0x76
cloud.google.com/go/longrunning/autogen.(*OperationsClient).GetOperation(0xc0007860c0, 0x18a4be0, 0xc000612600, 0xc0003124c0, 0xc00051edc0, 0x1, 0x2, 0x0, 0x0, 0x0)
	/Users/ridhwaan/go/pkg/mod/cloud.google.com/[email protected]/longrunning/autogen/operations_client.go:259 +0x37a
cloud.google.com/go/longrunning.(*Operation).Poll(0xc0000d86c0, 0x18a4b60, 0xc00003a060, 0x18a2ee0, 0xc000312480, 0xc00051edc0, 0x1, 0x1, 0x16a7fc0, 0x1ce7a01)
	/Users/ridhwaan/go/pkg/mod/cloud.google.com/[email protected]/longrunning/longrunning.go:96 +0x2df

Provide examples for gax.Invoke

The examples in the current docs for this package all involve making a Retryer and then building a retry loop manually without using Invoke. Presumably we want to encourage users to use Invoke instead of doing this.

If so we should provide examples-- including for a non-gRPC transport client.

Refactor path template to match other languages

Particularly, use go's yacc to generate the parser.

Warning: a recent release failed

The following release PRs may have failed:

• #194

Warning: a recent release failed

The following release PRs may have failed:

• #241 - The release job is 'autorelease: tagged', but expected 'autorelease: published'.

Dependency Dashboard

This issue provides visibility into Renovate updates and their statuses. Learn more

Awaiting Schedule

These updates are awaiting their schedule. Click on a checkbox to get an update now.

• chore(all): update golang.org/x/exp commit hash to a062eea
• chore(all): update google.golang.org/genproto commit hash to fe13028
• chore(all): update module github.com/googleapis/gax-go/v2 to v2.1.0
• chore(all): update module google.golang.org/api to v0.56.0
• chore(all): update module google.golang.org/grpc to v1.40.0
• chore(all): update module honnef.co/go/tools to v0.2.1

Ignored or Blocked

These are blocked by an existing closed PR and will not be recreated unless you click a checkbox below.

• chore(all): update golang.org/x/lint commit hash to 6edffad

---

• Check this box to trigger a request for Renovate to run again on this repository

v2: [build failed] failed

This test failed!

To configure my behavior, see the Build Cop Bot documentation.

If I'm commenting on this issue too often, add the buildcop: quiet label and
I will stop commenting.

---

commit: 5a082ae
buildURL: Build Status, Sponge
status: failed

internal/kokoro: enable the Build Cop Bot

Instructions here: https://github.com/googleapis/repo-automation-bots/tree/master/packages/buildcop

While we're at it, we're currently dropping test logs from v1 with tee:

gax-go/internal/kokoro/test.sh

Lines 28 to 34 in b443e5a

	try3 go mod download
	./internal/kokoro/vet.sh
	go test -race -v . 2>&1 | tee $KOKORO_ARTIFACTS_DIR/$KOKORO_GERRIT_CHANGE_NUMBER.txt
	cd v2
	try3 go mod download
	go test -race -v . 2>&1 | tee $KOKORO_ARTIFACTS_DIR/$KOKORO_GERRIT_CHANGE_NUMBER.txt

update README with v2 install instructions

Using the install instructions in the README today results in the wrong version being pulled when using go.mod per #89.

all: do not hide original error code

Is your feature request related to a problem? Please describe.
this code hides original error message into context deadline exceeded
cloud.google.com/go/bigquery/storage/apiv1/big_query_write_client.go:396

func (c *bigQueryWriteGRPCClient) GetWriteStream(ctx context.Context, req *storagepb.GetWriteStreamRequest, opts ...gax.CallOption) (*storagepb.WriteStream, error) {
	if _, ok := ctx.Deadline(); !ok && !c.disableDeadlines {
		cctx, cancel := context.WithTimeout(ctx, 600000*time.Millisecond)
		defer cancel()
		ctx = cctx
	}
	md := metadata.Pairs("x-goog-request-params", fmt.Sprintf("%s=%v", "name", url.QueryEscape(req.GetName())))

	ctx = insertMetadata(ctx, c.xGoogMetadata, md)
	opts = append((*c.CallOptions).GetWriteStream[0:len((*c.CallOptions).GetWriteStream):len((*c.CallOptions).GetWriteStream)], opts...)
	var resp *storagepb.WriteStream
	err := gax.Invoke(ctx, func(ctx context.Context, settings gax.CallSettings) error {
		var err error
		resp, err = c.bigQueryWriteClient.GetWriteStream(ctx, req, settings.GRPC...)
		return err // ----> this one can be anything, eg. API limit or something
	}, opts...)
	if err != nil {
		return nil, err // ----> this one context deadline exceeded
	}
	return resp, nil
}

Describe the solution you'd like
it's better to return last error instead so we can know what's the cause of the context deadline exceeded

Describe alternatives you've considered
i don't see any other alternative

Additional context
annoying to trace to know what's the cause of context deadline exceded when the original error hidden.

Add call options for page size and page token

Standard Google API List methods usually have page size and page token fields in the request proto to support pagination. If the generated list method (or a hand-wrapped version of it) does not take the actual request proto, the easiest way to specify page size and page token is with options. Since methods that call RPCs already take call options, we should add options for page size and token.

SetPageSize should take an int

The signature of an iterator's SetPageSize method should have an int arg, not int32. When setting the int32 page_size field of a request, the arg is silently truncated:

req.PageSize = int32(pageSize)

call options changes: from map to struct; from option to method

Instead of using a map that requires the string names of methods, generate a CallOptions struct for each API that has the method names as fields.

Instead of having this as a client option, add these methods to the client:

• SetCallOptions(*CallOptions)
• CallOptions() *CallOptions

These changes are in preparation for moving to a new options package. See #29.

Warning: a recent release failed

The following release PRs may have failed:

• #194

[Policy Bot] found one or more issues with this repository.

Policy Bot found one or more issues with this repository.

• Default branch is 'main'
• Branch protection is enabled
• Renovate bot is enabled
• Merge commits disabled
• There is a CODEOWNERS file
• There is a valid LICENSE.md
• There is a CODE_OF_CONDUCT.md
• There is a CONTRIBUTING.md
• There is a SECURITY.md

Warning: a recent release failed

The following release PRs may have failed:

• #194

Configuration error for release-please

base (googleapis/gax-go): Missing required manifest config: release-please-config.json

Rename `PathTemplate.Instantiate` to `PathTemplate.Render`

go mod picking up wrong 2.0 version

When I run go get github.com/googleapis/gax-go using modules, the version I get is from the tag v2.0.2+incompatible (with sum h1:silFMLAnr330+NRuag/VjIGF7TLp/LBrV2CJKFLWEww=), when it seems that v2.0.5 is the latest.

Running a plain go get -u gets me v1.0.3.

What's happening? It feels like there's something that needs to happen from gax-go's end?

Getting go modules error when trying to use an older version

When running a go get with Go modules for some projects that use v2.0.0 it fails:

GO111MODULES=on go get google.golang.org/[email protected]
go: finding github.com/googleapis/gax-go/v2 v2.0.2
go: finding github.com/googleapis/gax-go/v2 v2.0.1
go: finding github.com/googleapis/gax-go/v2 v2.0.0
go: github.com/googleapis/gax-go/[email protected]: missing github.com/googleapis/gax-go/go.mod and .../v2/go.mod at revision v2.0.0
panic: error loading module requirements

goroutine 1 [running]:
cmd/go/internal/mvs.Downgrade.func2(0xc00023dac0, 0x1f, 0xc000232469, 0x6)
	/usr/local/go/src/cmd/go/internal/mvs/mvs.go:317 +0x659
cmd/go/internal/mvs.Downgrade(0xc00023cc80, 0x20, 0x0, 0x0, 0x16ad6e0, 0xc0001be230, 0xc0003676c0, 0x1, 0x1, 0x400, ...)
	/usr/local/go/src/cmd/go/internal/mvs/mvs.go:349 +0x617
cmd/go/internal/modget.runGet(0x1a1ac60, 0xc0000b4020, 0x1, 0x1)
	/usr/local/go/src/cmd/go/internal/modget/get.go:445 +0x1ec1
main.main()
	/usr/local/go/src/cmd/go/main.go:219 +0x82d

grpc.Code is deprecated

status.FromError and Code method instead

use new option and transport packages

There are new packages at google.golang.org/api/option and google.golang.org/transport that are meant to be shared between gax and the handwritten clients. Gax should use these packages.

tag a new version

v2.0.3 depends on [email protected], which has the bad lint dependency.

Add example docs to APIError package

There are good examples for how to unpack and use APIError in the google-cloud-go docs. However, if a user looks up the APIError type directly, there is not any clarity on how to use it in the docs for the package, which might be confusing. We should add similar docs there, or maybe link to the google-cloud-go docs in question.

Alternatively (or additionally), it also might be worth added a section on error handling using APIError to the auto-generated doc.go files for GAPIC clients for easier discoverability.

apierror: simplify extraction of custom errors

Within apierror, the ErrDetails type holds decoded values from an rpc status:
https://pkg.go.dev/github.com/googleapis/gax-go/v2/apierror#ErrDetails

Custom messages such as service-specific custom error types end up in the Unknown slice of ErrDetails, which is a slice of empty interface types. It would be beneficial to provide a utility mechanism for extracting a message of a given type from the ErrDetails.

By way of example, the bigquerystorage service has a StorageError type:
https://pkg.go.dev/google.golang.org/genproto/googleapis/cloud/bigquery/storage/v1#StorageError

Being able to more easily extract this from APIError would be lovely. Perhaps something like this might make sense?

if msg := apiErr.Details().ExtractDetail(&storagepb.StorageError{}); msg != nil {
  if se, ok := msg.(*storagepb.StorageError); ok {
    // do something with the storage error
  }
}

Also happy to make the change if this seems reasonable.

cc @noahdietz @quartzmo

Verify path template implemenation

Jonathan (kindly) provided the path template parser for Go. For all of our use cases, I believe it works and is sufficient for the far future.

I never got around to making sure it had 1:1 parity with the Python version (test cases, limitations), so it's worth checking as a very low priority item (I don't, for example, think it limits the valid characters in a literal).

Tag a new release sometime soon?

Summary

In the most recent release, the v2 module still requires go 1.12, though this has since been removed by @jadekler in b2b1826#diff-dc2e1be9e9810655cf485878bd63cde6

Go mod defaults to the latest release tag on the initial dependency pull, so anyone who is not on 1.12 or later will have to come here and debug a bit themselves

Warning: a recent release failed

The following release PRs may have failed:

• #194

Add HTTPCode to APIError

APIError was intended to be transport-agnostic, so it currently doesn't have any fields for HTTP or gRPC response codes. However, will lead to some clunky error handling code for users once we use APIError on apiary library calls. To extract the most important fields, users will have to call errors.As twice with two separate error types:

if err := bucket.Create(ctx, "bad-project-id", nil); err != nil {
        var aErr *googleapi.Error
        if errors.As(err, &aErr) {
                fmt.Printf("Status: %d\n", aErr.Code)  // pulls HTTP response code
        }
        var apiErr *apierror.APIError
        if errors.As(err, &apiErr) {
                fmt.Printf("Reason: %s\n", apiErr.Reason())  // pulls error reason
                fmt.Printf("Details: %sn", apiErr.Details())
        }
        // Handle error
}

Code is in fact the only field from googleapi.Error that we are currently missing out on in APIError. For manual clients such as storage where most users will be HTTP-only, it would be cleaner and easier to document if we could add a method to retrieve this into APIError as well.

It may also be nice to pull the gRPC status into APIError as well; however, I think it's preferable to keep these as separate methods (HTTPCode and GRPCCode say) in order to avoid having to map between HTTP and gRPC statuses.

Add something to the README describing this project's status

GA, experimental, support plan, etc?

Suggest for now that we say it's experimental and there is no support.

v2.1.0 is released but Version is still "2.0.5" ?

https://pkg.go.dev/github.com/googleapis/gax-go/v2
shows Version: v2.1.0 Latest

but

const Version = "2.0.5"

tag 2.1.0 is unexpected one?

Package should be in a subdirectory

The package name should be the same as the final part of the path.

I suggest github.com/googleapis/gax-go/gax

add support for release-please

• use release-please
• have changes file

Warning: a recent release failed

The following release PRs may have failed:

• #219 - The release job is 'autorelease: tagged', but expected 'autorelease: published'.

Match python usage of total retry and max rpc timeout

Panic in invoke.go

Go 1.7

My server just died with this error. Thought I would let you know...

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x7507d2]

goroutine 584 [running]:
panic(0x939380, 0xc4200100e0)
/usr/local/go/src/runtime/panic.go:500 +0x1a1
github.com/googleapis/gax-go.Sleep(0x0, 0x0, 0x18071e29, 0xc6d6c902c4, 0xc6d6c902c0)
/home/alex/go/src/github.com/googleapis/gax-go/invoke.go:56 +0x52
cloud.google.com/go/storage.runWithRetry(0x0, 0x0, 0xde8886fb08, 0xde8886faa8, 0xc5ea2f6de0)
/home/alex/go/src/cloud.google.com/go/storage/invoke.go:39 +0xc2
cloud.google.com/go/storage.(*ObjectIterator).fetch(0xc602fc4150, 0x0, 0xd2e6d72dc0, 0xb0, 0x454799, 0xc44fe3c118, 0x7f0700000000, 0$
/home/alex/go/src/cloud.google.com/go/storage/bucket.go:247 +0x543
cloud.google.com/go/storage.(*ObjectIterator).(cloud.google.com/go/storage.fetch)-fm(0x0, 0xd2e6d72dc0, 0xb0, 0x20, 0x974600, 0x1, 0$
/home/alex/go/src/cloud.google.com/go/storage/bucket.go:194 +0x48
google.golang.org/api/iterator.(*PageInfo).fill(0xc7633c01e0, 0x0, 0xc44fe25c00, 0x4954d0)
/home/alex/go/src/google.golang.org/api/iterator/iterator.go:143 +0x49
google.golang.org/api/iterator.(*PageInfo).next(0xc7633c01e0, 0x44ad5d, 0x8fa8c0)
/home/alex/go/src/google.golang.org/api/iterator/iterator.go:122 +0x80
google.golang.org/api/iterator.(*PageInfo).(google.golang.org/api/iterator.next)-fm(0x974601, 0x5381a9)
/home/alex/go/src/google.golang.org/api/iterator/iterator.go:100 +0x2a
cloud.google.com/go/storage.(*ObjectIterator).Next(0xc602fc4150, 0xc44fe3c0c0, 0xde8886fe68, 0xde8886fe88)
/home/alex/go/src/cloud.google.com/go/storage/bucket.go:224 +0x2f
_/home/alex/_code/graphserver/poly.(*PolyGraph).Worker(0xc4200849c0)
/home/alex/_code/graphserver/poly/workers.go:287 +0xb63
created by _/home/alex/_code/graphserver/poly.(*PolyGraph).StartWorkers
/home/alex/_code/graphserver/poly/workers.go:128 +0xc98

APIError.GRPCStatus returns "OK" for all http errors.

When using apierror.FromError with an http error, the status field does not get populated, so the GRPCStatus() method returns nil. It seems like this is intended to mean "no status is available". However, the status package treats nil as "OK":
https://github.com/grpc/grpc-go/blob/dba26e15a07f43875ccf806a2dd6cbcbc1c12eab/internal/status/status.go#L72

The status package also says that no non-nil error should have status of "OK".
To properly fulfil this contract, it seems that the APIError.status should never be nil. I would expect it to at least return UNKNOWN, but it looks like the json error may also include a status field with a grpc status code, and some of the http status code seem to map directly to grpc codes (e.g. 404, 429, 503):
https://cloud.google.com/apis/design/errors#handling_errors

no

gax-go: package needs package-level documentation

I stumbled upon the godoc for this package via a link from golang/oauth2#203, and immediately wondered "what the heck is gax"? There was no package-level documentation to answer this. Especially since it has such an odd name, it seems like this package needs some context.

options should be a separate package

For the handwritten clients, gax is an implementation detail, but it leaks through because it is the package that has the options.

Can we move options to their own package where they can be used by both handwritten and generated clients?

Suggested package names:

• options
• opts -- better, since it will be typed repeatedly: opts.WithThis(), opts.WithThat(), ...

Suggested import paths:

• github.com/googleapis/go/opts
• google.golang.org/api/opts

I'd be okay with pretty much anything that mentions APIs (not cloud, because they aren't cloud-specific).

@saicheems @mcgreevy @okdave @broady