ggicci / httpin

🍡 HTTP Input for Go - HTTP Request from/to Go Struct (Bi-directional Data Binding between Go Struct and http.Request)

AD

🔥 We're hiring! Message me for a referral :)

• Senior Software Development Engineer, AWS IAM

PROJECTS

• ⭐ httpin: Go - HTTP Request from/to Struct, i.e. Bi-directional Data Binding between Go Struct and http.Request (documentation)
• 🦉 owl: Go - a Go Struct Tag Framework and Algorithm Driver
• caddy-jwt: Go -Caddy Module for JWT Authentication (documentation)
• distlock: Go - Simple distributed locks using Redis, MySQL, PostgreSQL, etc.
• password: Go - Password Hashing & Verification
• goenv: Go - Populate go struct instance with OS environment variables.
• goplay: TypeScript - Embed Go Playground on your Website (demo)
• v2raym: Bash - Dead simple v2ray client for macOS (in terminal)
• droplet.sh: Bash - Import shell scripts like Go
• python-fossil-delta: Python wrapper of the C implementation of Fossil Delta Encoding Algorithm
• jsonla: C++ - Parse and manipulate JSON data structure

httpin's Issues

the error occurred when parsing form data of the http request is not explicit to clients

Errors occurred here can be posted to clients here. Which is ambiguous.

Can I use pointer types as query params?

For example something like

type SearchCollectionParams struct {
	CacheTtl                *int    `in:"query=cacheTtl"`
	DropTokensThreshold     *int    `in:"query=dropTokensThreshold"`
}

Currently I get the error unsupported type *int. The idea is for them to become nil when not provided.

Any help is appreciated, thanks!

Feature Request: ignore query parameters during encoding

I'd like to have the option to ignore a query parameter during encoding if the parameter is empty. For simplicity, empty could be considered the zero value of the type.

If more complex support is desired, this could work with both pointers and values. If it is a pointer-type and nil, it can be considered empty and ignored (if the optional struct-tag option is given). If it is a value and is set to the type's zero-value, it can be considered empty and ignored (if the optional struct-tag option is given). WDYT?

Support `multipart/form-data` for file uploading requests

Gain the ability to process multipart/form-data data.

e.g.

type UpdateProfileInput struct {
	Email  string      `in:"form=email"`
	Gender string      `in:"form=gender"`
	Avatar httpin.File `in:"form=avatar"`
}

The httpin.File struct might be:

type File struct {
	multipart.File
	Header *multipart.FileHeader
}

implement a "default" directive

Refer to gorilla/schema#182 gorilla/schema#153

type Pagination struct {
	Page     int64 `in:"query=page;default=1"`
	PageSize int64 `in:"query=page_size;default=20"`
}

replace default decoder for JSON

i've been using json-iterator/go for a long time, having ability to replace default decoder for json would be great.
or better, being able to add custom request body format like bson,msgpack,cbor etc.

Feature Request: support encoding

This package is awesome for a service receiving requests and unmarshalling them into Go structs. But it would be even better to be able to have a single struct for clients and services to use to marshal a request into an http.Request struct, and unmarshal a http.Request struct into a Go struct.

I played around with this idea myself and I'm open to contributing if it is approved as an idea.

Is it possible to use message pack instead of json/xml for request bodies?

Hi, thanks for the great library!

Right now, the library supports json, xml for request bodies. Is it possible to support for msgpack as well? Can I write a custom directive executor for this new format?

Should I be using RegisterBodyFormat, RegisterBodyDecoder or something else?

Any guidance is appreicated. TIA!

Parse and validate custom time format

Hi, is there a way to parse and validate custom time format? For example, I want a query parameter which doesn't have to have a timezone, like: 2022-05-07T12:45:13

Thanks in advance!

Resolve failed on unexported fields

Error Message

2021/08/14 08:59:33 http: panic serving 172.23.0.3:51146: reflect: reflect.Value.Set using value obtained using unexported field
goroutine 38 [running]:
net/http.(*conn).serve.func1(0xc00031c460)
        /go/src/net/http/server.go:1824 +0x153
panic(0xdab0a0, 0xc0004011b0)
        /go/src/runtime/panic.go:971 +0x499
reflect.flag.mustBeAssignableSlow(0x1b5)
        /go/src/reflect/value.go:257 +0x1b9
reflect.flag.mustBeAssignable(...)
        /go/src/reflect/value.go:247
reflect.Value.Set(0xdf24a0, 0xc0003f43b8, 0x1b5, 0xdf24a0, 0xc000010e30, 0x195)
        /go/src/reflect/value.go:1558 +0x3b
github.com/ggicci/httpin.(*fieldResolver).resolve(0xc000190480, 0xc0000b4b00, 0x0, 0x8, 0xc0003f8cf0, 0xd73140, 0x203000)
        /home/coder/go/pkg/mod/github.com/ggicci/[email protected]/resolver.go:72 +0x68d
github.com/ggicci/httpin.(*Engine).Decode(0xc000033600, 0xc0000b4b00, 0x7f49b12b4108, 0x7f4988643720, 0x203000, 0x203000)
        /home/coder/go/pkg/mod/github.com/ggicci/[email protected]/httpin.go:88 +0x65
github.com/ggicci/httpin.NewInput.func1.1(0x1042840, 0xc0003242a0, 0xc0000b4b00)
        /home/coder/go/pkg/mod/github.com/ggicci/[email protected]/chain.go:24 +0x65
net/http.HandlerFunc.ServeHTTP(0xc000033640, 0x1042840, 0xc0003242a0, 0xc0000b4b00)
        /go/src/net/http/server.go:2069 +0x44
github.com/ggicci/pebble/api.LoadCollectionOfUser.func1(0x1042840, 0xc0003242a0, 0xc0000b4800)
        /home/coder/pebble/api/mwcollections.go:56 +0x9b7
net/http.HandlerFunc.ServeHTTP(0xc00000f740, 0x1042840, 0xc0003242a0, 0xc0000b4800)
        /go/src/net/http/server.go:2069 +0x44
github.com/ggicci/pebble/api.LoginRequired.func1(0x1042840, 0xc0003242a0, 0xc000332900)
        /home/coder/pebble/api/mwaccess.go:51 +0x569
net/http.HandlerFunc.ServeHTTP(0xc00000f758, 0x1042840, 0xc0003242a0, 0xc000332900)
        /go/src/net/http/server.go:2069 +0x44
github.com/go-chi/chi.(*ChainHandler).ServeHTTP(0xc0003f2800, 0x1042840, 0xc0003242a0, 0xc000332900)
        /home/coder/go/pkg/mod/github.com/go-chi/[email protected]/chain.go:31 +0x52
github.com/go-chi/chi.(*Mux).routeHTTP(0xc00007dd40, 0x1042840, 0xc0003242a0, 0xc000332900)
        /home/coder/go/pkg/mod/github.com/go-chi/[email protected]/mux.go:436 +0x28b
net/http.HandlerFunc.ServeHTTP(0xc0002fd860, 0x1042840, 0xc0003242a0, 0xc000332900)
        /go/src/net/http/server.go:2069 +0x44
github.com/go-chi/chi.(*Mux).ServeHTTP(0xc00007dd40, 0x1042840, 0xc0003242a0, 0xc000332900)
        /home/coder/go/pkg/mod/github.com/go-chi/[email protected]/mux.go:70 +0x5ab
github.com/go-chi/chi.(*Mux).Mount.func1(0x1042840, 0xc0003242a0, 0xc000332900)
        /home/coder/go/pkg/mod/github.com/go-chi/[email protected]/mux.go:311 +0x17c
net/http.HandlerFunc.ServeHTTP(0xc0000334c0, 0x1042840, 0xc0003242a0, 0xc000332900)
        /go/src/net/http/server.go:2069 +0x44
github.com/go-chi/chi.(*Mux).routeHTTP(0xc00007dce0, 0x1042840, 0xc0003242a0, 0xc000332900)
        /home/coder/go/pkg/mod/github.com/go-chi/[email protected]/mux.go:436 +0x28b
net/http.HandlerFunc.ServeHTTP(0xc0002fd830, 0x1042840, 0xc0003242a0, 0xc000332900)
        /go/src/net/http/server.go:2069 +0x44
github.com/go-chi/chi.(*Mux).ServeHTTP(0xc00007dce0, 0x1042840, 0xc0003242a0, 0xc000332700)
        /home/coder/go/pkg/mod/github.com/go-chi/[email protected]/mux.go:87 +0x331
github.com/ggicci/pebble/api.Auth.func1(0x1042840, 0xc0003242a0, 0xc000332500)
        /home/coder/pebble/api/mwauth.go:86 +0x66a
net/http.HandlerFunc.ServeHTTP(0xc00040e588, 0x1042840, 0xc0003242a0, 0xc000332500)
        /go/src/net/http/server.go:2069 +0x44
net/http.serverHandler.ServeHTTP(0xc0004080e0, 0x1042840, 0xc0003242a0, 0xc000332500)
        /go/src/net/http/server.go:2887 +0xa3
net/http.(*conn).serve(0xc00031c460, 0x1045358, 0xc000302440)
        /go/src/net/http/server.go:1952 +0x8cd
created by net/http.(*Server).Serve
        /go/src/net/http/server.go:3013 +0x39b

Reproduce

type State struct {
	Name           string
	Upstream       []string
	upstreamLookup []string
}

Use State as the input structure to reproduce.

How do you parse form data into nested struct?

I have form like this:

curl --location 'localhost/upload' \
--form 'body=@"/Users/Phantom/Documents/pdf/sample/sample.pdf"' \
--form 'title="document pdf new upload"' \
--form 'author.name="someone"' \
--form 'author.address="github"' \

and my struct is like this:

type Upload struct {
	Body   httpin.File `in:"form=body"`
	Title  string      `in:"form=title"`
	Author author      `in:"form=author"`
}

type author struct {
	Name    string `in:"form=name"`
	Address string `in:"form=address"`
}

I am unable to get value from author. How can I get author from form-data using nested struct?

Determining type of error - i.e. client or implementation.

There are different types of error that can occur when decoding a request.

1. Client errors where invalid data was passed, i.e. a missing required field
2. Implementation errors - i.e. where a decoder is not registered.

httpin used to have declared errors, this meant that we could switch on the type of error.

So for "internal/implementation errors" we could then handle these and return a 500 and alert engineers so we could implement a fix.

// internalErrors are system level things that the client can not fix, we return internal
// errors for these and do not surface the route cause.
// httpin has a number of errors, some of these are panics that are caused when registering types.
// The following errors are ones that are evaluated at runtime when the handler structs have been misconfigured.
var internalErrors = []error{
	httpin.ErrDecoderNotFound,
	httpin.ErrMissingDecoderName,
	httpin.ErrUnregisteredExecutor,
}

And the remaining errors were then assumed to be client errors, so we would then return an appropriate 400 error code allowing the client to understand their error.

With the latest versions of httpin then it is much harder to determine the cause of these errors - meaning it's hard to determine whether to return a 500 or a 400 to the client - and what we should log.

How would you feel about

1. Having a couple of base errors - i.e. Client Error and Internal Error, that are wrapped. This would allow us to do a errors.Is and determine the correct action to take
2. Using declared Errors again so that custom error handling can be specific with how it manages different types of errors.

Happy to try and put together some PRs if you think this would be ok - or maybe you have some other idea about how this could be achieved.

r.Context().Value(httpin.Input) always nil

Hey, I think Im using httpin right but r.Context().Value(httpin.Input) is always nil.

This is my setup:

Query: /api/cases?is_member=true&sort_by[]=age&sort_desc[]=false
Router: rApi.HandleFunc("/cases", handlers.PostCases).Methods("GET")
Handler: fmt.Printf("form: %#v\n",r.Context().Value(httpin.Input))

Im using mux. Is there something I have missed to configure? Do I have to use httpin.UseGorillaMux("path", mux.Vars)?

Any way to pre-initialize the struct?

Hi,

I would like to be able to create and initialize an empty struct for each call to Decode instead of httpin creating a new instance using

rootValue := reflect.New(r.Type)

Any chance there is an option to do that?

integrate with more web frameworks or packages

• net/http
• go-chi/chi
• gorilla/mux
• gin-gonic/gin
• go-restful, by #23
• add more here...

Dealing with nested structs (again) and OpenAPI v3 - deepObject

Hello!

I am familiar with #59 and #61 and your decision to reject "nested structs", but at the same time I want to drag your attention to OpenAPI v3.x

There is the link to document about query parameters serialization

Please, pay attention to part about "deepObject".
To make possible to use your library with OpenAPI based server/client, you need to support one-level of nested object for query and form parameters

nickname[equal]="foo"&age[in]=1&age[in]=2&...

Also please pay attention there: https://github.com/go-playground/form

I totally understand your decision to reject "complex query parameters", but at the same time - how to deal with these kind of servers and these kind of query parameters?

You can like or dislike them, but in the real world you have the cases, where you need to deal with them.
Without some solution, I could not use your library :( I like it a lot, but at the same time I must to find some solution how to deal with nested objects in query parameters (show-stopper to me)

Thank you!

feature request - add commonly used structures, e.g. pagination?

1. Offset/Limit pagination --> OffsetPagination
2. Seek/Keyset/Cursor pagination --> SeekPagination

Optional body for POST requests

Hi! I have been using httpin for a while now and it has been an excellent library. However, I recently noticed that one of my post endpoints didn't have any payload defined, and now I need to add an optional json payload to it (optional because it is imperative I don't break backwards compatibility). This is the sample input type I have:

type RegisterAttendance struct {
	Slug string                   `in:"path=slug" validate:"required"`
}

I need to change it to

type RegisterPayload struct{
	VerifiedBy *string                   `json:"verifiedBy"`
}
type RegisterAttendance struct {
	Slug string                   `in:"path=slug" validate:"required"`
	Payload RegisterPayload `in:"body=json"`
}

However doing this means I can no longer make a post call with empty body. I mandatorily have to send {} as the payload. If I don't send any request body, I get the following error:

 failed: execute directive \"body\" with args [json] failed: EOF

We have clients that have been making calls without sending any request body to this endpoint, and I can't break those calls. If I was doing this without HTTPIN, I'd check of EOF while decoding JSON explicitly. Is there a way I can get this working with HTTPIN?

Thanks for the help!

querystring directive

would a querystring directive which extracts values from the request.URL.Query() values be of use?

I would be happy to draft a pull request...

something like:

type Pagination struct {
	Page int `in:"form=page;query=page"`

	// Decode from multiple keys in the same source, the former with higher priority
	PerPage int `in:"form=per_page,page_size;query=page_size"`
}

I took a look at implementing one as a custom directive in my own project but it seems like it would share so much logic with the form directive extractor that I thought it may be simpler to offer it in the box alongside the form extractor.

A common use case might be an API which allows a short query value passed in a URL param in a GET request but a longer or more complex query value to be passed in the form body of a POST

[Question] Custom slice/array of integers decoder

Hey @ggicci,

First of all amazing package, really useful thing 🙌

I have a question related to how can I decode a query array of integers. I want something like this: http://url.com?id=1,2,3 and I want this to be parsed as []int property in my struct.

I saw in the docs that there is a possibility with using ?id=1&id=2... but for me it's not really convinient when you can just set them all with comma between them.

I wanted to ask what is should I use to create something like a custom decoder in that case? The decoder/coder or custom directive?

Thanks!

Allow mix of body input and param input

Due to this check and potentially missing implementation, a mix of JSON/XML bodies along with other directives is not possible.

This blocks common REST calls, for example creating/updating entities (POST /entity/:id/PUT /entity/:id), where you wish to decode both the URL param :id and the incoming JSON body into a single struct.

integrate with go-restful

any suggestions?

decoder registered with ReplaceTypeDecoder not invoked

Hi, first of all, httpin is great! Thanks for your hard work!

Im trying to use ReplaceTypeDecoder to override the time.Time decoding like such (v0.10.1):

// register custom decoder for time.Time
func init() {
	httpin.ReplaceTypeDecoder(reflect.TypeOf(time.Now()), httpin.ValueTypeDecoderFunc(decodeDartTimestamp))
}

// custom time.Time decoder
func decodeDartTimestamp(value string) (interface{}, error) {
	return time.Parse(time.RFC3339, value)
}

// example of query input 
type UpcomingReturnsRequest struct {
	After      time.Time `in:"query=after"`
}

// example of body input
type PrioritizedInput struct {
	httpin.JSONBody
	PrioritizedAt    time.Time `json:"prioritized_at"`
}

Unfortunately, my custom decode never gets executed (verified with log as well as breakpoint. What am I doing wrong?

How do I use nested structs from query?

I have a type like this:

type ListPagesInput struct {
	CustomerID       string                  `in:"path=customerID" validate:"required"`
	PaginationParams PaginationParams `in:"query=pagination{}" validate:"required"`
}
type PaginationParams struct {
	Cursor    string `in:"query=cursor"`
	SortOrder string `in:"query=sortOrder"` 
	Limit     int32  `in:"query=limit"`
}

I am unable to figure out how to send it as part of query string. I tried making a request like this: http://localhost:8701/v1/customer/shop/pages?pagination={limit:1}, but the PaginationParams doesn't seem to be getting values from the query string (limit is always zero).
Any help is appreciated, thanks.

allow using json tag value as a fallback

Refer to gorilla/schema#184. We can borrow the idea to httpin as:

adding an Annotation to allow using the value of json tag as an implicit query directive for decoding.

For example, the following struct

type ListUsersInput struct {
	IsMember bool  `in:"query=is_member" json:"is_member"`
	AgeRange []int `in:"query=age_range" json:"age_range"`
}

can be rewritten to

type ListUsersInput struct {
	httpin.UseJSONTagAsQueryArgs // annotation: auto-inject struct tags `in:"query={value of json tag}"`
	IsMember bool  `json:"is_member"`
	AgeRange []int `json:"age_range"`
}

embed go playground into httpin docs

Since I've found a way to embed go playground on static websites. One enhancement can be done to the httpin docs is to add sample code with go playground to improve demonstrating how to use httpin package.

Invent a "path" directive to extract values from go-chi/chi's URLParam

type ListArticlesOfUserInput struct {
	Author string `in:"path=name"`  // should get value from chi.URLParam
	Page   int    `in:"form=page"`
}

Refer to https://github.com/ggicci/httpin/blob/main/gorilla.go

implement some frequently used directives as builtins

Though httpin had provided an easy way to add your custom directive. If httpin can provide some common used directives, it would be helpful.

e.g.

• to_lower
• to_upper
• default, by #26
• add more here...

Impossible to make streaming

Imagine the case, when I would like to upload multiple large files.
For instance, it could AWS S3 objects, or some large files from disk, or some other large content.

The following code - https://github.com/ggicci/httpin/blob/v0.15.2/core/requestbuilder.go#L151 - populate whole content inside bytes.Buffer.

Now imagine I am uploading the several GB of data...

Preferable way: use streaming. You could use io.Pipe (to divide source of data and destination) + + go-routine (to launch copy process under the hood + io.golang context (to cancel copy operation) + io.MultiReader (to combine multiple parts together)

Feature Request: non-global directives

Summary

I'd like to be able to register directives against an instance and then perform encoding and decoding against that instance.

Reason

I have an API (i.e. "API A") that also calls another API ("API B"). API B provides structs with httpin struct tags via an importable package. API A also provides structs with httpin tags. The structs in API A and API B have different needs and therefore different directives that need to be registered. However, with the current design, all directives are registered against the global namespace.

Proposed Solution

The Core struct seems perfectly set up to do this. Each instance could register a new namespace, or have a way of localizing the registrations inside its own scope.

Decode parameter struct with default values only works the first time

The first time the resolver is stored in a cache. The second time the resolver is loaded from the cache and seems to be modified, it cannot apply the default value anymore.

Second time there is an error message:

invalid field "Page": resolve field "Page (int)" failed: execute directive "default" with args [default] failed: strconv.ParseInt: parsing "default": invalid syntax

Version 014.1
The resolver attempts to parse the value "default" in stead of the value from the struct tag.

Test to reproduce the error:

type ThingWithDefaultValues struct {
	Id      uint `in:"query=id;required"`
	Page    int  `in:"query=page;default=1"`
	PerPage int  `in:"query=page_size;default=127"`
}

func TestDirectiveDefault2(t *testing.T) {

	r, _ := http.NewRequest("GET", "/?id=123", nil)

	expected := &ThingWithDefaultValues{
		Id:      123,
		Page:    1,
		PerPage: 127,
	}

        // First decode works as expected
	xxx := ThingWithDefaultValues{}
	err := httpin.Decode(r, &xxx)
	assert.NoError(t, err)
	assert.Equal(t, expected, &xxx)

        // Second decode generates eror
	err = httpin.Decode(r, &xxx)
	assert.NoError(t, err)
	assert.Equal(t, expected, &xxx)
}

implementation regexp for tag

this package is absolutely awesome, i have used in my project.

i have some suggestion. static tag is awesome give consistency from developer.
but some cases, i need to create custom decoder to add values with regex key my structs.

this my personal opinion, using input with tag regex pattern key thats make more flexibility

// Example Struct
type SomeStruct struct {
 Search []string `in:re_query=^search_by_(\w+)_op_(\w+)$`
 Sort []string `in:re_query=^sort_by_(\w+)$`
}

thanks

Go 1.22+ net/http new features support

Since go version 1.22+, the native net/http has brought some new features, including:

• Path parameters, e.g. /products/{id}
• Method based routing, e.g. "GET /products/{id}"
• Host based routing, e.g. "example.com/products/{id}"
• ???

Require doesn't work as expected when using mixed body input

Following this example, the required tag doesn't work as expected.

type UpdateUserInput struct {
	Payload struct {
		Display string `json:"display" in:"required"`
	} `in:"body=json"`
}

If body is not passed in request, no error is thrown (while it should)

bug: required errors use golang name instead of field name

Given a type with required fields:

type ProductQuery struct {
	CreatedAt time.Time `in:"form=created_at;required"`
	Color     string    `in:"form=colour,color"`
	IsSoldout bool      `in:"form=is_soldout"`
	SortBy    []string  `in:"form=sort_by"`
	SortDesc  []bool    `in:"form=sort_desc"`
	Pagination
	Authorization
}

When I make a request to an endpoint parsing this type with httpin
Then I expect the error to be

invalid field "created_at": missing required field

Currently the error is

invalid field "CreatedAt": missing required field

which can be misleading to clients, since submitting a request with a CreatedAt property is not guaranteed to bind.

bump version to 1.0.0

I'm planing to publish the first stable release of this program. If any of you were using this library in your production environment, please feel free to leave a comment below. A breif description of your application scenarios will bring great helps ❤️

Invoke decode manually

Thanks for this useful library

It's just a personal opinion but I find using a middleware to parse request parameters and pass the parsed values to handlers through context a bit odd. I prefer calling decode manually in a handler and handle error in the handler itself. To do that, I need to do something like this at the moment:

engine, err := httpin.New(MyParams{})
if err != nil {
	return nil, err
}

input, err := engine.Decode(req)
if err != nil {
	return nil, err
}

return input.(*MyParams), nil

it's much more friendly if it exposes a single function to invoke like err := httpin.Decode(req, &myParams). This requires a default engine that is not dependent on any specific input and might be a big change though

How do I use httpin.RegisterTypeDecoder in the latest version?

Hello everyone, I am currently on httpin 0.11.0 and I use httpin.RegisterTypeDecoder to register types for pointer fields (I do not want to use the new patch fields because of personal preferences). But looks like the method has been removed in 0.12.0 as part of refactor. What is the new way of registering a new type decoder?

Previous code:

func decodeStringPointer(value string) (any, error) {
	return &value, nil
}

func init() {
	strVal := "some string"
	httpin.RegisterTypeDecoder(reflect.TypeOf(&strVal), httpin.ValueTypeDecoderFunc(decodeStringPointer))
}

What's the new code?
Thanks in advance!

Feature: when context(httpin.Input) is nil, then create a new struct

func ListUsers(rw http.ResponseWriter, r *http.Request) {
    input := r.Context().Value(httpin.Input).(*ListUsersInput)
    fmt.Printf("input: %#v\n", input)
}

In this code, when r.Context().Value(httpin.Input) is nil, then create a new ListUsersInput with the refect pkg.

This can be done, but need some code changed. So we can avoid the nil panic error.