bnclabs / gofast Goto Github PK

Brainstorm and validate rolling upgrade scenarios using PeerVersion. Add a recommendation that can be used by gofast developers.

So that applications can learn remote peer's name, buffersize, and tags.

Socket Read() and Write() are blocking, in TCP/IP there can be scenarios of half open connections and other un-detectable network errors on the connections.

On approach is to use net/ package's feature of setting Deadline timeouts for reads and writes, but profiling shows that this can lead to performance issues when we scale connections.

Another approach is to use the heart-beat. That is both peers should set the heartbeat, and expose an API to learn when the last heartbeat was received, set a threshold and close the socket after that.
May be we need a separate monitor routine for this.

Also run travis for 1.7 and 1.8

Gofast is an active library package, a quick page on go-routine block diagram and the data/control flow between them will help understand the code better.

Behind the scenes, support load balancing based on random / moving-average of latency.

When NewTransport() is issued with nil settings, then automatically fallback to DefaultSettings().

TagId and TagData keynames are now encoded as uint64, encode them as tags Major-Type-6.

There won't be any change to the memory foot-print, but this will break nodes compiled with previous version of gofast.

This endpoint should return the list of active transport names in JSON format.
Content-Type: application/json.

Transport structure has got: pData, pStrms, pTxcmd

This can be replaced with lock-less buffer pool.

Http handler for gofast mounted by importing http/ package uses stdlib's encoding/json to create response text. Instead use gson.

This benchmark can be added to perf/ program that already benchmarks POST, REQUEST and STREAM messages.

This can be used by those who are not keen about versioning their application or protocol. I might suggest something like this:

type Version64 uint64

than implements Version{} interface. Subsequently users can simply do:

gofast.NewTransport("myapp", conn, gofast.Version64(1), setts)

This implementation assigns tags 37-39 with different meanings than in the document http://www.iana.org/assignments/cbor-tags/cbor-tags.xhtml#tags . Using these tags should be avoided, as CBOR data that makes use of them with the meanings other than specified in that document would make them less interoperable; instead, tags starting from 266 should be used; they're currently not assigned according to that document.

Hello,

I have recently come across gofast and was wondering how it might compare against QUIC or other similar UDP based protocols that are, in general, much faster than TCP protocols.

Any thoughts on this would be greatly appreciated.
Thanks

Since connections have to scale, we need some performance profiling on select blocks in gofast.

Picture can say thousand words. Create a block diagram of go-routines, how the interact with application logic and underlying socket. Mostly it is important to trace the execution path and its exceptional cases.

Opaque values can be anything >=256. Having said that there are some exceptions:

• tag-0xd9f7 - Self describing CBOR (as per RFC).
• tag-0x1000 to tag-0x100f is reserved by gofast.

While picking a opaque value from user supplied range filter the above values automatically.

Ref: https://github.com/bradleyfalzon/apicompat

Hello,

I wanted to build your example to test gofast but ran into some errors with a repository not being found.

Can you please help?
Thanks

test> go build .\example\
go: cannot find main module, but found .git/config in D:\FarpointFS\test-002\test
        to create a module there, run:
        go mod init
PS D:\FarpointFS\test-002\test> go mod init gofast
go: creating new go.mod: module gofast
go: to add module requirements and sums:
        go mod tidy
PS D:\FarpointFS\test-002\test> go mod tidy
go: finding module for package github.com/bnclabs/gofast/http
go: finding module for package github.com/bnclabs/gosettings
go: finding module for package github.com/bnclabs/golog
go: finding module for package github.com/bnclabs/gson
go: finding module for package github.com/bnclabs/gofast
go: downloading github.com/bnclabs/gofast v0.0.0-20210210111353-5c982d6094f0
go: downloading github.com/bnclabs/golog v0.0.0-20210210111405-2049724535b8
go: downloading github.com/bnclabs/gosettings v0.0.0-20210210111304-0d91c5ae496a
go: downloading github.com/bnclabs/gson v0.0.0-20210409011557-2e73bae93c0b
go: found github.com/bnclabs/golog in github.com/bnclabs/golog v0.0.0-20210210111405-2049724535b8
go: found github.com/bnclabs/gosettings in github.com/bnclabs/gosettings v0.0.0-20210210111304-0d91c5ae496a
go: found github.com/bnclabs/gson in github.com/bnclabs/gson v0.0.0-20210409011557-2e73bae93c0b
go: found github.com/bnclabs/gofast in github.com/bnclabs/gofast v0.0.0-20210210111353-5c982d6094f0
go: found github.com/bnclabs/gofast/http in github.com/bnclabs/gofast v0.0.0-20210210111353-5c982d6094f0
go: finding module for package golang.org/x/text/language
go: finding module for package github.com/prataprc/color
go: finding module for package golang.org/x/text/collate
go: found golang.org/x/text/collate in golang.org/x/text v0.5.0
go: found golang.org/x/text/language in golang.org/x/text v0.5.0
go: finding module for package github.com/prataprc/color
gofast imports
        github.com/bnclabs/golog imports
        github.com/prataprc/color: cannot find module providing package github.com/prataprc/color: module github.com/prataprc/color: git ls-remote -q origin in C:\Users\lonni\go\pkg\mod\cache\vcs\cd6867e44e63b3e2281366409487ea0ca5ceb7587e83ec3a555ba4b2b38e1358: exit status 128:
        remote: Repository not found.
        fatal: repository 'https://github.com/prataprc/color/' not found

Repeat perf/ on Raspberry-pi.

Gofast should support disabling or enabling component level logging. Proposal is to export an API, LogComponents() whose godoc should describe supported components to log.

Subsequently gofast source should use a local variant of infof, errorf, warnf, etc... to enable or disable log.

Calling log.Verbosef() or log.Debugf() escapes values to heap leading to GC overheads.

Repeat perf/ on Raspberry-pi.

TagVersion doesn't make sense. Once gofast protocol is specified and standardised, it can be extended without compromising on backward compatibility. In case we can't provide backward compatibility we should spawn gofast2.