A Nostr relay server framework.
License: The Unlicense
go: go.mod file not found in current directory or any parent directory.
'go get' is no longer supported outside a module.
To build and install a command, use 'go install' with a version,
like 'go install example.com/cmd@latest'
For more information, see https://golang.org/doc/go-get-install-deprecation
or run 'go help get' or 'go help install'.
We no longer neeed to run go get and go install. I'll follow this issue up with a pr
I'm developing nostr-relay using relayer.
Apparently the CPU load has not been stable since the most recent version.
For a while after startup, the program runs stably, but after some time, the CPU load suddenly becomes high.
Do you have any idea what is going on?
Single e tag works as expected
nostcat ws://localhost:9000
["REQ","RAND",{"kinds":[1],"#e":["8b1013c2d137bab5a7026e5adc65dcd641025c1a8ae2b7ecdec848cca44f1297"]}
Two values returns nothing:
nostcat ws://localhost:9000
["REQ","RAND",{"kinds":[1],"#e":["8b1013c2d137bab5a7026e5adc65dcd641025c1a8ae2b7ecdec848cca44f1297","e32f75bfce9513bab7c7e82b9a61fd56f80f68946a9d0a384f7f17989bd91f13"]}]
I suspect this has to do with the LIKE query generation. I will be working on a fix but if you can get it in quicker I'd gladly use that 😅
The new V2 refactor passes the request-scoped context to the storage methods, however since the save operation is handled async in a separate go routine the request-scoped context is canceled before the storage layer completes. The result is events are not saved.
Here's a quick hack of passing a new context to SaveEvent that proves this is the case:
diff --git a/add-event.go b/add-event.go
index b686258..d875fdb 100644
--- a/add-event.go
+++ b/add-event.go
@@ -23,7 +23,7 @@ func AddEvent(ctx context.Context, relay Relay, evt nostr.Event) (accepted bool,
advancedSaver.BeforeSave(ctx, &evt)
}
- if saveErr := store.SaveEvent(ctx, &evt); saveErr != nil {
+ if saveErr := store.SaveEvent(context.Background(), &evt); saveErr != nil {
switch saveErr {
case storage.ErrDupEvent:
return true, saveErr.Error()
I think there are a couple ways to handle this, but ultimately if handlers are to be async then we'll have to use non request-scoped contexts in the child routines.
root@relay:~/relayer/basic# docker-compose up
Creating network "basic_default" with the default driver
Pulling postgres (postgres:)...
latest: Pulling from library/postgres
42c077c10790: Already exists
3c2843bc3122: Pull complete
12e1d6a2dd60: Pull complete
9ae1101c4068: Pull complete
fb05d2fd4701: Pull complete
9785a964a677: Pull complete
16fc798b0e72: Pull complete
f1a0bfa2327a: Pull complete
061440f4e72e: Pull complete
fa9a28f9dd3e: Pull complete
7dc645eb0b15: Pull complete
492c34405b04: Pull complete
dbc5c05afd9a: Pull complete
Digest: sha256:e7ce23491fdb7259d8cb42bc5fd4f22cd38636d6625c98ec0469ff0c99c0a2cc
Status: Downloaded newer image for postgres:latest
Building relay
Sending build context to Docker daemon 16.55MB
Step 1/6 : FROM golang:1.15.5
---> 6d8772fbd285
Step 2/6 : WORKDIR /go/src/app
---> Using cache
---> 23606574f29d
Step 3/6 : COPY ./ .
---> e983f864961b
Step 4/6 : RUN go get -d -v ./...
---> Running in 6d2ad259594f
go: downloading github.com/jmoiron/sqlx v1.3.1
go: downloading github.com/lib/pq v1.8.0
go: downloading github.com/rs/cors v1.7.0
go: downloading github.com/rs/zerolog v1.20.0
go: downloading github.com/gorilla/mux v1.8.0
go: downloading github.com/gorilla/websocket v1.4.2
go: downloading github.com/fiatjaf/go-nostr v0.7.1
go: downloading github.com/btcsuite/btcd v0.22.1
go: downloading github.com/valyala/fastjson v1.6.3
go: downloading github.com/kelseyhightower/envconfig v1.4.0
go: downloading github.com/btcsuite/btcd/btcec/v2 v2.2.0
go: downloading github.com/decred/dcrd/dcrec/secp256k1/v4 v4.0.1
go: downloading github.com/btcsuite/btcd/chaincfg/chainhash v1.0.1
go: downloading github.com/decred/dcrd/crypto/blake256 v1.0.0
Removing intermediate container 6d2ad259594f
---> d03705440bef
Step 5/6 : RUN go install -v ./...
---> Running in 32789306df66
github.com/decred/dcrd/dcrec/secp256k1/v4
# github.com/decred/dcrd/dcrec/secp256k1/v4
/go/pkg/mod/github.com/decred/dcrd/dcrec/secp256k1/[email protected]/loadprecomputed.go:53:22: undefined: io.ReadAll
note: module requires Go 1.16
github.com/btcsuite/btcd/chaincfg/chainhash
github.com/decred/dcrd/crypto/blake256
github.com/gorilla/websocket
github.com/valyala/fastjson/fastfloat
github.com/valyala/fastjson
github.com/gorilla/mux
github.com/kelseyhightower/envconfig
github.com/rs/cors
github.com/rs/zerolog/internal/json
github.com/rs/zerolog
github.com/jmoiron/sqlx/reflectx
github.com/jmoiron/sqlx
github.com/lib/pq/oid
github.com/lib/pq/scram
github.com/lib/pq
github.com/rs/zerolog/log
The command '/bin/sh -c go install -v ./...' returned a non-zero code: 2
ERROR: Service 'relay' failed to build : Build failed
Not really an issue, just a writeup on which data might come in handy for various custom implementations:
AcceptEvent. Why have this information? Some people believing banning IPs or rate-limiting them is better than adding PoW cost or LN invoices. If you want to rate-limit users, you probably want to do it through the connection identifier.Both of these could also be used to send metrics to observe the system in live conditions i.e. how many different connections do we have, average number of events per connection, how many events we send to subscriptions on average, what are the outliers etc.
Was thinking about this lately:
QueryEvents(filter *nostr.Filter) (events []nostr.Event, err error)could be changed to:
QueryEvents(filters ...nostr.Filter) (events []nostr.Event, err error)or:
QueryEvents(filters []nostr.Filter) (events []nostr.Event, err error)so as to enable queries with a single call to SQL (using UNION)... but this would probably break other implementations using the relayer framework.
The current implementation potentially sends duplicate events back to the user (although this could be solved quickly via a deduplication step prior to writing to the websocket). Perhaps the above change would be a more principled solution.
Currently the onAccept event returns a Boolean. What about changing it type to error or some new type. This could help implement nip20.
The delete command get executed and a kind 5 gets recorded, however the original event doesn't get removed from its database.
I am running the expensive relay from master but unable to generate an invoice via the webpage.
Ive got CLN running, my env variables look correct, postgresql running without any issue, bitcoin-cli and lightning-cli are happy. When I go to http://localhost:7447 and paste in a nostr pubkey to generate an invoice I get:
Not authorized: method does not start with list AND method does not start with get AND method is not equal to summary. My commando plug in is not reporting any issues. Any clues on how I can troubleshoot this would be appreciated.
I have setup a nostr relay using expensive-relay with these commands:
torify /root/relay-project/cln/usr/local/bin/lightningd --bitcoin-rpcconnect=REDACTED.onion --bitcoin-rpcport=8332 --bitcoin-rpcuser=REDACTED --bitcoin-rpcpassword=REDACTED --bind-addr=127.0.0.1:12345
POSTGRESQL_DATABASE=postgres://postgres:REDACTED@localhost:5432/postgres LN_NODE_ID=REDACTED CLN_HOST=127.0.0.1:12345 TICKET_PRICE_SATS=10000 CLN_RUNE=REDACTED /usr/local/bin/expensive
I can see the webpage to get invoice:
How do I test the relay now and publish events? I could not see anything using nostcat:
$ nostcat ws://127.0.0.1:7447
^C
The README should contain a list of supported NIPs by this framework and ideally the basic reference implementation, as it is not clear otherwise and going through PRs might not paint the full picture. Thank you.
Minor consideration here, I was wondering why there's no relay struct field
Port int `envconfig:"PORT"`so as to then call server.Start("0.0.0.0", r.Port) in func main(), fetching the value from docker-compose.yml while trying things out locally.
Could this be added with a PR (at least for basic)?
Thanks
Mind pushing a new tag when you get a chance? Quite a few commits since the last one.
There are constants in the app that can be configured or changed. I believe using a config file makes managing constants easier. What do you think?
go version go1.19.2 linux/amd64
relayer v1.5.0 (go install github.com/fiatjaf/relayer/[email protected])
I'm using postgresql.
Started Nostr relayer.
function \"tags_to_tagvalues\" already exists with same argument types"}
<nil> DBG listening addr=127.0.0.1:3003
concurrent write to websocket connection
goroutine 39 [running]:
github.com/gorilla/websocket.(*messageWriter).flushFrame(0xc0003ac0c0, 0x1, {0x0, 0x7f03569d55b8, 0x30})
/home/meeseeks/go/pkg/mod/github.com/gorilla/[email protected]/conn.go:610 +0x52b
github.com/gorilla/websocket.(*messageWriter).Close(0x0)
/home/meeseeks/go/pkg/mod/github.com/gorilla/[email protected]/conn.go:724 +0x45
github.com/gorilla/websocket.(*Conn).beginMessage(0xc0000f4840, 0xc0003ac120, 0x1)
/home/meeseeks/go/pkg/mod/github.com/gorilla/[email protected]/conn.go:473 +0x42
github.com/gorilla/websocket.(*Conn).NextWriter(0xc0000f4840, 0x1)
/home/meeseeks/go/pkg/mod/github.com/gorilla/[email protected]/conn.go:513 +0x45
github.com/gorilla/websocket.(*Conn).WriteJSON(0xc0003ac0f0, {0x6ba0e0, 0xc000446ff0})
/home/meeseeks/go/pkg/mod/github.com/gorilla/[email protected]/json.go:24 +0x45
github.com/fiatjaf/relayer.handleWebsocket.func1.1.3({0xc0000f2c00, 0xd8, 0x200})
/home/meeseeks/go/pkg/mod/github.com/fiatjaf/[email protected]/handlers.go:147 +0x9c5
created by github.com/fiatjaf/relayer.handleWebsocket.func1.1
/home/meeseeks/go/pkg/mod/github.com/fiatjaf/[email protected]/handlers.go:72 +0x2b7
Main process exited, code=exited, status=2/INVALIDARGUMENT
Failed with result 'exit-code'.
Here's my nginx location directive:
location / {
proxy_pass http://127.0.0.1:3003;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
Considering that the relayer repo is a framework for building nostr relayers rather than a concrete implementation, I think that a user of the framework could benefit from the idea of having composable middlewares that can handle events on a pipeline fashion.
Imagine someone wants to enable the following verification/operations before notifying a client about a given event:
In this example event deletions aren't subject to be gated by PoW, but if PoW gate reject the events the chain breaks.
Every one of these actions described above could be implemented as a middleware using a signature similar to this:
type EventHandler func(event *nostr.Event) error
type Middleware func(EventHandler) EventHandler
func EventDeletionMiddleware(next EventHandler) EventHandler {
fn := func(event *Nostr) error {
if deleteEvent() {
doDeletion()
// break the middleware chain execution
return nil
}
// next will execute `PoW` verification
return next(event)
}
return fnrelayer could provide stock middlewares that the community might benefit from importing (most likely those defined on NIP).
Does this sound like a reasonable/useful feature to introduce?
It is possible to reject huge filters, or logging REQ request.
When starting up the project I noticed the websocket was not connecting. Turns out the port is set incorrectly. Because the environment variable isn't being used I removed that too.
PR to fix #78
Please consider providing linux arm64 binaries for the example releases for use on raspberry pis.
The postgresql storage QueryEvents method currently has a hard coded limit of 100 events.
I would like control over the maximum number of events my relay returns and propose adding a QueryLimit field to the exported type PostgresBackend. If set we use that as the upper bound, else we can default it to 100 as it is today.
Would you accept a pull request if I implement this?
Scanning the other storage implementations:
Any recommendations on where/how to go about implementing a rate limit on existing WS connections? For example, a malicious client establishes a connection to the relay then hammers the relay with expensive queries.
A cursory review of the handler code looks like every REQ is handled.
Here is a quick and dirty PoC that adds a rate limiter per connection. If you're OK with this approach I'm happy to make it configurable and submit a PR. Maybe replace the println with a NOTICE
diff --git a/go.mod b/go.mod
index 429ea50..d6927bf 100644
--- a/go.mod
+++ b/go.mod
@@ -23,6 +23,7 @@ require (
github.com/stretchr/testify v1.8.0
github.com/tidwall/gjson v1.14.4
golang.org/x/exp v0.0.0-20230425010034-47ecfdc1ba53
+ golang.org/x/time v0.0.0-20210220033141-f8bda1e9f3ba
)
require (
diff --git a/go.sum b/go.sum
index 66cef16..3e223d5 100644
--- a/go.sum
+++ b/go.sum
@@ -573,6 +573,7 @@ golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.8.0 h1:57P1ETyNKtuIjB4SRd15iJxuhj8Gc416Y78H3qgMh68=
golang.org/x/text v0.8.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8=
golang.org/x/time v0.0.0-20210220033141-f8bda1e9f3ba h1:O8mE0/t419eoIwhTFpKVkHiTs/Igowgfkj25AcZrtiE=
+golang.org/x/time v0.0.0-20210220033141-f8bda1e9f3ba/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/tools v0.0.0-20180525024113-a5b4c53f6e8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20180828015842-6cd1fcedba52/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
diff --git a/handlers.go b/handlers.go
index 2e77178..5ffe2ef 100644
--- a/handlers.go
+++ b/handlers.go
@@ -15,6 +15,7 @@ import (
"github.com/nbd-wtf/go-nostr/nip11"
"github.com/nbd-wtf/go-nostr/nip42"
"golang.org/x/exp/slices"
+ "golang.org/x/time/rate"
)
// TODO: consider moving these to Server as config params
@@ -61,6 +62,8 @@ func (s *Server) HandleWebsocket(w http.ResponseWriter, r *http.Request) {
ws := &WebSocket{
conn: conn,
challenge: hex.EncodeToString(challenge),
+ // 2 RPS, 4 bursts
+ limiter: rate.NewLimiter(2, 4),
}
// reader
@@ -102,6 +105,11 @@ func (s *Server) HandleWebsocket(w http.ResponseWriter, r *http.Request) {
break
}
+ if !ws.limiter.Allow() {
+ fmt.Println("RATE LIMITED")
+ continue
+ }
+
if typ == websocket.PingMessage {
ws.WriteMessage(websocket.PongMessage, nil)
continue
diff --git a/websocket.go b/websocket.go
index a5bdb41..92e48a2 100644
--- a/websocket.go
+++ b/websocket.go
@@ -4,6 +4,7 @@ import (
"sync"
"github.com/fasthttp/websocket"
+ "golang.org/x/time/rate"
)
type WebSocket struct {
@@ -13,6 +14,7 @@ type WebSocket struct {
// nip42
challenge string
authed string
+ limiter *rate.Limiter
}
func (ws *WebSocket) WriteJSON(any interface{}) error {
The Nostr clients I've tested with require the use of TLS. Is this supported? I don't see a way to set a certificate and start the relayer in TLS mode so if I implement a simple client using https://github.com/nbd-wtf/go-nostr I'm getting an error:
2023/02/10 09:43:12 error opening websocket to 'wss://localhost:7447': tls: first record does not look like a TLS handshake
I tried using other clients like the mobile app Damus but it won't connect likely due to the same problem.
Dapp Database clustering has recently become possible with CRDTs and partially trusted datasets. A relay doesn't need to blindly pass messages - but rather nodes can work together to create a logical database. CRDTs are the same data type that power Riak's Bitcask database engine which scales up to hundreds of millions of users (like Riot's Matrix.org project for example).
https://github.com/berty/go-orbit-db
Why is this needed:
It allows nostr to be run as a federated dapp and hosted by a community.
https://www.youtube.com/watch?v=cC-tXnMyiBc
All the storage adapters loose the association between tag key and value and generally just query by value. There's kinda some todos about this in the code and conversation about it on the sqlite3 PR.
But now I'm like "couldn't we just concatenate the key and value together and index that?" In general a storage adapter will index the tags outside of the actual event.tags field for better performance, so storage adapter can do the same at query time.
So the plan:
t->>0 || t->>1, update query to query with key || value too. Would also need to do some one time migration to recompute stored values for existing installs.tag table that is id, key, value... this one can do the more traditional sql thing similar to how it's done in nostr-rs-relaytags field to IndexedEvent that indexes key || value similar to postgres (since ES indexes are similar to gin indexes anyway... follow same approach). I won't worry about an update migration since I doubt anyone is using it except me.Anyway I'm happy to take a stab at this if it sounds good. Just wanted to check that it'd be okay to index the concatenated key + value and that I'm not missing something.
Still missing a handful of commits that were removed from master a week ago. Here's one example.
What can we do to restore these commits?
And based on seeing https://github.com/fiatjaf/khatru, will relayer be maintained moving forward? It seems like khatru is a hard fork.
ERROR: duplicate key value violates unique constraint "pubkeytimeidx"
DETAIL: Key (pubkey, created_at)=(, 10) already exists.
Here is a test-event, that gives invalid signature on this relayer.
{"kind":3,"content":"","created_at":1654748977,"sig":"e13561045e400f6eac07d79ce68c7c0a309a19148a97a4fc902ea8001243e2902dc3afd8bc91a74ab402d4ee05b7fc8564e5d02f8691dde62aa2bba8c387c849","id":"f53d4271290b089cd5347761562934a8b11851f30010ab6d4ac2be6a1dc49bc2","pubkey":"3a48879e58ebdcb0c99d07b5342dc06811f31afb8dead7eba4ae840eb74de9c8","tags":[["p","3a48879e58ebdcb0c99d07b5342dc06811f31afb8dead7eba4ae840eb74de9c8",null,"bobbie"]]}
# github.com/fiatjaf/relayer/storage/sqlite3
../../go/pkg/mod/github.com/fiatjaf/[email protected]/storage/sqlite3/query.go:113:40: filter.Since.Unix undefined (type *nostr.Timestamp has no field or method Unix)
../../go/pkg/mod/github.com/fiatjaf/[email protected]/storage/sqlite3/query.go:117:40: filter.Until.Unix undefined (type *nostr.Timestamp has no field or method Unix)
../../go/pkg/mod/github.com/fiatjaf/[email protected]/storage/sqlite3/query.go:156:19: cannot use time.Unix(timestamp, 0) (value of type time.Time) as nostr.Timestamp value in assignment
../../go/pkg/mod/github.com/fiatjaf/[email protected]/storage/sqlite3/save.go:34:42: evt.CreatedAt.Unix undefined (type nostr.Timestamp has no field or method Unix)
It seems those error already fixed in HEAD. Could you please tag v1.7.4 for the HEAD?
The handler/storage code seems to return the earliest events (uses postgreSQL default ordering, when events[:limit] is returned):
Lines 233 to 235 in 38d0f48
Also perhaps can be optimized by using the limit in the SQL query? The limit filter field does not seem to be used in:
relayer/storage/postgresql/query.go
Line 15 in 38d0f48
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.