Coder Social home page Coder Social logo

y-redis's Introduction

y-redis ๐ŸŽฉ

y-websocket compatible backend using Redis for scalability. This is beta software!

y-redis is an alternative backend for y-websocket. It only requires a redis instance and a storage provider (S3 or Postgres-compatible).

  • Memory efficient: The server doesn't maintain a Y.Doc in-memory. It streams updates through redis. The Yjs document is only loaded to memory for the initial sync.
  • Scalable: You can start as many y-redis instances as you want to handle a fluctuating number of clients. No coordination is needed.
  • Auth: y-redis works together with your existing infrastructure to authenticate clients and check whether a client has read-only / read-write access to a document.
  • Database agnostic: You can persist documents in S3-compatible backends, in Postgres, or implement your own storage provider.

Licensing

y-redis is dual-licensed (either AGPL or proprietary).

Please contact me to buy a license if you intend to use y-redis in your commercial product: <kevin.jahns at pm.me>

Otherwise, you may use this software under the terms of the AGPL, which requires you to publish your source code under the terms of the AGPL too.

Components

Redis is used as a "cache" and a distribution channel for document updates. Normal databases are not fast enough for handling real-time updates of fast-changing applications (e.g. collaborative drawing applications that generate hundreds of operations per second). Hence a redis-cache for temporary storage makes sense to distribute documents as fast as possible to all peers.

A persistent storage (e.g. S3 or Postgres) is used to persist document updates permanently. You can configure in which intervals you want to persist data from redis to the persistent storage. You can even implement a custom persistent storage technology.

The y-redis server component (/bin/server.js) is responsible for accepting websocket-connections and distributing the updates via redis streams. Each "room" is represented as a redis stream. The server component assembles updates stored redis and in the persistent storage (e.g. S3 or Postgres) for the initial sync. After the initial sync, the server doesn't keep any Yjs state in-memory. You can start as many server components as you need. It makes sense to put the server component behind a loadbalancer, which can potentially auto-scale the server component based on CPU or network usage.

The separate y-redis worker component (/bin/worker.js) is responsible for extracting data from the redis cache to a persistent database like S3 or Postgres. Once the data is persisted, the worker component cleans up stale data in redis. You can start as many worker components as you need. It is recommended to run at least one worker, so that the data is eventually persisted. The worker components coordinate which room needs to be persisted using a separate worker-queue (see y:worker stream in redis).

You are responsible for providing a REST backend that y-redis will call to check whether a specific client (authenticated via a JWT token) has access to a specific room / document. Example servers can be found in /bin/auth-server-example.js and /demos/auth-express/server.js.

Missing Features

I'm looking for sponsors that want to sponsor the following work:

  • Ability to kick out users when permissions on a document changed
  • Configurable docker containers for y-redis server & worker
  • Helm chart
  • More exhaustive logging and reporting of possible issues
  • More exhaustive testing
  • Better documentation & more documentation for specific use-cases
  • Support for Bun and Deno
  • Perform expensive tasks (computing sync messages) in separate threads

If you are interested in sponsoring some of this work, please send a mail to <kevin.jahns at pm.me>.

Quick Start (docker-compose)

You can get everything running quickly using docker-compose. The compose file runs the following components:

  • redis
  • minio as a s3 endpoint
  • a single y-redis server
  • a single y-redis worker

This can be a good starting point for your application. If your cloud provider has a managed s3 service, you should probably use that instead of minio. If you want to use minio, you need to setup proper volumes and backups.

The full setup gives insight into more specialized configuration options.

git clone https://github.com/yjs/y-redis.git
cd y-redis
npm i

Setup the environment variables

cp .env.docker.template .env
# generate unique authentication tokens
npx 0ecdsa-generate-keypair --name auth >> .env

The sample configuration configures s3 using minio. Have a look at .env.template for more configuration options.

Run demo

cd ./demos/auth-express
docker compose up
# open http://localhost:5173 in a browser

Full setup

Components are configured via environment variables. It makes sense to start by cloning y-redis and getting one of the demos to work.

Note: If you want to use any of the docker commands, feel free to use podman (a more modern alternative) instead.

Start a redis instance

Setup redis on your computer. Follow the official documentation. This is recommended if you want to debug the redis stream.

Alternatively, simply run redis via docker:

# start the official redis docker container on port 6379
docker run -p 6379:6379 redis
# or `npm run redis`

Start an S3 instance

Setup an S3-compatible store at your favorite cloud provider.

Alternatively, simply run a minio store as a docker container:

docker run -p 9000:9000 -p 9001:9001 quay.io/minio/minio server /data --console-address \":9001\"
# or `npm run minio`

This is just a dev setup. Have a look at the minio documentation if you want to run it in production.

Clone demo

git clone https://github.com/yjs/y-redis.git
cd y-redis
npm i

All features are configurable using environment variables. For local development it makes sense to setup a .env file, that stores project-specific secrets. Use .env.template as a template to setup environment variables. Make sure to read the documentation carefully and configure every single variable.

# setup environment variables
cp .env.template .env
nano .env

Then you can run the different components in separate terminals:

# run the server
npm run start:server
# run a single worker in a separate terminal
npm run start:worker
# start the express server in a separater terminal
cd demos/auth-express
npm i
npm start

Open http://localhost:5173 in a browser.

y-redis's People

Contributors

beorn avatar brecke avatar dmonad avatar doodlewind avatar jackycute avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

y-redis's Issues

Is this working?

This replaces y-leveldb for persisting realtime changes, am I right?

Handling Rapid Connection Closures in WebSocket Open Event to Prevent Errors in uWebSockets.js

Checklist

Describe the bug
I encountered an error when using uWebSockets.js in scenarios where clients quickly refresh or reconnect, specifically during operations within the open event's ws.cork() callback. The error message I received was:

@y/redis/ws: client connection closed (uid=34, code=1001, message="") +65ms
file:///Users/dwz/code/study/y-redis/src/ws.js:187
        ws.cork(() => {
           ^
Error: Invalid access of closed uWS.WebSocket/SSLWebSocket

This error appears to occur because the WebSocket connection may prematurely close right after being established, while server-side code still attempts to perform operations assuming the connection remains open.

Proposed Observation and Workaround:

To address this challenge, I implemented a simple flag (isClosed) within the user's session data to monitor the connection state. This flag helps determine whether operations should proceed based on the connection's current state. Here are the key modifications:

  • In the open handler: Initiate user.isClosed = false to mark the connection as active.
  • Before performing critical operations: Check the user.isClosed flag to ensure the connection is still active.
  • In the close handler: Set user.isClosed = true to mark the connection as closed when such an event occurs.

Example Modification:

open: async (ws) => {
  const user = ws.getUserData();
  user.isClosed = false;

  try {
    if (!user.isClosed) {
      ws.cork(() => {
        if (!user.isClosed) {
          // Safe operations
        }
      });
    }
  } catch (error) {
    console.error("Error in open handler:", error);
  }
},

close: (ws, code, message) => {
  const user = ws.getUserData();
  user.isClosed = true;
}

While this workaround has mitigated the error in my use case, it may be beneficial for the library to include built-in mechanisms to handle such scenarios, or at least to document this potential issue to assist developers in managing similar cases. I would appreciate any thoughts on this approach or suggestions for more integrated solutions within uWebSockets.js.

npx 0ecdsa-generate-keypair --name auth -> is not in registry

hi , swithching from y-websocket to y-redis, when i run
npx 0ecdsa-generate-keypair --name auth

i get a

npm ERR! code E404
npm ERR! 404 Not Found - GET https://registry.npmjs.org/0ecdsa-generate-keypair - Not found
npm ERR! 404 
npm ERR! 404  '0ecdsa-generate-keypair@*' is not in this registry.

any idea ?

i got it working running

cd y-redis/node_modules/lib0
$ ./bin/0ecdsa-generate-keypair.js

Memory Leak Suspected in y-redis during Repeated WebSocket Connections and Disconnections

Checklist

Describe the bug
I am observing a potential memory leak in the y-redis module that becomes apparent during repeated connections and disconnections to the same room ID. Memory usage continuously increases with each connect and disconnect cycle, especially when handling large ProseMirror documents over WebSocket connections. This eventually leads to an out-of-memory error in Node.js.

To Reproduce

  1. Set up a WebSocket server with y-redis using the registerYWebsocketServer function.
  2. Implement a setInterval to log memory usage every 10 seconds:
setInterval(() => {
    const memoryUsage = process.memoryUsage();
    const memoryInMB = {
      rss: (memoryUsage.rss / 1024 / 1024).toFixed(2) + " MB",
      heapTotal: (memoryUsage.heapTotal / 1024 / 1024).toFixed(2) + " MB",
      heapUsed: (memoryUsage.heapUsed / 1024 / 1024).toFixed(2) + " MB",
      external: (memoryUsage.external / 1024 / 1024).toFixed(2) + " MB",
    };
    console.log(`ใ€${new Date().toLocaleString()}ใ€‘`, "###connections counts###", connections.size, "memory:", memoryInMB);
    // @ts-ignore
    heapdump.writeSnapshot((err, filename) => {
      console.log("Heap dump written to", filename);
    });
}, 10000);
  1. Use a ProseMirror document larger than 1MB.
  2. Open and close a WebSocket connection from the frontend multiple times to the same room ID.
  3. Observe the memory usage as it increases with each connect/disconnect cycle.

Expected behavior
I expected that the memory would stabilize after initial allocations given that the connections are being properly closed on the frontend.

Observed Behavior

The heapUsed metric starts at a few tens of megabytes and continuously climbs with each cycle of connecting and disconnecting to the same room ID. This increase does not stabilize or decrease, even after connections are properly closed, leading to a progressive memory buildup. Eventually, the memory usage surpasses 4GB, resulting in a Node.js out-of-memory error and the termination of the program.

Additional context

I have not yet pinpointed the exact cause but am suspecting issues in how resources are handled upon disconnection in the y-redis setup. I am currently unsure how to further investigate and identify the specific root cause of this memory leak, so I am reporting this issue to seek guidance and possibly get more insights into what might be causing this behavior. Any assistance on how to proceed with diagnosing this problem would be greatly appreciated.

Is there any way to synchronize awareness?

The implementation of synchronous YDoc based on the streaming API is awesome, but awareness sync is still not implemented. Is there any difficulty in implementing it?

At the moment, I broadcast update like this.

doc.awareness.on('update', ({ added, updated, removed }: AwarenessUpdate) => {
      const changedClients = added.concat(updated, removed);

      const update = encodeAwarenessUpdate(
          data.document.awareness,
          changedClients || Array.from(doc.awareness.getStates().keys()),
      );

      redis.publishBuffer(awarenessChannel, Buffer.from(update));
});

JWT exp should be unix timestamp according to RFC 7519

Checklist

Describe the bug
I believe the current implementation on exp claim is not conform to RFC 7519

[The "exp" (expiration time)] value MUST be a number containing a NumericDate value.

Also, the NumericDate value is described here

NumericDate
A JSON numeric value representing the number of seconds from
1970-01-01T00:00:00Z UTC until the specified UTC date/time,
ignoring leap seconds.

To Reproduce
Steps to reproduce the behavior:

  1. Generate a JWT using the RFC implementation of exp
  2. Try to connect to the y-redis server
  3. See error
Failed to auth to endpoint /y-redis-demo-app Error: Expired JWT
    at Module.verifyJwt (file:///Users/naydenoff/dev/y-red-dve/y-redis/node_modules/lib0/crypto/jwt.js:51:11)
    at async registerYWebsocketServer.redisPrefix.redisPrefix (file:///Users/naydenoff/dev/y-red-dve/y-redis/src/server.js:57:38)
    at async upgrade (file:///Users/naydenoff/dev/y-red-dve/y-redis/src/ws.js:117:50)

Expected behavior
The exp field should be unix timestamp to match RFC 7519.

Screenshots
A screenshot showing the current expiration date being 4/25/56333
image

A screenshot showing the expected expiration date being 5/12/2024
image

(Yes, the token is meant to be valid 5s for test purposes in this screenshot)

Environment Information

  • Browser / Node.js [e.g. Chrome, Firefox, Node.js]
    Node 20
  • Yjs version and the versions of the y-* modules you are using [e.g. yjs v13.0.1, y-webrtc v1.2.1]. Use npm ls yjs to find out the exact version you are using.
@y/[email protected] /y-redis
โ”œโ”€โ”ฌ [email protected]
โ”‚ โ”œโ”€โ”ฌ [email protected]
โ”‚ โ”‚ โ””โ”€โ”€ [email protected] deduped
โ”‚ โ”œโ”€โ”ฌ [email protected]
โ”‚ โ”‚ โ””โ”€โ”€ [email protected] deduped
โ”‚ โ””โ”€โ”€ [email protected] deduped
โ””โ”€โ”€ [email protected]

Additional context
I understand that it does work if I generate a token with the expected exp of y-redis.
But as it is supposed to integrate with the user's server, I believe it's best if we respect the RFC so that anyone can just use the project without surprises.

Handling of errors in uWebSockets handlers

Is your feature request related to a problem? Please describe.
Currently an error in one of the uWebSockets handlers crashes the server, e.g. when getDoc fails in the open handler. So e.g. when a file for one room can't be retrieved from S3, the server crashes and all clients connected to other rooms also lose their connection.

Unfortunately I did not find a possibility to attach some kind of error handler like in express. (Or is there one? I'm rather new to uWebSockets.)

Describe alternatives you've considered
One way could be to wrap the code in the handlers in a try-catch and execute a given callback (that would be parameter of registerYWebsocketServer) in the catch, e.g.:

open: async (ws) => {
   try {
        const user = ws.getUserData()
        ...
    } catch (error) {
        errorCallback(error, ws)
    }
},

switch to io-redis to be able to work with redis-sentinel

Checklist

[ ] Are you reporting a bug? Use github issues for bug reports and feature requests. For general questions, please use https://discuss.yjs.dev/
[ ] Try to report your issue in the correct repository. Yjs consists of many modules. When in doubt, report it to https://github.com/yjs/yjs/issues/

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
We want to run redis as a cluster using sentinel as far as i know only the io-redis library supports redis sentinel, it also has a retry mechanism which i'm keen to use

Describe the solution you'd like
A clear and concise description of what you want to happen.
use io-redis instead of basic redis library

Describe alternatives you've considered
none

  • I'm a sponsor ๐Ÿ’–
  • This feature is critical for my project.

Incorrect redis hostname in the env.docker.template results in getaddrinfo ENOTFOUND

Describe the bug
Incorrect redis hostname in the env.docker.template results in getaddrinfo ENOTFOUND

To Reproduce
Steps to reproduce the behavior:

  1. Follow the Quick Start guide here - https://github.com/yjs/y-redis/tree/master?tab=readme-ov-file#quick-start-docker-compose
  2. Run the following commands
    git clone https://github.com/yjs/y-redis.git
    cd y-redis
    npm i
    cp .env.docker.template .env
    npx 0ecdsa-generate-keypair --name auth >> .env
  3. Here's how the .env looks like:
# This is a simple configuration to get this running using docker compose.

REDIS=redis://docker:6379

S3_ENDPOINT=minio
S3_PORT=9000
S3_SSL=false
S3_ACCESS_KEY=minioadmin
S3_SECRET_KEY=minioadmin

AUTH_PERM_CALLBACK=http://demo:5173/auth/perm
YDOC_UPDATE_CALLBACK=http://demo:5173/ydoc

LOG=* # log everything

## Generate the auth tokens with `npx 0ecdsa-generate-keypair --name auth >> .env

AUTH_PUBLIC_KEY={"key_ops":["verify"],"ext":true,"kty":"EC","x":"mI2DWTzk2Xxwzng07DK2TmAAXccw0L-SOji1MPbbJxagxOhr5IZlSP2THq3pe4Qy","y":"qaWkkuHq_Xbw64cRHsdZkF1W7QJtYoYa2K-dgclZlIm0Zv4EjeZM1Ql3RmiBqgP1","crv":"P-384"}
AUTH_PRIVATE_KEY={"key_ops":["sign"],"ext":true,"kty":"EC","x":"mI2DWTzk2Xxwzng07DK2TmAAXccw0L-SOji1MPbbJxagxOhr5IZlSP2THq3pe4Qy","y":"qaWkkuHq_Xbw64cRHsdZkF1W7QJtYoYa2K-dgclZlIm0Zv4EjeZM1Ql3RmiBqgP1","crv":"P-384","d":"5NefOVtM_NOxH5Chq5EvqZ_wyc_OueXtzGrV2FxwnUMz_Ik0J4TgpIsFCCfk6TnX"}
  1. Run demo:
    cd ./demos/auth-express
    docker compose up
  2. Get the error listed below
worker-1  | using s3 store
server-1  | using s3 store
demo-1    | Express Demo Auth server listening on port 5173
worker-1  | node:internal/process/promises:289
worker-1  |             triggerUncaughtException(err, true /* fromPromise */);
worker-1  |             ^
worker-1  |
worker-1  | Error: getaddrinfo ENOTFOUND docker
worker-1  |     at GetAddrInfoReqWrap.onlookupall [as oncomplete] (node:dns:120:26)
worker-1  | Emitted 'error' event on Commander instance at:
worker-1  |     at RedisSocket.<anonymous> (/usr/src/app/node_modules/@redis/client/dist/lib/client/index.js:412:14)
worker-1  |     at RedisSocket.emit (node:events:519:28)
worker-1  |     at RedisSocket._RedisSocket_connect (/usr/src/app/node_modules/@redis/client/dist/lib/client/socket.js:166:18)
worker-1  |     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
worker-1  |     at async Commander.connect (/usr/src/app/node_modules/@redis/client/dist/lib/client/index.js:185:9)
worker-1  |     at async createApiClient (file:///usr/src/app/src/api.js:93:3)
worker-1  |     at async Module.createWorker (file:///usr/src/app/src/api.js:339:13) {
worker-1  |   errno: -3008,
worker-1  |   code: 'ENOTFOUND',
worker-1  |   syscall: 'getaddrinfo',
worker-1  |   hostname: 'docker'
worker-1  | }
worker-1  |
worker-1  | Node.js v20.13.0
server-1  | node:internal/process/promises:289
server-1  |             triggerUncaughtException(err, true /* fromPromise */);
server-1  |             ^
server-1  |
server-1  | Error: getaddrinfo ENOTFOUND docker
server-1  |     at GetAddrInfoReqWrap.onlookupall [as oncomplete] (node:dns:120:26)
server-1  | Emitted 'error' event on Commander instance at:
server-1  |     at RedisSocket.<anonymous> (/usr/src/app/node_modules/@redis/client/dist/lib/client/index.js:412:14)
server-1  |     at RedisSocket.emit (node:events:519:28)
server-1  |     at RedisSocket._RedisSocket_connect (/usr/src/app/node_modules/@redis/client/dist/lib/client/socket.js:166:18)
server-1  |     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
server-1  |     at async Commander.connect (/usr/src/app/node_modules/@redis/client/dist/lib/client/index.js:185:9)
server-1  |     at async Module.createApiClient (file:///usr/src/app/src/api.js:93:3)
server-1  |     at async Promise.all (index 0)
server-1  |     at async registerYWebsocketServer (file:///usr/src/app/src/ws.js:82:32)
server-1  |     at async Module.createYWebsocketServer (file:///usr/src/app/src/server.js:43:3) {
server-1  |   errno: -3008,
server-1  |   code: 'ENOTFOUND',
server-1  |   syscall: 'getaddrinfo',
server-1  |   hostname: 'docker'
server-1  | }
server-1  |
server-1  | Node.js v20.13.0
worker-1 exited with code 1
server-1 exited with code 1
  1. Observe that changing the Redis hostname in the .env to: REDIS=redis://redis:6379 fixes the above issue.

Expected behavior
Build the project using the Quick Start guide.

Environment Information

"dependencies": {
    "lib0": "^0.2.93",
    "redis": "^4.6.12",
    "uws": "github:uNetworking/uWebSockets.js#v20.40.0",
    "yjs": "^13.5.6"
  },
  "optionalDependencies": {
    "postgres": "^3.4.3",
    "minio": "^7.1.3"
  },
  "engines": {
    "npm": ">=8.0.0",
    "node": ">=20.0.0"
  },
  "devDependencies": {
    "@codemirror/lang-javascript": "^6.2.2",
    "@codemirror/state": "^6.4.1",
    "@codemirror/view": "^6.24.1",
    "@rollup/plugin-commonjs": "^25.0.7",
    "@rollup/plugin-node-resolve": "^15.2.3",
    "@types/node": "^20.11.5",
    "@types/ws": "^8.5.10",
    "codemirror": "^6.0.1",
    "concurrently": "^8.2.2",
    "rollup": "^4.12.0",
    "standard": "^17.1.0",
    "typescript": "^5.3.3",
    "ws": "^8.16.0",
    "y-codemirror.next": "^0.3.2",
    "y-websocket": "^2.0.0"
  }
docker version
Client:
 Cloud integration: v1.0.35+desktop.5
 Version:           24.0.7
 API version:       1.43
 Go version:        go1.20.10
 Git commit:        afdd53b
 Built:             Thu Oct 26 09:04:20 2023
 OS/Arch:           darwin/arm64
 Context:           desktop-linux

Server: Docker Desktop 4.26.1 (131620)
 Engine:
  Version:          24.0.7
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.10
  Git commit:       311b9ff
  Built:            Thu Oct 26 09:08:15 2023
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.6.25
  GitCommit:        d8f198a4ed8892c764191ef7b3b06d8a2eeb5c7f
 runc:
  Version:          1.1.10
  GitCommit:        v1.1.10-0-g18a0cb0
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Additional context
After changing the hostname, the error disappears but a new issue appears which I've outlined in this ticket - #23

Failed to auth to endpoint /y-redis-demo-app Error: Invalid JWT

Describe the bug
Invalid JWT after issuing keys and running the docker as per the guide.

To Reproduce
Steps to reproduce the behavior:

  1. Follow the Quick Start guide here - https://github.com/yjs/y-redis/tree/master?tab=readme-ov-file#quick-start-docker-compose
  2. Run the following commands
    git clone https://github.com/yjs/y-redis.git
    cd y-redis
    npm i
    cp .env.docker.template .env
    npx 0ecdsa-generate-keypair --name auth >> .env
  3. Here's how the .env looked like after running:
# This is a simple configuration to get this running using docker compose.

REDIS=redis://redis:6379

S3_ENDPOINT=minio
S3_PORT=9000
S3_SSL=false
S3_ACCESS_KEY=minioadmin
S3_SECRET_KEY=minioadmin

AUTH_PERM_CALLBACK=http://demo:5173/auth/perm
YDOC_UPDATE_CALLBACK=http://demo:5173/ydoc

LOG=* # log everything

## Generate the auth tokens with `npx 0ecdsa-generate-keypair --name auth >> .env

AUTH_PUBLIC_KEY={"key_ops":["verify"],"ext":true,"kty":"EC","x":"mI2DWTzk2Xxwzng07DK2TmAAXccw0L-SOji1MPbbJxagxOhr5IZlSP2THq3pe4Qy","y":"qaWkkuHq_Xbw64cRHsdZkF1W7QJtYoYa2K-dgclZlIm0Zv4EjeZM1Ql3RmiBqgP1","crv":"P-384"}
AUTH_PRIVATE_KEY={"key_ops":["sign"],"ext":true,"kty":"EC","x":"mI2DWTzk2Xxwzng07DK2TmAAXccw0L-SOji1MPbbJxagxOhr5IZlSP2THq3pe4Qy","y":"qaWkkuHq_Xbw64cRHsdZkF1W7QJtYoYa2K-dgclZlIm0Zv4EjeZM1Ql3RmiBqgP1","crv":"P-384","d":"5NefOVtM_NOxH5Chq5EvqZ_wyc_OueXtzGrV2FxwnUMz_Ik0J4TgpIsFCCfk6TnX"}
  1. Run demo:
    cd ./demos/auth-express
    docker compose up
  2. Get the error listed below
worker-1  | using s3 store
server-1  | using s3 store
worker-1  |
demo-1    | Express Demo Auth server listening on port 5173
server-1  |
server-1  | Failed to auth to endpoint /y-redis-demo-app Error: Invalid JWT
server-1  |     at Module.verifyJwt (file:///usr/src/app/node_modules/lib0/crypto/jwt.js:47:11)
server-1  |     at async registerYWebsocketServer.redisPrefix.redisPrefix (file:///usr/src/app/src/server.js:51:36)
server-1  |     at async upgrade (file:///usr/src/app/src/ws.js:117:50)
server-1  | Failed to auth to endpoint /y-redis-demo-app Error: Invalid JWT
server-1  |     at Module.verifyJwt (file:///usr/src/app/node_modules/lib0/crypto/jwt.js:47:11)
server-1  |     at async registerYWebsocketServer.redisPrefix.redisPrefix (file:///usr/src/app/src/server.js:51:36)
server-1  |     at async upgrade (file:///usr/src/app/src/ws.js:117:50)
server-1  | Failed to auth to endpoint /y-redis-demo-app Error: Invalid JWT
server-1  |     at Module.verifyJwt (file:///usr/src/app/node_modules/lib0/crypto/jwt.js:47:11)
server-1  |     at async registerYWebsocketServer.redisPrefix.redisPrefix (file:///usr/src/app/src/server.js:51:36)
server-1  |     at async upgrade (file:///usr/src/app/src/ws.js:117:50)
server-1  | Failed to auth to endpoint /y-redis-demo-app Error: Invalid JWT
server-1  |     at Module.verifyJwt (file:///usr/src/app/node_modules/lib0/crypto/jwt.js:47:11)
server-1  |     at async registerYWebsocketServer.redisPrefix.redisPrefix (file:///usr/src/app/src/server.js:51:36)
server-1  |     at async upgrade (file:///usr/src/app/src/ws.js:117:50)
server-1  | Failed to auth to endpoint /y-redis-demo-app Error: Invalid JWT
server-1  |     at Module.verifyJwt (file:///usr/src/app/node_modules/lib0/crypto/jwt.js:47:11)
server-1  |     at async registerYWebsocketServer.redisPrefix.redisPrefix (file:///usr/src/app/src/server.js:51:36)
server-1  |     at async upgrade (file:///usr/src/app/src/ws.js:117:50)
server-1  | Failed to auth to endpoint /y-redis-demo-app Error: Invalid JWT
server-1  |     at Module.verifyJwt (file:///usr/src/app/node_modules/lib0/crypto/jwt.js:47:11)
server-1  |     at async registerYWebsocketServer.redisPrefix.redisPrefix (file:///usr/src/app/src/server.js:51:36)
server-1  |     at async upgrade (file:///usr/src/app/src/ws.js:117:50)

Expected behavior
Build the project using the Quick Start guide.

Environment Information

"dependencies": {
    "lib0": "^0.2.93",
    "redis": "^4.6.12",
    "uws": "github:uNetworking/uWebSockets.js#v20.40.0",
    "yjs": "^13.5.6"
  },
  "optionalDependencies": {
    "postgres": "^3.4.3",
    "minio": "^7.1.3"
  },
  "engines": {
    "npm": ">=8.0.0",
    "node": ">=20.0.0"
  },
  "devDependencies": {
    "@codemirror/lang-javascript": "^6.2.2",
    "@codemirror/state": "^6.4.1",
    "@codemirror/view": "^6.24.1",
    "@rollup/plugin-commonjs": "^25.0.7",
    "@rollup/plugin-node-resolve": "^15.2.3",
    "@types/node": "^20.11.5",
    "@types/ws": "^8.5.10",
    "codemirror": "^6.0.1",
    "concurrently": "^8.2.2",
    "rollup": "^4.12.0",
    "standard": "^17.1.0",
    "typescript": "^5.3.3",
    "ws": "^8.16.0",
    "y-codemirror.next": "^0.3.2",
    "y-websocket": "^2.0.0"
  }
docker version
Client:
 Cloud integration: v1.0.35+desktop.5
 Version:           24.0.7
 API version:       1.43
 Go version:        go1.20.10
 Git commit:        afdd53b
 Built:             Thu Oct 26 09:04:20 2023
 OS/Arch:           darwin/arm64
 Context:           desktop-linux

Server: Docker Desktop 4.26.1 (131620)
 Engine:
  Version:          24.0.7
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.10
  Git commit:       311b9ff
  Built:            Thu Oct 26 09:08:15 2023
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.6.25
  GitCommit:        d8f198a4ed8892c764191ef7b3b06d8a2eeb5c7f
 runc:
  Version:          1.1.10
  GitCommit:        v1.1.10-0-g18a0cb0
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Additional context

  1. I tried running the demo in separate terminals after installing redis using the following commands as per the full setup guide:
docker run -p 6379:6379 redis
docker run -p 9000:9000 -p 9001:9001 quay.io/minio/minio server /data --console-address ":9001"
npm run start:server
npm run start:worker

And then running the demo

cd demos/auth-express
npm i
npm start

In one of the attempts I got a different JWT error from the server - Expired JWT in addition to Invalid JWT:

Failed to auth to endpoint /y-redis-demo-app Error: Expired JWT
    at Module.verifyJwt (file:///Users/naydenoff/dev/y-red-dve/y-redis/node_modules/lib0/crypto/jwt.js:51:11)
    at async registerYWebsocketServer.redisPrefix.redisPrefix (file:///Users/naydenoff/dev/y-red-dve/y-redis/src/server.js:57:38)
    at async upgrade (file:///Users/naydenoff/dev/y-red-dve/y-redis/src/ws.js:117:50)
Failed to auth to endpoint /y-redis-demo-app Error: Invalid JWT
    at Module.verifyJwt (file:///Users/naydenoff/dev/y-red-dve/y-redis/node_modules/lib0/crypto/jwt.js:47:11)
    at async registerYWebsocketServer.redisPrefix.redisPrefix (file:///Users/naydenoff/dev/y-red-dve/y-redis/src/server.js:57:38)
  1. In a separate application I tried authenticating using lib0 with the token that I logged from y-redis-demo-app, and it wasn't marked as invalid or expired.

Allow custom configuration for cluster connections

Is your feature request related to a problem? Please describe.
I'm presently using y-redis via the redis extension for hocus-pocus.
When you create a new cluster object using ioredis you can pass two parameters, the first for the initial startupNodes and the second for additional options.
https://github.com/luin/ioredis/blob/master/API.md#new-clusterstartupnodes-options

AFAICT y-redis is coded to only pass the first, leaving us with the default options for the second:
https://github.com/yjs/y-redis/blob/master/src/y-redis.js#L106

Describe the solution you'd like
It would be great to be able to pass custom configuration with the options object when initialising a cluster connection.

Additional context
The immediate problem that led me to write this issue is described here:
https://github.com/luin/ioredis#special-note-aws-elasticache-clusters-with-tls
The resolution being to pass additional config.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.