asjas / prisma-redis-middleware Goto Github PK

Describe the bug
I copied the code form README page and although the code is working well, typescript gives me an error.

Code :

import Prisma from 'prisma';
import Redis from 'ioredis';
import { createPrismaRedisCache } from 'prisma-redis-middleware';

const redis = new Redis();

const cacheMiddleware: Prisma.Middleware = createPrismaRedisCache({
  storage: {
    type: 'redis', // error here : Type '"redis"' is not assignable to type '"memory"'.ts(2322)
    options: {
      client: redis,
      invalidation: { // error here : Type '{ referencesTTL: number; }' is not assignable to type 'boolean | undefined'.ts(2322)
        referencesTTL: 300
      },
      log: console
    }
  },
  // rest of the code
});

Desktop (please complete the following information):

• OS: VSCode (1.70.2) devcontainer on WSL2 (Debian)

This issue provides visibility into Renovate updates and their statuses. Learn more

This repository currently has no open or pending branches.

• Check this box to trigger a request for Renovate to run again on this repository

Currently, custom MiddlewareParams is being used, but this makes an type error: Property 'model' is optional in type 'MiddlewareParams' but required in type 'MiddlewareParams'. because Prisma only have to use their own MiddlewareParams type.

In order to resolve this problem, we can simply remove the MiddlewareParams and, import MiddlewareParams from Prisma .

import { Prisma } from "@prisma/client";
 :
export declare type MiddlewareParams = Prisma.MiddlewareParams

I can't propose a solution for it right now, but I think it is a valuable issue to map over here once...

With a pseudo schema like

model User {
	id String
	nodes Node[]
}
model Note {
	id String
	text String
}

if I try to get something like

const getUserNotes = async (userId: string) => {
  const { notes } = await prisma.findUnique({ where: { id: userId }, include: { notes: true } });
  return notes
};

it would produce the key:

{
  params: {
    action: "findUnique",
    args: { select: { notes: true }, where: { id: "ckyen2nkj0579g171l47kfz93" } },
    dataPath: ["select", "notes"],
    model: "User",
    runInTransaction: false
  }
}

then if I happen to delete a note, there would be no way for the cache to be able to know it was invalidated.
and excluding Note model from cache wouldn't fix it as this query would be cached on User model

I know it isn't so simple to fix this... but may be worth noticing it here

I'm unable to add this project because it's missing Node 18 as one of the engines.

error [email protected]: The engine "node" is incompatible with this module. Expected version "^14.x || ^16.x". Got "18.0.0"
error Found incompatible module.

Would love to see it added/updated, thank you!

Describe the bug
When using this config the cacheTime refers to seconds, not milliseconds as described in README.

const CacheMiddleware = createPrismaRedisCache({
  cacheTime: 10,
  onHit: (key) => {
    console.log('hit', key)
  },
  onMiss: (key) => {
    console.log('miss', key)
  },
  onError: (key) => {
    console.log('error', key)
  },
})

To Reproduce
Steps to reproduce the behavior:

1. Setup middleware with config from above
2. Run query
3. Wait 2 seconds and run query again
4. onHit is triggered

Expected behavior
onMiss should be triggered

Desktop (please complete the following information):

• OS: MacOS 12.4
• Node: 16.12.0
• Version 4.0.8

Describe the bug
Currently, the keys are being added to the root, without a namespace identifier, if you use a GUI this will be your view (available in screenshot section)

Expected behavior
A clean view of my keys, and I can easily invalidate the cache by just deleting the namespace (root).

Screenshots

So it works now... (After I did FLUSHALL)

Why is there sometimes an r:MODELNAME and k:MODELNAME tho?

and sometimes not?!

Can u explain that ?

Thanks

Hi there! Love the project, it's been awesome so far.

I would really appreciate if one could skip some methods so they wouldn't be cached (e.g. I would like to skip findMany). This could be a configuration option of createPrismaRedisCache, maybe having the inverse of this (an array of all allowed methods), would also benefit many projects I'm sure.

The count method is being incorrectly referred to as aggregate in Prisma Middleware.

const count = await prisma.user.count();

{"action":"aggregate","args":{"select":{"_count":{"select":{"_all":true}}}},"dataPath":[],"model":"User","runInTransaction":false}

prisma/prisma#12222

Would be great if this library support clustered redis

Describe the bug
Seeing the following error on Node.js v18.16.0

/app/node_modules/async-cache-dedupe/src/storage/index.js:36
throw new Error('Redis storage is not supported in the browser')

Error: Redis storage is not supported in the browser.

Running a backend api server in Docker container with node:18.16-buster-slim as base image.
Using ioredis to handle redis interactions.
Prisma middleware is not the only place I use redis in, so I'm sure ioredis works and behaves fine.

I don't know why it thinks it is running in a browser, is there a workaround to this?

My method:

await prisma.settings.upsert({
    where: { 
        guildId: message.guildId
    },
    create: getDefaultSettingsData(message.guildId, {prefix: args[0]}),
    update: { prefix: args[0] },
}).catch(console.error)

My setup:

prisma.$use(createPrismaRedisCache({...cacheData, storage: { 
  type: "redis",
  models: [
    { model: "Settings" },
  ],
  cacheTime: 1300, 
  excludeCacheMethods: ["findMany"],
  options: { 
    client: redis, invalidation: { referencesTTL: 3600, size: 1000000 }
  }
}}));

My model:

model Settings {
  guildId                String         @id
  prefix                 String
  botchannels            String[] // on lists there cant be default
  language               Languages      @default(en)
  alldaymusic            String         @default("disabled") // "disabled" || "{ channel: '', link: '' }"
  defaultvolume          Int            @default(100)
  defaultautoplay        Boolean        @default(true)
  allowyoutube           Boolean        @default(false)
  defaultsearchplattform SearchPlatform @default(ytm)
  extraData              String? // can be anything inside a JSON.stringify({});
}

Now when i call the upsert (update) method
It's not updating the redis-cache and still pulling OLD CACHE DATA..

meaning if prefix was before: "!" and i change it to "$" in the db it says "$" but when i do:

const data = await prisma.settings.findFirst({
  where: {
    guildId: message.guildId
  },
  select: {
    prefix: true, otheroptions
  }
});

the data.prefix still says it's "!"

It only does not work when i fetch multiple fields at the same time....
what could be the problem for that?!?!

Describe the bug
Seems to be no caching when using create or upsert methods, can I ask why?

My test config:

    const cacheMiddleware: Prisma.Middleware = createPrismaRedisCache({
      storage: { type: 'memory', options: { invalidation: true } },
      cacheTime: 4096,
      onMiss(key) {
        logger.cache(JSON.parse(key).params.model);
      },
      onError: (error) => {
        logger.error('RedisCacheError', exactLine(), error);
      },
    });

    prisma.$use(cacheMiddleware);

Currently type of excludeModels is string[], to use Prisma.ModelName[], we can ensure right input

Describe the bug
executeRaw and executeRawUnsafe did not invalidate model caches.

To Reproduce
Steps to reproduce the behavior:

1. Do findMany action of a model
2. Do an update to their columns using executeRaw or executeRawUnsafe
3. Do findMany action of a model again, cache hit
4. The result of the second findMany is using the cache despite mutation from executeRaw or executeRawUnsafe

Expected behavior
It should invalidate the caches.

Screenshots

Additional context
As you can see from the screenshot, I'm currently using executeRawUnsafe, but the params.action is executeRaw, and executeRaw is not in defaultMutationMethods. I'm currently using Prisma 5.6.0. I'm not sure if this would also apply to Prisma 4 or lower.

I've made a PR for this #660

In cache I see keys that are either ${modelName}~, k:${modelName}~ or r:${modelName}~. What do the prefixes mean?

Is there going to be support for nodejs version 19.x? If not, why?

Describe the bug
When setting the keyPrefix to any value the cache doen's get invalidate, in this screenshot the value is app:dev:jobs:prisma:

Expected behavior
Cache should invalidate on mutation with any keyPrefix string set

Screenshots
With keyPrefix

W/o keyPrefix

Additional context
This is run from the example and the redis and db has been flushed before executing

Hello,

I'm excited to share that Nove.js v20 has been recently released, and I saw that the prisma-redis-middleware already supports the new Node.js engine.
Therefore, I kindly request an update to the npm package to ensure compatibility with this latest version. Your prompt action on this matter would be greatly appreciated.
Thank you!

so i'm not sure if this is intended or not, but when fetching data the cache works fine, but when you then update that data, the cache still exists so when you fetch again, you get out of date data.

any chance this could be added? i thought it would be an expected feature

At the moment this Prisma middleware isn't very flexible and I feel that the caching API isn't that nice to use.

prismaClient.$use(
  createPrismaRedisCache({
    models: ["User", "Post"],
    cacheTime: 300,
    redis,
    excludeCacheMethods: ["findMany"],
  }),
);

I want to update the API to possibly look like this:

• models: Array of objects. The key is the model to cache and the value is the time to cache it.
• excludeModels: Array of strings. These models will NOT be cached.
• redis: Optional. Falls back to using an in-memory LRU cache
• cacheTime: Still an integer
• excludeCacheMethods: Still an array of methods to ignore when caching

Example 1:

prismaClient.$use(
  createPrismaRedisCache({
    cacheTime: 300,
  }),
);

You can invoke the middleware while leaving the models out which will cause all models to be cached based on cacheTime.

And if you leave the redis instance out this should fall back to using an in-memory LRU cache. This should make it easier for someone to test this in development without needing to configure or install Redis.

Thoughts: In such a case it should print out a warning stating that an in-memory cache is being used and that you should pass a Redis instance to persist the cache.

Example 2:

prismaClient.$use(
  createPrismaRedisCache({
    models: [{ "User", 300 }, { "Post": 900 }],
    cacheTime: 300,
    redis,
    excludeCacheMethods: ["findMany"],
  }),
);

You can specify the caching time per model.

The global cacheTime middleware option is then used for any other Model that isn't explicitly specified in models.

You can then specify a value of 0 to cacheTime or leave the cacheTime option out to stop it from caching any explicit models you didn't specify.

Some thoughts about the models API

I don't know if I want the models to have an API that looks like this.

 models: [{ "User", 300 }, { "Post": 900 }],

Or like this:

 models: [{ model: "User", cacheTime: 300 }, { model: "Post", cacheTime: 900 }],

I like the 2nd one, but it also feels a slightly verbose... 🤔

Some other thoughts.

1. Should I just ignore cacheTime if there is a models value (which will include a caching value per model)? Or should I use it for any model that isn't specified?

2. Allowing someone to specify their own caching key per model...? yikes

3. Allowing an option called excludeModels which could be an Array of strings which will NOT be cached?

Describe the bug
A clear and concise description of what the bug is.

Prisma.$queryRaw is being cached but documentation mentions that it should not happen.
They generated key has the format:

{ "params":{ "action": "queryRaw", "args": { "parameters": ... } } }

To Reproduce
Steps to reproduce the behavior:

1. Configure onHit and onMiss functions to print the key
2. Execute a query with Prisma.$queryRaw -- > onMiss will be executed
3. Execute same query with as step 2 -- > onHit will be executed

Expected behavior
To not trigger the cache because of being 'queryRaw' query type

Screenshots

Desktop (please complete the following information):

• OS: any
• Browser any
• Version any

Smartphone (please complete the following information):

• Device: any
• OS: any
• Browser any
• Version any

Additional context

"@prisma/client": "^4.8.0"

Possible fix
include "queryRaw" in defaultMutationMethods array

Describe the bug
We have a high traffic app which is running in production, in there the read throughput is usually ~30k-50k read/write queries/per second, 24/7. And we're facing a serious issue where read query is stuck randomly, without any reason.

Overview: we have 3 tables:

-Order (has a field name orderNumber)

• Product
• OrderItem: has a foreign key to Order and Product

The structure is simple.

Our flow is, before creating new order we need to check if product exist using findFirst and check if order number exist using findFirst

During peak hours, the 2 find queries failed randomly at about every 1 hour, they're stuck, never return.

Here are what I have tried:

• reduce cacheTime and referencesTTL, it improves a bit , but the error still happens after few hours
• remove invalidation: better than keeping invalidation, but error is still there
• completely disable caching: works fine, no issue at all

(I made change and observed during 1 week)

Server:

• OS: Docker, Node 14 Alpine, NextJS 13
• Browser: Chrome
• Version: "@prisma/client": "^4.5.0", "prisma": "^4.6.1", "prisma-redis-middleware": "^4.3.0"

Describe the bug
the version [email protected] giving error types:

Type 'import("G:/Projects/Sitra/node_modules/.pnpm/[email protected]/node_modules/ioredis/built/Redis").default' is not assignable to type 'import("G:/Projects/Sitra/node_modules/.pnpm/[email protected]/node_modules/ioredis/built/Redis").default'.
  Types of property 'options' are incompatible.
    Type 'import("G:/Projects/Sitra/node_modules/.pnpm/[email protected]/node_modules/ioredis/built/redis/RedisOptions").RedisOptions' is not assignable to type 'import("G:/Projects/Sitra/node_modules/.pnpm/[email protected]/node_modules/ioredis/built/redis/RedisOptions").RedisOptions'.
      Type 'RedisOptions' is not assignable to type 'CommonRedisOptions'.
        The types returned by 'new Connector(...)' are incompatible between these types.
          Type 'import("G:/Projects/Sitra/node_modules/.pnpm/[email protected]/node_modules/ioredis/built/connectors/AbstractConnector").default' is not assignable to type 'import("G:/Projects/Sitra/node_modules/.pnpm/[email protected]/node_modules/ioredis/built/connectors/AbstractConnector").default'.
            Property 'connecting' is protected but type 'AbstractConnector' is not a class derived from 'AbstractConnector'.

To Reproduce
Steps to reproduce the behavior:
use [email protected] with typescript and nodejs

import Prisma from 'prisma';
import { PrismaClient } from '@prisma/client';
import { createPrismaRedisCache } from 'prisma-redis-middleware';
import Redis from 'ioredis';

export default (
  prisma: PrismaClient<
    Prisma.PrismaClientOptions,
    never,
    Prisma.RejectOnNotFound | Prisma.RejectPerOperation
  >
) => {
  const redis = new Redis();
  const cacheMiddleware: Prisma.Middleware = createPrismaRedisCache({
    models: [{ model: 'User', cacheTime: 60 }],
    storage: {
      type: 'redis',
      options: { client: redis, invalidation: { referencesTTL: 300 } },
    },
    cacheTime: 300,
    onHit: (key) => {
      console.log('hit', key);
    },
    onMiss: (key) => {
      console.log('miss', key);
    },
    onError: (key) => {
      console.log('error', key);
    },
  });

  prisma.$use(cacheMiddleware);
};

Desktop (please complete the following information):

• OS: Windows 11 pro
• Nodejs
• Version 16.16

Additional context
I guess you just need to update the dependency of the project

The count method is returning an object { _count: { _all: 2 } } and not a number such as 2. Based on some testing the issue seems to come from the async-cache-dedupe cache function. Something is causing it to act differently.

// Add a cache function for every model that hasn't been registered yet or excluded
if (!cache[params.model] && !excludeCacheModels?.includes(params.model)) {
  cache.define(
    params.model,
      {
        references: (args: any, key: string, result: any) => {
          return result ? [`${args.model}~${key}`] : null;
        },
      },
      async (params: MiddlewareParams) => {
        result = await next(params); <-- this is the issue
        console.log("result in async cache function: ", result);

        return result;
      },
    );
  }
}

The console is logging this value.

result in async cache function: { _count: { _all: 2 } }

It does however work if I call the next function outside the cache function.

const result2 = await next(params);
console.log("result2 value: ", result2);

This is what is logged to the terminal.

result2 value: 2

Logging the params value inside and outside the cache function shows that both are using the same shape.

{"action":"aggregate","args":{"select":{"_count":{"select":{"_all":true}}}},"dataPath":[],"model":"User","runInTransaction":false}

rDescribe the bug
When using a transaction of two parts (count and findMany) the result never comes back, if skip parameter is specified and is greater than 0. If skip is 0, the result comes back as expected. If skip is not specified, the result comes back as expected. This makes pagination totally unusable.

To Reproduce
Request a transaction with a count and a findMany. The findMany should have a limit and a skip parameter. The skip parameter should be greater than 0. The result will never come back.
Hears an example function from my application that demonstrates the issue:

export const getTeams = async (
  filter?: PaginationDataFilter
): Promise<PaginationData<Team>> => {
  try {
    const result = await prisma.$transaction([
      prisma.team.count({
        where: {
          ...(((filter?.search) != null) && {
            OR: [
              { name: { contains: filter?.search, mode: 'insensitive' } },
              { description: { contains: filter?.search, mode: 'insensitive' } }
            ]
          })
        }
      }),
      prisma.team.findMany({
        where: {
          ...(((filter?.search) != null) && {
            OR: [
              { name: { contains: filter?.search, mode: 'insensitive' } },
              { description: { contains: filter?.search, mode: 'insensitive' } }
            ]
          })
        },
        ...(((filter?.limit) != null) ? { take: +filter?.limit } : {}),
        ...(((filter?.limit) != null) && ((filter?.page) != null)
          ? {
              take: +filter?.limit,
              skip: +filter?.limit * (+filter?.page - 1)
            }
          : {}),
        ...(((filter?.sort) != null) && { orderBy: { [filter?.sort.name]: filter?.sort.order } }),
        include: { members: true, invites: true, subscriptions: { include: { product: true } }, projects: true }
      })
    ])
    const response = {
      data: result[1],
      meta: {
        current_page: ((filter?.page) != null) ? filter?.page : 1,
        from: ((filter?.limit) != null) && ((filter?.page) != null) ? +filter?.limit * +filter?.page : 1,
        last_page: ((filter?.limit) != null) ? result[0] % +filter?.limit : 1,
        to: ((filter?.limit) != null) ? filter?.limit : result[0],
        total: result[0]
      }
    }
    utilLogger({ meta: { ...meta, function: 'getUsers' }, data: response })
    return response
  } catch (e) {
    utilLogger({ meta: { ...meta, function: 'getTeams' }, error: e })
    return {
      data: [],
      meta: {
        current_page: 1,
        from: 0,
        last_page: 1,
        to: 0,
        total: 0
      }
    }
  }
}
``


export interface PaginationDataFilter {
  page?: number
  limit?: number
  sort?: Sorting
  search?: string
}

Expected behavior
Return a paginated result based on all the filters provided.

Additional context
I suspect the problem may lie with the way this library handles transaction.

I'm using Typescript 5.1 and get an error pointing at invalidationTTL. I used the "as any" to get it to stop for now.

const cacheMiddleware = createPrismaRedisCache({
  models: [{ model: "CodeCampYear", cacheTime: 7200 }], // all models cache except excluded, this just lets you set cache time for each model
  excludeModels: ["MeetupCodeCampYear"],
  storage: {
    type: "redis",
    options: { client: redis, invalidation: { invalidationTTL: 300 } as any },
  },
  cacheTime: 60, // defaults to 1 minute
  onHit: (key: string) => {
    console.log("Hit: ✅", key);
  },
  onMiss: (key: string) => {
    console.log("Miss: ❌", key);
  },
});

Sorry for posting as a bug as I'm sure it's a misunderstanding.

I've got middleware like below.

What's odd is that I can see in RedisInsights that the model "Attendees" cache gets set to the defaultCacheTime and not the attendeesCacheTime as I would expect.

If I leave out cacheTime: defaultCacheTime then I get cache load errors. Not exactly understanding that.

const defaultCacheTime = 60 * 15; // 60 * 15 = 15 minutes

const attendeesCacheTime = 3600 * 24 * 1;  // 3600 * 24 * 1 = 1 days

const cacheMiddleware = createPrismaRedisCache({
  models: [{ model: "Attendees", cacheTime: attendeesCacheTime }],
  excludeModels: ["MeetupCodeCampYear"],
  storage: {
    type: "redis",
    options: { client: redis, invalidation: { referencesTTL: defaultCacheTime } },
  },
  cacheTime: defaultCacheTime,

  onHit: (key: string) =>
    process.env.NODE_ENV !== "production" &&
    process.env.SHOW_CACHE_HITS === "true"
      ? console.log("Hit: ✅  \n", key)
      : null,

  onMiss: (key: string) =>
    process.env.NODE_ENV !== "production" &&
    process.env.SHOW_CACHE_MISSES === "true"
      ? console.log("Miss: ❌  \n", key)
      : null,
});

Describe the bug
onMiss callback is triggered multiple times. I verified if the request is being made multiple times but not. (See the screenshot below, "///////////// FIND MANY" is the log line from the function requesting the db.

To Reproduce
Define a logger on onMiss callback.

Expected behavior
Only one console print.

Screenshots

Desktop:

• OS: MacOS Monterey M1 Pro
• Node : 16.17.0

Describe the bug
for case prisma.$transaction([queryA, queryB]) , if queryA was found in cache, queryB will stuck in transaction.

To Reproduce
Steps to reproduce the behavior:

async paginatedResult(skip: number) {
  const [totalCount, result] = await this.prisma.$transaction([
    this.prisma.article.count(),
    this.prisma.article.findMany({ skip })
  ]);
  return { totalCount, result };
}

1. query with paginatedResult(0)
2. response 200 successfully, and result cached
3. query with paginatedResult(1)
4. request freeze until timeout

Expected behavior
for case prisma.$transaction([queryA, queryB]) , return cached result if both queryA and queryB was found in cache, otherwise fetch from prisma.

Desktop (please complete the following information):

• OS: MacOS
• Browser: postman
• Version: ^4.8.0

The readme claims that cacheTime is in milliseconds, however, that's not what I am experiencing. Upon looking at the code (line 36 of src/index), the cacheTime option maps to the ttl option of async-cache-dedupe which is a value of seconds. I think we need to clarify how this option is used and potentially update the documentation accordingly.

Currently (at least that's what is looks like to me), one can only cache one Prisma model at a time. It would be useful to make this option an array, and if an empty one or undefinedis provided, it could cache all models (disable the check in https://github.com/Asjas/prisma-redis-middleware/blob/main/index.ts#L50).

Describe the bug

The package is not turning the value back the Date when using redis as caching store and Datetime field in prisma, this will cause any validator like zod an error.

However, date value works in memory cache, redis does not

To Reproduce
Steps to reproduce the behavior:

Store Datetime value in prisma and using redis as caching, then validating the prisma data with zod date

Expected behavior
A clear and concise description of what you expected to happen.

Output validation failed

Screenshots

If the database gets updated, the associated cache keys should be invalidated (either by updating or deleting).

Describe the bug
Not able to invalidate keys that previously return null as value.

To Reproduce
Steps to reproduce the behavior:

1. Try to make any findFirst query in Prisma that returns null (start caching)
2. Create a record that matches findFirst from point 1.
3. Invalidation is not picking up previously created key, because references arg was null.
4. Try to make the same findFirst query. Should return record from point 2, but instead of that returns null.

Expected behavior
Invalidation of null values works.

Additional context
The problem is with function:

references: ({ params }: { params: MiddlewareParams }, key: string, result: Result) => {
   return result ? [`${params.model}~${key}`] : null;
},

When the result is null the references returns also null and invalidation function cannot pick up this key and cannot remove it. After changing this part of the code to:

references: ({ params }: { params: MiddlewareParams }, key: string, result: Result) => {
   return [`${params.model}~${key}`];
},

It worked.

Describe the bug
Still cache for Prisma query actions even set property in CreatePrismaRedisCache.models.excludeMethods

To Reproduce

1. Setup options like this

• schema.prisma

model User {

}

prismaClient.$use(
  createPrismaRedisCache({
    models: [
      { model: "User", cacheTime: 60, excludeMethods: ['findFirst']},  // not work as expected
    ]
// ..other options
})

3. See error
   still cache for method findFirst

onHit {"params":{"action":"findFirst","args":{"where":{"id":""}},"dataPath":[],"model":"User","runInTransaction":false}}

Expected behavior
not cache for findFirst method

Screenshots

log in dashboard

Runtime (please complete the following information):

• OS: CentOS Linux release 7.9.2009 (Core)
• Version NodeJs v16.14.1

Additional context

    "@prisma/client": "^3.12.0",
    "prisma-redis-middleware": "3.3.0",
    "prisma": "^3.12.0",