Potential Concurrency Issue Leading to Cache Inconsistency in Multithreaded Environment about django-cacheops

[Suor]

You can try using queryset.cache(lock=True).

[BoHuang-aspecta]

So, does this issue that I mentioned indeed exist? I will try to reproduce the problem and attempt to solve it using lock = True. THX

[Suor]

I don't understand what did this mean "Thread B invalidates the cache but doesn’t actually clear anything".

About whether there issue exists I am not sure, you can try to create a test using before_after library - there is already one test using it.

[BoHuang-aspecta]

I don't understand what did this mean "Thread B invalidates the cache but doesn’t actually clear anything".

About whether there issue exists I am not sure, you can try to create a test using before_after library - there is already one test using it.

Because when Thread B invalidates the cache, Thread A's cache write operation hasn't completed yet. At that point, the cache is actually empty (assuming the initial state is empty).

In summary, could it happen that Thread B's cache invalidation occurs before Thread A writes to the cache, in a concurrent scenario?

I will try to use before_after lib to reproduce this issue. Thanks for your advice.

[BoHuang-aspecta]

I reproduced this issue using a concurrent approach.

I tried to solve it using lock = True, but it didn’t completely resolve the issue. Some requests still couldn’t retrieve data and resulted in errors. However, it doesn’t seem to cause the previous irrecoverable problem.

By using the invalidate_obj approach mentioned above, it seems to have completely resolved the issue, and there are no more errors.

[Suor]

Actually looking a little bit more at problematic issue, it has concurrency issues outside of cache - the code does not have a transaction, doesn't set ._for_write = True for initial .get(). Using .get_or_create() should solve this problem too.