Coder Social home page Coder Social logo

django-cache-machine / django-cache-machine Goto Github PK

View Code? Open in Web Editor NEW
873.0 26.0 160.0 314 KB

Automatic caching and invalidation for Django models through the ORM.

Home Page: https://cache-machine.readthedocs.org/en/latest/

License: BSD 3-Clause "New" or "Revised" License

Python 100.00%

django-cache-machine's Introduction

Cache Machine

Cache Machine provides automatic caching and invalidation for Django models through the ORM.

For full docs, see https://cache-machine.readthedocs.org/en/latest/.

Requirements

Cache Machine currently works with:

  • Django 2.2, 3.0, 3.1, 3.2, and 4.0
  • Python 3.6, 3.7, 3.8, 3.9, and 3.10

The last version to support Python 2.7 and Django 1.11 is django-cache-machine==1.1.0.

Installation

Get it from pypi:

pip install django-cache-machine

Running Tests

Get it from github:

git clone git://github.com/django-cache-machine/django-cache-machine.git
cd django-cache-machine
pip install -r dev-requirements.txt
python run_tests.py

django-cache-machine's People

Contributors

afani97 avatar bradjasper avatar colons avatar ebrelsford avatar jasonthomas avatar jbalogh avatar jezdez avatar jnormore avatar oremj avatar remohammadi avatar rlr avatar robhudson avatar timdawborn avatar timgates42 avatar tobiasmcnulty avatar twidi avatar vkurup avatar wetneb avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

django-cache-machine's Issues

Caching Consistency if DB is used by few Django apps

Hello. I have 3 django apps sharing the same database.

Will they all share the same consistent cache?

If an item is modified by app1, will the other apps still get a consistent cache?

Note that the other apps will just import the models from app1 (including of course Cache Machine object managers), so I would like to know if the behavior will be as expected.

Thanks

Manual cache method not working

I'm having issues with calling the manual cached function.

from caching.base import cached

def test(x):
    return x+1

print cached(test(1),'test',60)

I then receive this error

    262     if val is None:
    263         log.debug('cache miss for %s' % key)
--> 264         val = function()
    265         cache.set(key, val, duration)
    266     else:

TypeError: 'int' object is not callable

Disable cache logging

Hello,

Is there a way to disable this logging?

I get A LOT of lines like this in my terminal console.

DEBUG:caching:cache hit: SELECT "profiles_profile"."id", "profiles_profile"."mugshot", "profiles_profile"."privacy", "profiles_profile"."language", "profiles_profile"."gender", "profiles_profile"."user_id", "profiles_profile"."website", "profiles_profile"."location", "profiles_profile"."birth_date", "profiles_profile"."about_me", "profiles_profile"."receive_newsletters", "profiles_profile"."female_newsletters", "profiles_profile"."male_newsletters" FROM "profiles_profile" WHERE "profiles_profile"."user_id" = 40

There should be a setting to configure this behavior.

No invalidation on ManyToManyField

class UserProfile(caching.base.CachingMixin, TimeStampedModel):
    groups = models.ManyToManyField('containers.ContainerGroup', verbose_name=_(u'Groups'), blank=True,)
    objects = caching.base.CachingManager()

class ContainerGroup(caching.base.CachingMixin, models.Model):
    container = models.ForeignKey('containers.Container')
    name = models.CharField(max_length=100)
    objects = caching.base.CachingManager()

>>> from django.core.cache import cache
>>> cache.clear()
>>> profile = UserProfile.objects.get(user='246')
>>> container_group = ContainerGroup(container=Container('Test','test'))
>>> profile.groups.all() # => cache
[]
>>> profile.groups.add(container_group) # => no invalidation
>>> profile.groups.all()
[]
>>> cache.clear()
>>> profile.groups.all() # group visible
[<ContainerGroup: Administrator>]

UnicodeDecodeError in MethodWrapper.__call__

The key conversion here doesn't look like it is handling non-ascii characters correctly. It can probably be changed to o.encode('utf8', 'replace') if isinstance(o, unicode) and that will handle 98% of the non-ascii cases.

Traceback (most recent call last):

 File "/data/www/addons.mozilla.org/zamboni/vendor/src/django/django/core/handlers/base.py", line 111, in get_response
   response = callback(request, *callback_args, **callback_kwargs)

 File "/data/www/addons.mozilla.org/zamboni/apps/addons/decorators.py", line 29, in wrapper
   return f(request, addon, *args, **kw)

 File "/data/www/addons.mozilla.org/zamboni/vendor/src/django-mobility/mobility/decorators.py", line 23, in wrapper
   return f(request, *args, **kw)

 File "/data/www/addons.mozilla.org/zamboni/apps/amo/decorators.py", line 29, in wrapper
   return func(request, *args, **kw)

 File "/data/www/addons.mozilla.org/zamboni/apps/addons/decorators.py", line 67, in wrapper
   return f(request, addon, *args, **kw)

 File "/data/www/addons.mozilla.org/zamboni/apps/reviews/views.py", line 174, in add
   if addon.has_author(request.user):

 File "/data/www/addons.mozilla.org/zamboni/vendor/src/django-cache-machine/caching/base.py", line 330, in __call__
   arg_keys, kwarg_keys)

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 9: ordinal not in range(128)

Automatic Invalidation not happening in Django 1.7

I have a very simple caching model which is inherits from CachingMixin. When I call save() in Django 1.7 (to add a new instance), the cache is not invalidated (the new object does not show up in subsequent queries.)

When I switch back to Django 1.6, the same code works perfectly. Pulling the CachingManager and using 1.7 also works.

I am on trunk.

problem with locale and invalidation

If i update on object in a current locale, say "fr", all keys for locale "fr" for this object will be invalidated, but not if after that a user in an other locale "en" check the object (common case in multilangual site), so i think you souldn't use the locale, or invalidate for all locales (which is more work, so i thing no locale is the best solution)

FYI, i'll fork the project in order to remove the locale

FETCH_BY_ID is undocumented

@jbalogh, the FETCH_BY_ID setting doesn't appear in the documentation.

I'd be willing to submit a documentation patch if you wouldn't mind giving me a little steering on it. In a previous PR I wrote a test for it and it looks like FETCH_BY_ID is designed to get partially-iterated querysets out of cache first (using multi_get) and then from the DB. This looks to me like it should reduce the number of required cache hits considerably. Does this sound accurate to you?

failed to invalidate query cache when add new record to the table

repo steps:

  1. iterator all the objects in the model and fill the cache
    for m in MyModel.objects.all(): foo(m)
  2. save a new record in this model
    MyModel().save()
  3. the cache for the step 1 should be invalided, but it is not. And the iteration result is the same as the step1

I'm using the latest git version and the model decleartion looks like this:

class MyModel(caching.base.CachingMixin, models.Model):
#.....
#....
objects = caching.base.CachingManager()

Adding an object did not invalidate queries

I had a strange issue on my site and i made some test and it seems that when adding an object of a model, invalidation is not done for queries like all()
Of course the all query (or other) is invalidated when an object is updated.

Exemple:

from django.db import models
import caching.base
class MyTest(caching.base.CachingMixin, models.Model):
  vartest = models.IntegerFields()
  objects = caching.base.CachingManager()
  def __repr__:
    return '<MyTest(id=%d, var=%d)>' % (self.id, self.vartest)

Then in a python/django shell:

MyTest.objects.create(vartest=1)
=> <MyTest(id=1, var=1)>
MyTest.objects.all()
=> [<MyTest(id=1, var=1)>]
MyTest.objects.create(vartest=2)
=> <MyTest(id=2, var=2)>
MyTest.objects.all()
=> [<MyTest(id=1, var=1)>]

t=MyTest.objects.get(id=1)
t.vartest=3
t.save()
t
=> <MyTest(id=1, var=3)>
MyTest.objects.all()
=> [<MyTest(id=1, var=3)>, <MyTest(id=2, var=2)>]

I understand why this happens but not really how to correct it

PS : i use the latest version, but tried too without our recent "if not to_cache"

Caching empty querysets breaks get_or_create

When empty queryset caching is enabled, get_or_create seems to always force creation causing multiple calls to produce IntegrityErrors for existing primary keys. I'll see what I can do about writing a test to prove this, but this is what I'm seeing in my app's tests.

my bad issue

sorry, just forgot to set mixin as parent class

support django 1.3 logging

django 1.3 already have a logging module, can not set the appropriate level of module.

please review the patch:

diff -r 0c212a1ee276 server/caching/base.py
--- a/server/caching/base.py    Thu May 05 15:04:58 2011 +0800
+++ b/server/caching/base.py    Fri May 06 16:26:58 2011 +0800
@@ -10,16 +10,7 @@
 
 from .invalidation import invalidator, flush_key, make_key, byid
 
-
-class NullHandler(logging.Handler):
-
-    def emit(self, record):
-        pass
-
-
 log = logging.getLogger('caching')
-log.setLevel(logging.INFO)
-log.addHandler(NullHandler())
 
 FOREVER = 0
 NO_CACHE = -1

Count on queryset with empty list fails

For example in amo:

Addon.objects.filter(id__in=[])
[]

User.objects.filter(id__in=[]).count()
0

Addon.objects.filter(id__in=[]).count()
[...snip]

/Users/andy/sandboxes/zamboni/vendor/src/django/django/db/models/sql/where.pyc in make_atom(self, child, qn, connection)
177 if lookup_type == 'in':
178 if not value_annot:
--> 179 raise EmptyResultSet
180 if extra:
181 return ('%s IN %s' % (field_sql, extra), params)

EmptyResultSet:

Doing a where lookup on an empty list means that as_sql throws an error. Presumably because doing an IN () is invalid SQL. Because as_sql is used by cache machine to figure out the query_key, the counts fail.

Not sure what the right answer here is, since counts with and EmptyResultSet are always zero, that can be special cased in cache machine.

problem in admin

Example code
import caching.base

class NewsItem(caching.base.CachingMixin, IblockItem):

class Meta:
    verbose_name = u'Новость'
    verbose_name_plural = u'Новости'

preview_text = models.TextField(verbose_name=u'Краткое описание')

objects = caching.base.CachingManager()

in admin this list object cached
When if i add object and press save and continue edit and pres link in in pathway in list no new item

@caching.cached_method returns inconsistent results

The caching.cached_method is returning inconsistent results. Given a method that returns True or False we get:

result of app.has_author(): True
result of app.has_author(): True
result of app.has_author(): 1

Not sure why yet.

Make it easy to invalidate auth.User (and other 3rd-party models)

This looks interesting and is very similar to what I've been working on. I could be missing something, but from what I can tell queries on related tables won't get invalidated. For example:

profile = Profile.objects.filter(user__username="test")

should get invalidated when either profile or user are saved/deleted, but I think this would only invalidate when profile is saved/deleted.

Also, the post_save and post_delete signals won't get fired on queryset.update, so if I were to do something like Profile.objects.filter(user__username="test").update(username="something else")

then Profile.objects.filter(user__username="test") will return a stale object. You could override the queryset's update method, but you'd still have the same problem as above with related models.

Non empty queryset cast to list results in empty list

This happens on some instances of v. Works fine for others.

In [2]: v.browse.all()
Out[2]: [<Browse: Prosthodontics>, <Browse: Restorative Dentistry>]

In [3]: list(v.browse.all())
Out[3]: []

In [4]: list(v.browse.all().no_cache())
Out[4]: [<Browse: Prosthodontics>, <Browse: Restorative Dentistry>]

Another example for different instance that works fine:

In [14]: list(v2.browse.all())
Out[14]:
[<Browse: Proximal Humerus Reverse Shoulder>,
 <Browse: Reverse Shoulder Arthroplasty>]

Another problem in admin

See my manager code

class PrumaCatalogItemManager(CachingManager):

    def get_query_set(self):
        return CachingQuerySet(self.model).filter(price__gt=0)

I filter all catalog items on frontend by price
Its work, but in admin items also filtered and i cant got item with price = 0

Help me plz

Support for multiple configuration

Currently only one backing cache can be setup, for all models relying on Django-Cache-Machine.
In a complex application, one would want to choose different cache setup for different models.
E.g.:
model Category (low cardinality, never deleted)
model Article (high cardinality, full CRUD)

For Category one would use LocalMem, with 1 day lease, for Article Memcached, with 5 minutes...

usage with django.test.TestCase rollback

After adding django-cache-machine to my application, I ran my tests with the LocMemCache backend. I got some test failures, objects having foreign key references that weren't supported by corresponding rows in the database table. I haven't written a minimal reproduction yet, but I strongly suspect this is what's happening:

  • test 1 calls Foo.objects.get_or_create(name="foo"), creating "foo" in database and cache.
  • test 1 completes, django.test.TestCase does a rollback of the database state, removing "foo" from database.
  • test 2 calls Foo.objects.get_or_create(name="foo"), pulls "foo" from cache, though it does not exist in database.
  • failure follows.

My tests do pass with the DummyCache backend. And they pass if I put a get_cache('default').clear() in the test's setUp method.

(Django 1.6.2, d-c-m commit 449861a)

At the very least this could use some documentation. I think it could be sensible to have a TestCase base class that clears the django-cache-machine backend on TestCase.setUp(). Open to other ideas about how to best handle this.

Invalidating empty related manager queryset

There seems to be a bug invalidating the empty querysets returned by related managers:

>>> from testapp.models import User, Addon
>>> 
>>> author_with_no_addons = User.objects.create()
>>> author_with_no_addons.addon_set.all()
[]
>>> addon = Addon.objects.create(val=42, author1=author_with_no_addons, author2=author_with_no_addons)
>>> author_with_no_addons.addon_set.all()
[]
>>> author_with_no_addons.addon_set.count()
1
>>> author_with_no_addons.author2_set.all()
[<Addon: Addon object>]

It works as expected when the User has an Addon for the first query:

>>> author_with_addons = User.objects.get(pk=2)
>>> author_with_addons.addon_set.all()
[<Addon: Addon object>, <Addon: Addon object>]
>>> addon = Addon.objects.create(val=42, author1=author_with_addons, author2=author_with_addons)
>>> author_with_addons.addon_set.all()
[<Addon: Addon object>, <Addon: Addon object>, <Addon: Addon object>]

Python 2.4 compatibility

Given that Django is still supporting Python 2.4, I think it should be important for related tools to hold the same support. I'm putting this application into production on a CentOS 5.5 machine, so adding this compatibility become a necessity for me. It wasn't even a difficult task, luckily! Here are the changes necessary:

diff --git a/caching/base.py b/caching/base.py
index 44f5fed..416025e 100644
--- a/caching/base.py
+++ b/caching/base.py
@@ -1,6 +1,4 @@
-import collections
-import functools
-import hashlib
+from compat import *
 import logging

 from django.conf import settings
@@ -147,7 +145,7 @@ class CacheMachine(object):
         flush_keys = [o.flush_key() for o in objects]
         query_flush = flush_key(self.query_string)

-        flush_lists = collections.defaultdict(list)
+        flush_lists = defaultdict(list)
         for key in flush_keys:
             flush_lists[key].extend([query_key, query_flush])
         flush_lists[query_flush].append(query_key)
@@ -252,13 +250,13 @@ class CachingRawQuerySet(models.query.RawQuerySet):

 def flush_key(obj):
     """We put flush lists in the flush: namespace."""
-    key = obj if isinstance(obj, basestring) else obj.cache_key
+    key = isinstance(obj, basestring) and obj or obj.cache_key
     return FLUSH + make_key(key, with_locale=False)


 def add_to_flush_list(mapping):
     """Update flush lists with the {flush_key: [query_key,...]} map."""
-    flush_lists = collections.defaultdict(set)
+    flush_lists = defaultdict(set)
     flush_lists.update(cache.get_many(mapping.keys()))
     for key, list_ in mapping.items():
         if flush_lists[key] is None:
@@ -276,7 +274,7 @@ def make_key(k, with_locale=True):
     # memcached keys must be < 250 bytes and w/o whitespace, but it's nice
     # to see the keys when using locmem.
     if 'memcached' in cache.scheme:
-        return hashlib.md5(encoding.smart_str(key)).hexdigest()
+        return md5_constructor(encoding.smart_str(key)).hexdigest()
     else:
         return key

@@ -301,8 +299,7 @@ def cached(function, key_, duration=None):
 def cached_with(obj, f, f_key, timeout=None):
     """Helper for caching a function call within an object's flush list."""
     try:
-        obj_key = (obj.query_key() if hasattr(obj, 'query_key')
-                   else obj.cache_key)
+        obj_key = hasattr(obj, 'query_key') and obj.query_key() or obj.cache_key
     except AttributeError:
         log.warning(u'%r cannot be cached.' % obj)
         return f()
@@ -352,7 +349,7 @@ class MethodWrapper(object):
         self.cache = {}

     def __call__(self, *args, **kwargs):
-        k = lambda o: o.cache_key if hasattr(o, 'cache_key') else o
+        k = lambda o: hasattr(o, 'cache_key') and o.cache_key or o
         arg_keys = map(k, args)
         kwarg_keys = [(key, k(val)) for key, val in kwargs.items()]
         key = 'm:%s:%s:%s:%s' % (self.obj.cache_key, self.func.__name__,

This also adds a new file, compat.py:

try:
    import functools
except ImportError:
    import django.utils.functional as functools

# GPL licensed code taken from NLTK - http://code.google.com/p/nltk/
# collections.defaultdict
# originally contributed by Yoav Goldberg <[email protected]>
# new version by Jason Kirtland from Python cookbook.
# <http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/523034>
try:
    from collections import defaultdict
except ImportError:
    class defaultdict(dict):
        def __init__(self, default_factory=None, *a, **kw):
            if (default_factory is not None and
                not hasattr(default_factory, '__call__')):
                raise TypeError('first argument must be callable')
            dict.__init__(self, *a, **kw)
            self.default_factory = default_factory
        def __getitem__(self, key):
            try:
                return dict.__getitem__(self, key)
            except KeyError:
                return self.__missing__(key)
        def __missing__(self, key):
            if self.default_factory is None:
                raise KeyError(key)
            self[key] = value = self.default_factory()
            return value
        def __reduce__(self):
            if self.default_factory is None:
                args = tuple()
            else:
                args = self.default_factory,
            return type(self), args, None, None, self.iteritems()
        def copy(self):
            return self.__copy__()
        def __copy__(self):
            return type(self)(self.default_factory, self)
        def __deepcopy__(self, memo):
            import copy
            return type(self)(self.default_factory,
                              copy.deepcopy(self.items()))
        def __repr__(self):
            return 'defaultdict(%s, %s)' % (self.default_factory,
                                            dict.__repr__(self))

    # [XX] to make pickle happy in python 2.4:
    import collections
    collections.defaultdict = defaultdict

from django.utils.hashcompat import md5_constructor

__all__ = ['functools','defaultdict','md5_constructor']

Problem with deprecated memcached.CacheClass.

Django==1.6.5
django-cache-machine==0.8

Use django-cache-machine witch caching.backends.memcached.MemcachedCache I got:
AttributeError: 'module' object has no attribute 'CacheClass'

and found in Django 1.5:
class CacheClass(BaseMemcachedCache):
def init(self, server, params):
import warnings
warnings.warn(
"memcached.CacheClass has been split into memcached.MemcachedCache and memcached.PyLibMCCache. Please update your cache backend setting.",
DeprecationWarning
)
...

and

if django.VERSION[:2] >= (1, 3):

class MemcachedCache(InfinityMixin, memcached.MemcachedCache):
    pass

class PyLibMCCache(InfinityMixin, memcached.PyLibMCCache):
    pass

don't work properly. On pip is old version.

Caught UnicodeEncodeError while rendering: 'ascii' codec can't encode character u'\u6211' in position 274: ordinal not in range(128)

Environment:

Request Method: POST
Request URL: http://127.0.0.1:8000/service_area/list/1/
Django Version: 1.2.3
Python Version: 2.6.6
Installed Applications:
['django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.sites',
'django.contrib.messages',
'sentry',
'sentry.client',
'paging',
'indexer',
'sentry.plugins.sentry_servers',
'sentry.plugins.sentry_sites',
'sentry.plugins.sentry_urls',
'hchq.untils',
'hchq.account',
'hchq.service_area',
'hchq.department',
'hchq.check_project']
Installed Middleware:
('django.middleware.common.CommonMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware')

Template error:
In template e:\djcode\hchq\templates\service_area\service_area_table_list.html, error at line 19
Caught UnicodeEncodeError while rendering: 'ascii' codec can't encode character u'\u6211' in position 274: ordinal not in range(128)
9 : 编号

10 : 名称

11 : 创建者

12 : 创建时间

13 : 更新时间

14 : 查询关联单位

15 : 添加关联单位

16 : 删除关联单位

17 :

18 :

19 : {% for result in results_page.object_list %}

20 :

21 : {{ result.id }}

22 : {{ result.name }}

23 : {{ result.creater }}

24 : {{ result.created_at|date:"Y-m-d H:i:s"}}

25 : {{ result.updated_at|date:"Y-m-d H:i:s"}}

26 :

27 : 查询关联单位

28 :

29 :

Traceback:
File "C:\Python26\lib\site-packages\django\core\handlers\base.py" in get_response

  1.                 response = callback(request, _callback_args, *_callback_kwargs)
    
    File "C:\Python26\lib\site-packages\django\utils\decorators.py" in _wrapped_view
  2.                 response = view_func(request, _args, *_kwargs)
    
    File "C:\Python26\lib\site-packages\django\contrib\auth\decorators.py" in _wrapped_view
  3.             return view_func(request, _args, *_kwargs)
    
    File "E:\djcode\hchq..\hchq\service_area\views.py" in service_area_list
  4.                               context_instance=RequestContext(request))
    
    File "C:\Python26\lib\site-packages\django\shortcuts__init__.py" in render_to_response
  5. return HttpResponse(loader.render_to_string(_args, *_kwargs), **httpresponse_kwargs)
    
    File "C:\Python26\lib\site-packages\django\template\loader.py" in render_to_string
  6. return t.render(context_instance)
    
    File "C:\Python26\lib\site-packages\django\template__init__.py" in render
  7.         return self._render(context)
    
    File "C:\Python26\lib\site-packages\django\template__init__.py" in _render
  8.     return self.nodelist.render(context)
    
    File "C:\Python26\lib\site-packages\django\template__init__.py" in render
  9.             bits.append(self.render_node(node, context))
    
    File "C:\Python26\lib\site-packages\django\template\debug.py" in render_node
  10.         result = node.render(context)
    
    File "C:\Python26\lib\site-packages\django\template\loader_tags.py" in render
  11.     return compiled_parent._render(context)
    
    File "C:\Python26\lib\site-packages\django\template__init__.py" in _render
  12.     return self.nodelist.render(context)
    
    File "C:\Python26\lib\site-packages\django\template__init__.py" in render
  13.             bits.append(self.render_node(node, context))
    
    File "C:\Python26\lib\site-packages\django\template\debug.py" in render_node
  14.         result = node.render(context)
    
    File "C:\Python26\lib\site-packages\django\template\loader_tags.py" in render
  15.     return compiled_parent._render(context)
    
    File "C:\Python26\lib\site-packages\django\template__init__.py" in _render
  16.     return self.nodelist.render(context)
    
    File "C:\Python26\lib\site-packages\django\template__init__.py" in render
  17.             bits.append(self.render_node(node, context))
    
    File "C:\Python26\lib\site-packages\django\template\debug.py" in render_node
  18.         result = node.render(context)
    
    File "C:\Python26\lib\site-packages\django\template\loader_tags.py" in render
  19.     return compiled_parent._render(context)
    
    File "C:\Python26\lib\site-packages\django\template__init__.py" in _render
  20.     return self.nodelist.render(context)
    
    File "C:\Python26\lib\site-packages\django\template__init__.py" in render
  21.             bits.append(self.render_node(node, context))
    
    File "C:\Python26\lib\site-packages\django\template\debug.py" in render_node
  22.         result = node.render(context)
    
    File "C:\Python26\lib\site-packages\django\template\loader_tags.py" in render
  23.         result = block.nodelist.render(context)
    
    File "C:\Python26\lib\site-packages\django\template__init__.py" in render
  24.             bits.append(self.render_node(node, context))
    
    File "C:\Python26\lib\site-packages\django\template\debug.py" in render_node
  25.         result = node.render(context)
    
    File "C:\Python26\lib\site-packages\django\template\loader_tags.py" in render
  26.         result = block.nodelist.render(context)
    
    File "C:\Python26\lib\site-packages\django\template__init__.py" in render
  27.             bits.append(self.render_node(node, context))
    
    File "C:\Python26\lib\site-packages\django\template\debug.py" in render_node
  28.         result = node.render(context)
    
    File "C:\Python26\lib\site-packages\django\template\loader_tags.py" in render
  29.         return self.template.render(context)
    
    File "C:\Python26\lib\site-packages\django\template__init__.py" in render
  30.         return self._render(context)
    
    File "C:\Python26\lib\site-packages\django\template__init__.py" in _render
  31.     return self.nodelist.render(context)
    
    File "C:\Python26\lib\site-packages\django\template__init__.py" in render
  32.             bits.append(self.render_node(node, context))
    
    File "C:\Python26\lib\site-packages\django\template\debug.py" in render_node
  33.         result = node.render(context)
    
    File "C:\Python26\lib\site-packages\django\template\defaulttags.py" in render
  34.     len_values = len(values)
    
    File "C:\Python26\lib\site-packages\django\db\models\query.py" in len
  35.             self._result_cache = list(self.iterator())
    
    File "C:\Python26\lib\site-packages\django_cache_machine-0.3-py2.6.egg\caching\base.py" in iterator
  36.     for obj in CacheMachine(query_string, iterator):
    
    File "C:\Python26\lib\site-packages\django_cache_machine-0.3-py2.6.egg\caching\base.py" in iter
  37.         query_key = self.query_key()
    
    File "C:\Python26\lib\site-packages\django_cache_machine-0.3-py2.6.egg\caching\base.py" in query_key
  38.     return make_key('qs:%s' % self.query_string)
    
    File "C:\Python26\lib\site-packages\django_cache_machine-0.3-py2.6.egg\caching\base.py" in make_key
  39.     return hashlib.md5(key).hexdigest()
    

Exception Type: TemplateSyntaxError at /service_area/list/1/
Exception Value: Caught UnicodeEncodeError while rendering: 'ascii' codec can't encode character u'\u6211' in position 274: ordinal not in range(128)

Exception: set() got an unexpected keyword argument 'version'

Hi,

I get the following exception when running django-cache-machine with Django 1.3:

TypeError at /items/
set() got an unexpected keyword argument 'version'
Request Method: GET
Request URL:    http://local.server.com:8000/items/
Django Version: 1.3
Exception Type: TypeError
Exception Value:    
set() got an unexpected keyword argument 'version'
Exception Location: /data/sw/python/darwin64/lib/python2.6/site-packages/django/core/cache/backends/base.py in set_many, line 173
Python Executable:  /data/sw/python/darwin64/bin/python
Python Version: 2.6.6

My settings:

CACHE_PREFIX = 'cm:'

Thanks for help.

Make log behavior configurable

It would be great if we could configure the log behavior in settings.py (in particular whether to log messages at all, and possibly the formatting).

Models are missing dynamic attributes (e.g. django-thumbs) if fetched from cache

This only happens under certain circumstances:

  • only on collections (i.e. filter) not single objects
  • only if the collection is fetched via iterator, i.e. for entry in queryset:
  • it never fails if the query is evaluated at once i.e. list(queryset)
  • the collection must have at least 100 objects for this error to occur

I'm using the fork of theatlantic with redis, but I've verified that this problem also exists in jbalogh/master with memcached.

Given the following model which has a ImageWithThumbsField (provided by django-thumbs)

class Debug(caching.base.CachingMixin, models.Model):
    text = models.CharField(max_length=512, blank=True, null=True)
    img = ImageWithThumbsField(
        verbose_name='image',
        upload_to='uploads/projects/',
        styles=({'w': 100, 'h': 100},),
        thumb_method=generate_thumb_square_top,
        blank=True, null=True
    )
    objects = caching.base.CachingManager()

and the following debug management command:

from django.core.management.base import NoArgsCommand
class Command(NoArgsCommand):
    def __init__(self):
        pass

    def handle_noargs(self, **options):
        from django.core.cache import cache
        from atizo.apps.debug.models import Debug

        def _add_objects(count):
            print "add %d" % count
            n = 0
            while n < count:
                Debug().save()
                n += 1

        def _fetch_via_list():
            print "via list()"
            cache.clear()
            # populate cache
            query = list(Debug.objects.all())
            for entry in query: img = entry.img.url_100x100
            # fetch from cache
            query = list(Debug.objects.all())
            for entry in query: img = entry.img.url_100x100

        def _fetch_via_iterator():
            print "via iterator"
            cache.clear()
            # populate cache
            query = Debug.objects.all()
            for entry in query: img = entry.img.url_100x100
            # fetch from cache
            query = Debug.objects.all()
            try:
                for entry in query: img = entry.img.url_100x100 # <- will fail if there are more than 100 obj's
            except:
                print "fail!"


        Debug.objects.all().delete()
        cache.clear()

        _add_objects(1)
        _fetch_via_list() # success
        _fetch_via_iterator() # success

        _add_objects(98) # total 99
        _fetch_via_list() # success
        _fetch_via_iterator() # success

        _add_objects(1) # total 100
        _fetch_via_list() # success
        _fetch_via_iterator() # fail

        _add_objects(270) # total 370
        _fetch_via_list() # success
        _fetch_via_iterator() # fail

django.db.model.query has a constant CHUNK_SIZE = 100. If changed to 200, the problem will only appear with collections of 200 objects or bigger.

Also there is a difference in the pickled cache entry:

cache entry that was populated with query = list(Debug.objects.all()):
http://pastie.org/3446872

cache entry that was populated with for entry in query: img = entry.img.url_100x100:
http://pastie.org/3446860

work with Redis problem

I setup it with redis,
At my project launch up, it work fine. but it working about 1~2 hours later,
I push a data and insert into database, i found the redis monitor which print del cache command, but i access my model api by "get method api", it alwasy the cache data there are no new record existed.

Cache keys choke on non-ascii characters.

Just saw a collection of stack traces that look like this:

File "/data/virtualenvs/kitsune/src/django-cache-machine/caching/base.py", line 105, in __iter__
  query_key = self.query_key()

File "/data/virtualenvs/kitsune/src/django-cache-machine/caching/base.py", line 101, in query_key
  return make_key('qs:%s' % self.query_string)

File "/data/virtualenvs/kitsune/src/django-cache-machine/caching/base.py", line 271, in make_key
  return hashlib.md5(key).hexdigest()

UnicodeEncodeError: 'ascii' codec can't encode characters in position 676-684: ordinal not in range(128)

Python 3.3 support?

Thanks for the awesome open source library -I've been using it with python 2.7.x for some time and finally decided to look at the python 3.3 upgrade path.

Is python 3.x support something on the list for 2014? Thanks again for the open source library!

Invalidation of Foreign Keys

Line 244 of base.py specifically finds foreign keys and invalidates those as well.

def _cache_keys(self):
    """Return the cache key for self plus all related foreign keys."""
    fks = dict((f, getattr(self, f.attname)) for f in self._meta.fields
                if isinstance(f, models.ForeignKey))

    keys = [fk.rel.to._cache_key(val, self._state.db) for fk, val in fks.items()
            if val is not None and hasattr(fk.rel.to, '_cache_key')]
    return (self.cache_key,) + tuple(keys)

What is the reasoning behind including foreign keys in the invalidation. In our experience, the foreign keys are rarely if ever updated when the related object is updated and so invalidation of the related object seems unnecessary. I am curious, however, if there is a reason that I am not considering.

Thanks.

underlying model changes cause exceptions

if you have a cached object, then change the underlying model by adding a field, then load that object from the cache, you can get exceptions because the objects dict will not contain an entry for that new field.

i know this is probably not a safe thing to do anyways, probably best to just flush the cache.

but i think if CacheMixin implemented setstate you could watch for this state by iterating through the model's fields. if that field didn't appear in the new state, then use the field's default value.

i'm gonna try implementing this, but i figured i'd post here in case someone else had thoughts on this.

pypi outdated

Looks like pypi is at version 0.6, which doesn't have the classes needed for the Django 1.3 syntax described in the docs, e.g. caching.backends.locmem.LocMemCache

Development server hangs

Hi all,

I do have a very strange issue:

  1. I Have added the CachingMixin to my model ==> everything works fine
  2. I have added the objects = CachingManager() to my model ==> all models validat, development server starts at the command line as expected. But it will not serve any browser requests (just hanging). Strange is that I can open the shell and and retrieve data through the model.

Any help is highly appreciated. I am not very confident to move this to production (event if it looks like it is working), as long I have not understood this issue. I also have no clue how to further debug.

Regards,
Juergen

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.