mailgun / expiringdict Goto Github PK

View Code? Open in Web Editor NEW

336.0 336.0 75.0 33 KB

Dictionary with auto-expiring values for caching purposes.

License: Apache License 2.0

Python 100.00%

expiringdict's People

Contributors

Stargazers

Watchers

Forkers

pmclanahan defcube svisser avleen lunaticneko j105rob ale26 bayangroup gklots gwillem llchen223 barbuza rubensayshi alex-dr pombredanne samdfonseca kormat containerz benman1 chenjialin eplaut steverweber folowing machunleilei stephen-mw udayan12167 cngo-github byron70 syrusakbary os roboticsandcloud nihalnayak klaviyo sacherjj juharris mujige77 ar45 frobcode weaming sundarnagarajan timurshuk chanchiem matteougolotti kyriony dmariash fagan2888 travisdieckmann nicktimko jonnyyu jibhat sthagen chzeze kevinkevin1987 nixiaocang sahilx13 killermank asellappen wrktech kdsbami manjunathb4461 neulchance c0c1 s-t-e-v-e-n-k shunchizhou codedevninja goodeejay thinker007 shenrun davidt99

expiringdict's Issues

Upload new version to PyPI

Is it possible to upload the current version to PyPI?
There is the "ImportError: No module named 'md5'" error on the PyPI release what breaks installation for py3. ATM i'm porting talon to py3. talon is based on flanker, flank on expiringdict - so it's very hard to build an working test setup for talon and py3.
Thank thank you in advance!

Version available in Pip is outdated

Hi,
The version of expiringdict available in Pip does not include the lastest commit (Fix for Python3). Would it be possible to have the pip versions updated?

Regards,
Alexander

Is this project abandoned?

@mailgun There are a few outstanding issues and PRs in the tracker that have gone unanswered, and Python3 support is just waiting for an official release on PyPI.

I'm volunteering to fix issues, test and merge code, cut releases etc. if you need help.

calling 'keys' method doesn't clean up expirations at first.

example:

from expiringdict import ExpiringDict
from time import sleep

expirationtest = ExpiringDict(max_len=100, max_age_seconds=5)
expirationtest["testing123"]="somedata"
print expirationtest.keys()
sleep(6)
print expirationtest.keys()

above example will print 'testing123' twice.

i am using python 2.7.10 and freshly installed expiringdict (via pip)

Add support for Python 3.x

If I'm not mistaken the following steps are sufficient to have this package publicly support Python 3.x:

Add PyPI classifiers in setup.py for the setup() call (such as https://github.com/svisser/logaugment/blob/master/setup.py#L31-L37) to indicate 3.x support.
Package a new release to ensure that the above change plus this change are included.
Upload to PyPI.

I'm not sure what Python versions you're willing to support but these changes should be all (I have tested the package already in 3.x).

Add an option to reset remove timer on access

For exemple:

from expiringdict import ExpiringDict
import time
cache = ExpiringDict(max_len=10, max_age_seconds=10, refresh_on_access=True)
cache["foo"] = "bar"
time.sleep(7)
print(cache["foo"]) #Value is accessed, timer is reset for this value
time.sleep(7)
print(cache["foo"]) #Value is still there

Alternatively, a function to reset the timer

from expiringdict import ExpiringDict, ResetTimer
import time
cache = ExpiringDict(max_len=10, max_age_seconds=10)
cache["foo"] = "bar"
time.sleep(7)
ResetTimer(cache, "foo") #Timer is reset for this value
time.sleep(7)
print(cache["foo"]) #Value is still there

Key expiration does not work with Python 3.5

I guess the inherited OrderedDict implementation has changed.

I do not use ttl or iteration, so I have fixed my use case with an extra try around the popitem(last=False) in __setitem__.

======================================================================
ERROR: tests.expiringdict_test.test_basics
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/willem/.virtualenvs/expiringdict/lib/python3.5/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/willem/git/expiringdict/tests/expiringdict_test.py", line 37, in test_basics
    d['e'] = 'z'
  File "/home/willem/git/expiringdict/expiringdict/__init__.py", line 72, in __setitem__
    self.popitem(last=False)
KeyError: 'a'

======================================================================
ERROR: tests.expiringdict_test.test_iter
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/willem/.virtualenvs/expiringdict/lib/python3.5/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/willem/git/expiringdict/tests/expiringdict_test.py", line 78, in test_iter
    eq_([k for k in d.values()], [])
  File "/home/willem/git/expiringdict/expiringdict/__init__.py", line 124, in values
    for key in self:
RuntimeError: OrderedDict mutated during iteration

======================================================================
ERROR: tests.expiringdict_test.test_ttl
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/willem/.virtualenvs/expiringdict/lib/python3.5/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/willem/git/expiringdict/tests/expiringdict_test.py", line 101, in test_ttl
    Mock(return_value=('x', 10**9))):
  File "/home/willem/.virtualenvs/expiringdict/lib/python3.5/site-packages/mock/mock.py", line 1460, in __enter__
    setattr(self.target, self.attribute, new_attr)
TypeError: can't set attributes of built-in/extension type 'collections.OrderedDict'

----------------------------------------------------------------------

Upload v1.1.4 to PyPi

The latest version in this repo is 1.1.4 with Python 3 support. PyPi only has 1.1.3. Anything you need to get this published out to PyPi?

ExpiringDict objects cant be pickled by Dill - but OrderedDict can be pickled !!!

In order to use that objects together with wrapt_timeout_decorator
https://github.com/bitranox/wrapt_timeout_decorator

it needs to be pickable - but it is not.
Ordered Dict can be pickled.

Test Code :

from expiringdict import ExpiringDict
import dill

try:
    from collections import OrderedDict
except ImportError:
    # Python < 2.7
    from ordereddict import OrderedDict


def test_pickle_ordered_dict():
    """
    >>> # test to pickle ordered dict
    >>> dict_test=OrderedDict()
    >>> dict_test['test'] = 1
    >>> pickled_object = dill.dumps(dict_test)
    >>> original_object = dill.loads(pickled_object)
    >>> original_object['test']
    1
    """
    pass


def test_pickle_expiring_dict():
    """
    >>> # test to pickle ordered dict
    >>> dict_test=ExpiringDict(max_len=200000, max_age_seconds=1800)
    >>> dict_test['test'] = 1
    >>> pickled_object = dill.dumps(dict_test)
    >>> original_object = dill.loads(pickled_object)  # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
    Traceback (most recent call last):
    ...
    TypeError: __init__() missing 2 required positional arguments: 'max_len' and 'max_age_seconds'

    """
    pass


def test_pickle_expiring_dict_pass():
    """
    >>> # test to pickle ordered dict
    >>> dict_test=ExpiringDict(max_len=200*1E3, max_age_seconds=30*60)
    >>> dict_test['test'] = 1
    >>> dict_test_pickable = OrderedDict(dict_test)
    >>> pickled_object = dill.dumps(dict_test_pickable)
    >>> original_object = dill.loads(pickled_object)  # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
    >>> original_object['test']
    1

    """
    pass


def main():
    pass


if __name__ == "__main__":
    main()

RuntimeError with Python3.5

I am seeing a very annoying bug when using this library with Python 3.5:


In [1]: import sys
   ...: 
   ...: # Python 3.5 has a broken 'optimised' version of OrderedDict which can corrupt itself at high load.
   ...: if sys.version_info.major == 3 and sys.version_info.minor == 5:
   ...:     from test import support
   ...: 
   ...:     py_coll = support.import_fresh_module('collections', blocked=['_collections'])
   ...:     OrderedDict = py_coll.OrderedDict
   ...: else:
   ...:     from collections import OrderedDict
   ...:     

In [2]: from expiringdict import ExpiringDict

In [3]: ExpiringDict(max_len=60, max_age_seconds=4)
Out[3]: ExpiringDict()

In [4]: d=ExpiringDict(max_len=60, max_age_seconds=4)

In [5]: d["soc3"] = {"soc":14}

In [6]: d["soc2"] = {"soc":11}

In [7]: d
Out[7]: ---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
~/.virtualenvs/hagenv2g/lib/python3.5/site-packages/IPython/core/formatters.py in __call__(self, obj)
    670                 type_pprinters=self.type_printers,
    671                 deferred_pprinters=self.deferred_printers)
--> 672             printer.pretty(obj)
    673             printer.flush()
    674             return stream.getvalue()
~/.virtualenvs/hagenv2g/lib/python3.5/site-packages/IPython/lib/pretty.py in pretty(self, obj)
    366                 if cls in self.type_pprinters:
    367                     # printer registered in self.type_pprinters
--> 368                     return self.type_pprinters[cls](obj, self, cycle)
    369                 else:
    370                     # deferred printer

~/.virtualenvs/hagenv2g/lib/python3.5/site-packages/IPython/lib/pretty.py in _ordereddict_pprint(obj, p, cycle)
    829             p.text('...')
    830         elif len(obj):
--> 831             p.pretty(list(obj.items()))
    832 
    833 def _deque_pprint(obj, p, cycle):

~/.virtualenvs/hagenv2g/lib/python3.5/site-packages/expiringdict/__init__.py in items(self)
    111         """ Return a copy of the dictionary's list of (key, value) pairs. """
    112         r = []
--> 113         for key in self:
    114             try:
    115                 r.append((key, self[key]))

RuntimeError: OrderedDict mutated during iteration

What if no one lockup inserted key after expiration? (record stay in dictionary for ever)

items will remove from data structure when some one lockup them and the expiration time is lower than current time.
if no one lockup the key after expiration time that key will stay in memory for ever and this could result serious memory problem.

Just saw another project that might have violated Apache License v2.0

The repo which looks derived from this expiringdict project: https://github.com/rfyiamcool/expiredict/blob/05900e6411015c177a82e1323a8e0aff7da717a8/expiredict/__init__.py
And the same project submitted to PYPI: https://pypi.python.org/pypi/expiredict

Potentially violated clauses include (but are not limited to):

4 . Redistribution.
...
(a) You must give any other recipients of the Work or Derivative Works a copy of this License; and

(b) You must cause any modified files to carry prominent notices stating that You changed the files; and

set max_len=None or infinite

Hello

I want an orderdict that removes a key-value pair after a certain time.
expiringdict is very well suited for this. But I want to add an unlimited number of values.
Is that possible at all?

for example setting the max_len=None
or setting a flag infinite?

Thanks!

Getting the length of the dictionary should return # of items not expired

Since items are only removed on retrieval, calling the len method on the dictionary will return all the items in the dictionary--including the expired items.

Length should iterate through the internal list and find what items are expired and remove them appropriately. Since it's internally implemented using an OrderedDict, we can perform a binary search to find what items in the dictionary have expired in Log(n) time

Disclaimer: I am the author