zopefoundation / zodb

Python object-oriented database

ZODB, a Python object-oriented database

ZODB provides an object-oriented database for Python that provides a high-degree of transparency. ZODB runs on Python 3.7 and above. It also runs on PyPy.

• no separate language for database operations
• very little impact on your code to make objects persistent

• no database mapper that partially hides the database.

  Using an object-relational mapping is not like using an object-oriented database.

• almost no seam between code and database.

ZODB is an ACID Transactional database.

To learn more, visit: https://zodb-docs.readthedocs.io

The github repository is at https://github.com/zopefoundation/zodb

If you're interested in contributing to ZODB itself, see the developer notes.

zodb's Issues

MANIFEST.in issues

$ check-manifest
listing source files under version control: 173 files and directories
copying source files to a temporary directory
building an sdist: ZODB-4.0.1dev.tar.gz: 165 files and directories
files in version control do not match the sdist!
missing from sdist:
.travis.yml
COPYING
bootstrap.py
buildout.cfg
ez_setup.py
log.ini
release.py
tox.ini
suggested MANIFEST.in rules:
include *.py
include .travis.yml
include COPYING
include buildout.cfg
include log.ini
include tox.ini

import ZODB UnicodeDecodeError

---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
<ipython-input-1-d3c08fd6406c> in <module>()
----> 1 import ZODB, ZODB.FileStorage
      2 
      3 db = ZODB.DB(None)
      4 connection = db.open()
      5 root = connection.root

C:\Users\user\Miniconda3\lib\site-packages\ZODB\__init__.py in <module>()
     26 del mapping, list, sys
     27 
---> 28 from ZODB.DB import DB, connection

C:\Users\user\Miniconda3\lib\site-packages\ZODB\DB.py in <module>()
     21 import warnings
     22 
---> 23 from ZODB.broken import find_global
     24 from ZODB.utils import z64
     25 from ZODB.Connection import Connection

C:\Users\user\Miniconda3\lib\site-packages\ZODB\broken.py in <module>()
     21 
     22 import ZODB.interfaces
---> 23 from ZODB._compat import IMPORT_MAPPING
     24 from ZODB._compat import NAME_MAPPING
     25 

C:\Users\user\Miniconda3\lib\site-packages\ZODB\_compat.py in <module>()
     40     # Python 3.x: can't use stdlib's pickle because
     41     # http://bugs.python.org/issue6784>
---> 42     import zodbpickle.pickle
     43     HIGHEST_PROTOCOL = 3
     44     from _compat_pickle import IMPORT_MAPPING, NAME_MAPPING

C:\Users\user\Miniconda3\lib\site-packages\zodbpickle\pickle.py in <module>()
      2 
      3 if sys.version_info[0] >= 3:
----> 4     from .pickle_3 import *
      5 else:
      6     from .pickle_2 import *

C:\Users\user\Miniconda3\lib\site-packages\zodbpickle\pickle_3.py in <module>()
   1481 # Use the faster _pickle if possible
   1482 try:
-> 1483     from zodbpickle._pickle import *
   1484 except ImportError:
   1485     Pickler, Unpickler = _Pickler, _Unpickler

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x83 in position 4: invalid start byte

Zope loses data irretrievably upon restart

We’ve been running Zope servers for about 15 years. In the last few years we have had a number of Zope servers instantly lose data upon restart. This has happened to us about 5 times over the past 2 years on 4 different servers.

Upon restarting Zope we notice a rollback of data to sometimes months old and a Data.fs with a much older timestamp than expected. At this point the data was irretrievable. We find that it is not in any backups or anywhere on the file system.

We have finally worked out what is happening but not exactly why. Below is a detailed explanation of what we are seeing.

We are keen to know if anybody else is having similar issues or if perhaps we have setup something incorrectly and need to change how we pack the database. Also this may serve as a warning to others.

Explanation
The issue is that Zope is losing the file descriptor, the Data.fs still remains in the folder but Zope keeps writing to a (deleted) file descriptor which can be seen with lsof. Every server affected by this was running ZEO and had zeopack in the default configuration. We believe for some reason when zeopack tries to pack the ZODB, that when the Data.fs is moved and renamed to Data.fs.old and eventually deleted, Zope doesn’t recognize the new file.

This is what we believe happens:

1. Zope is connected to a file Data.fs.
2. zeopack packs this file into a new file Data.fs.tmp
3. zeopack deletes any file named Data.fs.old
4. zeopack then renames Data.fs to Data.fs.old
5. zeopack then renames Data.fs.tmp to Data.fs

The problem is that Zope is connected directly to the Data.fs file via its handler and even though it is renamed it continues to write to Data.fs.old.

Later on, we do a subsequent zeopack and the Data.fs.old file is deleted in the process, but Zope keeps writing to this deleted file.

We now have a situation where Zope is connected to a deleted file and is writing all its data to it. If Zope is ever stopped, the connection is lost and so effectively is the file.

It is true that expert forensics could retrieve the file but in the first few instances that this happened we were totally baffled as to where the data had gone. In one case we had not restarted Zope for 4 months and thus we lost 4 months of data.

Environment
We are running standard Plone servers on Debian 5.0.10 in this case, with following:
Plone 4.3.4.1 (4309)
CMF 2.2.8
Zope 2.13.22
Python 2.7.8 (default, Jan 27 2015, 14:19:28) [GCC 4.3.2]
PIL 2.3.0 (Pillow)
ZEO: plone.recipe.zeoserver 1.2.6
Effective-user is set correctly for this instance “landcarevic” and properly mounts the Data.fs under normal circumstances.

Detailed Symptoms
This is a real case we discovered and managed to retrieve the lost file.

Zope will continue to write to a Data.fs that has been technically deleted. We are able to notice this because the Data.fs.tmp will have a much more recent timestamp than the Data.fs in the var/filestorage directory.

-rw-r--r-- 1 landcarevic psacln  6735952811 2015-04-12 00:11 Data.fs
-rw-r--r-- 1 landcarevic psacln    18594749 2015-04-12 00:11 Data.fs.index
-rw-r--r-- 1 landcarevic psacln           6 2015-03-30 11:20 Data.fs.lock
-rw-r--r-- 1 landcarevic psaserv   14600344 2015-05-26 10:27 Data.fs.tmp

An “lsof | grep Data.fs” can show if the file descriptor is pointing to a deleted file on the fs:

python2.7  9307  landcarevic    7uW     REG               0,31          6   4986845 /var/www/vhosts/landcarevic.net.au/plone43/var/filestorage/Data.fs.lock
python2.7  9307  landcarevic   11u      REG               0,31   14600344   4986235 /var/www/vhosts/landcarevic.net.au/plone43/var/filestorage/Data.fs.tmp
python2.7  9307  landcarevic   12u      REG               0,31 7058741434   4986349  (deleted)/var/www/vhosts/landcarevic.net.au/plone43/var/filestorage/Data.fs.old

The process id for the instance is 9307 and it’s still holding open a Data.fs.old that has been deleted. Most likely due to a ZEO pack before from the cron in this case. Because it packs the Data.fs and then moves the old one to Data.fs.old which eventually gets deleted. But Zope never properly recognises the new Data.fs for some reason.

Recovery
As long as the Zope server doesn’t get shut down or restarted, it will hold this open and continue to write to the disk. It can be recovered by copying from /proc where /proc/{process ID}/fd/{file descriptor ID} can be found from running lsof as previously:

# cp /proc/9307/fd/12 /home/alyssa/Data.fs-current
# ls -al /home/alyssa/
... 
-rw-r--r--  1 root root 7058741434 2015-05-26 11:11 Data.fs-current

Which has the correct timestamp and file size.

Write "Transactions" topic

From @jimfulton on August 22, 2016 17:41

Copied from original issue: zopefoundation/zodbdocs#13

recent changes seem to break winbot

http://winbot.zope.org/builders/ZODB_dev%20py_270_win32/builds/2116/steps/test/logs/stdio

(I plan to update builders to the currently supported platforms)

(also, @jimfulton you need failure emails from winbot?)

Could we get the ZODB guide in github?

I think this is it: http://svn.zope.org/zodbdocs/
If it would help, I can svn2git it to my github account.

Uncaught exception when shutting down server

Copy&Pasted from zodb/relstorage#14
written by @neilferreira

Hi all

I'm getting an error when shutting down my server in my development environment.
Plone 4.3.3
RelStorage = 1.5.1

sauna and get the same issue when CTRL + C'ing the insance.

[instance]
<= instance_base
recipe = plone.recipe.zope2instance
http-address = 8080
blob-dir ${buildout:var-dir}/blobstorage
dsn dbname = ${buildout:database-name} host=${buildout:database-host} user=${buildout:database-user}

^C2014-06-17 14:15:21 INFO SignalHandler Caught signal SIGINT
2014-06-17 14:15:21 INFO Z2 Shutting down
Traceback (most recent call last):
  File "/home/neilf/development/website/website-website.wa.gov.au-v2/parts/instance/bin/interpreter", line 328, in 
    exec(compile(file__f.read(), __file, "exec"))
  File "/home/neilf/development/website/buildout-cache/eggs/Zope2-2.13.22-py2.7.egg/Zope2/Startup/run.py", line 76, in 
    run()
  File "/home/neilf/development/website/buildout-cache/eggs/Zope2-2.13.22-py2.7.egg/Zope2/Startup/run.py", line 26, in run
    starter.run()
  File "/home/neilf/development/website/buildout-cache/eggs/Zope2-2.13.22-py2.7.egg/Zope2/Startup/init.py", line 108, in run
    self.shutdown()
  File "/home/neilf/development/website/buildout-cache/eggs/Zope2-2.13.22-py2.7.egg/Zope2/Startup/init.py", line 113, in shutdown
    db.close()
  File "/home/neilf/development/website/buildout-cache/eggs/ZODB3-3.10.5-py2.7-linux-x86_64.egg/ZODB/DB.py", line 624, in close
    @self._connectionMap
  File "/home/neilf/development/website/buildout-cache/eggs/ZODB3-3.10.5-py2.7-linux-x86_64.egg/ZODB/DB.py", line 506, in _connectionMap
    self.pool.map(f)
  File "/home/neilf/development/website/buildout-cache/eggs/ZODB3-3.10.5-py2.7-linux-x86_64.egg/ZODB/DB.py", line 206, in map
    self.all.map(f)
  File "/home/neilf/development/website/buildout-cache/eggs/transaction-1.1.1-py2.7.egg/transaction/weakset.py", line 58, in map
    f(elt)
  File "/home/neilf/development/website/buildout-cache/eggs/ZODB3-3.10.5-py2.7-linux-x86_64.egg/ZODB/DB.py", line 628, in _
    c._release_resources()
  File "/home/neilf/development/website/buildout-cache/eggs/ZODB3-3.10.5-py2.7-linux-x86_64.egg/ZODB/Connection.py", line 1075, in _release_resources
    c._storage.release()
AttributeError: 'NoneType' object has no attribute 'release'

missing windows eggs

Seems that the current BTree release is missing windows eggs.

readCurrent has no effect in read-only transactions

Because connections don't join transaction unless there are writes.

This is based on code inspection. I/we should verify this problem with a failing test and then fix it by calling self.register() in readCurrent.

(Or maybe I'm wrong.)

zodb assumes the transaction description is an ascii string

I'm writing a Pyramid application, using ZODB 4.2.0 for the database and pyramid_tm to manage transactions.

In case it matters, I'm using Python 3.4.

pyramid_tm sets the description of a transaction to the path_info of the request.

For example, if a client sends a GET /foo/bar request, then the transaction description will be set to the string /foo/bar.

Then ZODB, in its FileStorage.format:TxnHeader.asString method, uses its utils.as_bytes function to encode a few strings related to the transaction, among which the description.

And utils.as_bytes uses the ascii encoder.

This all breaks down when the request path_info is not made up only of ASCII characters.

In many web applications, path_info can be any sort of things. In mine, I can even have Chinese characters, for example a GET /森 request.

So it seems ZODB shouldn't assume everything can be encoded as ASCII.

_p_resolveConflict() should be called even if a conflicting change doesn't change the state

What I'd like is to commit the following change:

--- a/src/ZODB/ConflictResolution.py
+++ b/src/ZODB/ConflictResolution.py
@@ -271,13 +271,6 @@ def tryToResolveConflict(self, oid, committedSerial, oldSerial, newpickle,
         if not committedData:
             committedData  = self.loadSerial(oid, committedSerial)

-        if newpickle == oldData:
-            # old -> new diff is empty, so merge is trivial
-            return committedData
-        if committedData == oldData:
-            # old -> committed diff is empty, so merge is trivial
-            return newpickle
-
         newstate = unpickler.load()
         old       = state(self, oid, oldSerial, prfactory, oldData)
         committed = state(self, oid, committedSerial, prfactory, committedData)

(i.e. revert to the behaviour of ZODB 3.10.3-)

For example, we need such behaviour to implement an "asynchronous" cache:

• initially, a cache value is not filled (e.g. None is used to describe this state)
• a transaction fills the cache (actually done by a background zope) (None -> "foo")
• a concurrent transaction invalidates the cache due to some user action (None -> None), and pushes a new background task to fill the cache

In our case, the final value must be None and we implement _p_resolveConflict() for that. However, with ZODB 3.10.4+, None -> None is seen as "no change" and the result is "foo".

Weird discarded comparison lurking in untested code

I was reading the source code for some reason yesterday and I noticed this bit in FileStorage.py, in the read_index function:

    # Caution:  fsIndex doesn't have an efficient __nonzero__ or __len__.
    # That's why we do try/except instead.  fsIndex.maxKey() is fast.
    try:
        maxoid = index.maxKey()
    except ValueError:
        # The index is empty.
        maxoid == z64

    return pos, maxoid, ltid

I think the except branch meant to do an assignment instead of a comparison. There are no other assignments to maxoid in this function, so this branch, if ever taken, would raise an UnboundLocalError.

I don't have the time right now to come up with a unit test triggering this, or I'd fix it. But I don't want it to be forgotten, so I'm filing this isssue.

Write "Performance" topic.

From @jimfulton on August 22, 2016 18:35

There isn't a placeholder reference for this yet.

Copied from original issue: zopefoundation/zodbdocs#21

tryToResolveConflict masks BTreesConflictError details

Most important would be the reason
Including details in ConflictError might be hard, but what about

    except (ConflictError, BadClassName), e:
        logger.error(str(e))

Might duplicate log entries, but oh well...

BaseStorage.copyTransactionsFrom fails to copy extension data.

From this comment.

Should it? If so, RelStorage should too.

setup.py classifiers not inline with sys.version_info checking

Is there a way to save all the undoInfo when packing?

I don't know if this is the correct place to add this issue, so feel free to close if inappropriate.

We're using Plone which uses Zope. In Zope2 we have an "Undo" tab that lists some information about the transactions:

We get this info by using undoInfo:

[{'description': '/Plone/folder_listing', 'size': 573, 'user_name': ' admin', 'id': 'A7KTm8ZgC0Q=', 'time': 1445450626.494181}, {'description': '/Plone/portal_quickinstaller/reinstallProducts', 'size': 1636907, 'user_name': ' admin', 'id': 'A7KTm55XNyI=', 'time': 1445450617.111098}, {'description': '/Plone/manage_delObjects', 'size': 70665, 'user_name': ' admin', 'id': 'A7KThLXFqpk=', 'time': 1445449242.602844}, {'description': '/Plone/atct_edit', 'size': 244, 'user_name': ' admin', 'id': 'A7KTc2rAXLs=', 'time': 1445448205.019863}, {'description': '/Plone/atct_edit', 'size': 3231, 'user_name': ' admin', 'id': 'A7KTc1N618w=', 'time': 1445448199.565591}, {'description': '/Plone/folder_listing', 'size': 573, 'user_name': ' admin', 'id': 'A7KTcz/Ptsw=', 'time': 1445448194.955793}, {'description': '/Plone/portal_quickinstaller/reinstallProducts', 'size': 1634373, 'user_name': ' admin', 'id': 'A7KTcxC6wBE=', 'time': 1445448183.920975}, {'description': '/Plone/manage_delObjects', 'size': 67971, 'user_name': ' admin', 'id': 'A7KTcsyp0kQ=', 'time': 1445448167.967976}, {'description': '/Plone/atct_edit', 'size': 244, 'user_name': ' admin', 'id': 'A7KTcY6ZhBE=', 'time': 1445448093.421798}, {'description': '/Plone/atct_edit', 'size': 2803, 'user_name': ' admin', 'id': 'A7KTcYC9YkQ=', 'time': 1445448090.173386}, {'description': '/Plone/folder_listing', 'size': 573, 'user_name': ' admin', 'id': 'A7KTcV9iAKo=', 'time': 1445448082.355349}, {'description': '/Plone/portal_quickinstaller/reinstallProducts', 'size': 1593303, 'user_name': ' admin', 'id': 'A7KTcTMyqRE=', 'time': 1445448071.999506}, {'description': '/Plone/manage_delObjects', 'size': 67932, 'user_name': ' admin', 'id': 'A7KTcPcptUQ=', 'time': 1445448057.92881}, {'description': '/Plone/portal_quickinstaller/reinstallProducts', 'size': 681680, 'user_name': ' admin', 'id': 'A7KTb6yxMxE=', 'time': 1445447980.474731}, {'description': '/Plone/manage_delObjects', 'size': 67945, 'user_name': ' admin', 'id': 'A7KTbNoEUZk=', 'time': 1445447811.097704}, {'description': '/Plone/portal_quickinstaller/reinstallProducts', 'size': 679331, 'user_name': ' admin', 'id': 'A7KTbKbS25k=', 'time': 1445447799.099296}, {'description': '/Plone/manage_delObjects', 'size': 67931, 'user_name': ' admin', 'id': 'A7KTbAUBiWY=', 'time': 1445447761.173282}, {'description': '/Plone/portal_quickinstaller/reinstallProducts', 'size': 676241, 'user_name': ' admin', 'id': 'A7KTazldHRE=', 'time': 1445447713.444623}, {'description': '/Plone/manage_delObjects', 'size': 67946, 'user_name': ' admin', 'id': 'A7KTZskwOu4=', 'time': 1445447447.153531}, {'description': '/Plone/portal_quickinstaller/reinstallProducts', 'size': 672906, 'user_name': ' admin', 'id': 'A7KTZpX2yWY=', 'time': 1445447435.147815}]

Then it's rendered in html.

When packing, we lose this information that would be useful in some situations. Is it feasible to write a log to disk (it can be the dict above that we got from undoInfo) of this information when packing? This could be an environment variable if you don't want to add this overhead to all systems that use ZODB.

Multiple indexed views for object collection?

Hello ZODB team

My use case is rather simple:
I have some BTree collections of objects keyed by some unique integer key (i.e IOBTree collections of Persistent objects). My application is not really search-centric, but in these very few cases (few, but frequently happening) that a lookup has to be carried-out by another (non-key) field, i have to iterate the entire container sequentially.

Is it possible to maintain some secondary index on a certain BTree collection? I do not mean directly (i.e on the same tree) but maybe maintain an auxiliary value->{set-of-keys} mapping?

I have implemented some Collection class on top of BTree, something like:

tree = root['users'] = IOBTree()
ix1 = root['users.by_name'] = OOBTree()
users = Collection(tree, name=ix1) # declare primary tree and indices

# The collection should follow underlying BTree api as much as possible
u1 = User(id=12, name='Foo')
users.insert(u1) # also inserts to name-based index
...
# The collection should return proxy objects on search operations 
p1 = users.get(12) # returns a proxy (to u1) that intercepts __setattr__, __delattr__
p1.name #  delegate to u1.name
p1.name = 'Baz' # delegate to u1, but also update name-based index

Of course, my implementation has several limitations and is only a workaround for my use case. Maybe some more mature solutions already exist ? or maybe this is a reason to migrate to a relational model ?

Documentation: Writing persistent objects: Be careful about __eq__ and __hash__

In general, objects without custom __eq__ and __hash__ objects are going to be friendlier on the DB and the cache (if you can get away with identity semantics).

Suppose you have two classes:

import uuid

class WithHash(Persistent):
   def __init__(self):
      self.id = uuid.uuid()
   def __eq__(self, other):
      return self.id == other.id
   def __hash__(self):
      return hash(self.id)

class WithoutHash(Persistent):
    def __init__(self):
       self.id = uuid.uuid()

Now if you had some dictionaries using those classes as keys:

conn = db.open()
conn.root.with_hash_dict = {WithHash(): i for i in range(5000)}
conn.root.wout_hash_dict = {WithoutHash(): i for i in range(5000)}

Doing something like:

conn = db.open()
len(conn.root.with_hash_dict)

is going to unghost 5000 WithHash objects, whereas

conn = db.open()
len(conn.root.wout_hash_dict)

isn't going to unghost any objects (because unpickling a dictionary re-hashes all the keys, and hash() looks up __hash__ on the class not the instance, so if the __hash__ method doesn't access any attributes---like the default one---nothing has to be unghosted) .

Even if your object cache is sized appropriately, large-ish dictionaries can take a long time to unpickle when accessed for the first time in a particular connection, adding lots of load to the DB and/or cache system; the same happens when creating a dictionary in memory for the first time of such persistent objects.

If you can accept identity semantics (and for persistent objects, you surprisingly often can), it's better to avoid custom __eq__ and __hash__ methods if you'll ever be creating dictionaries or sets of your persistent objects.

This is a lesson we learned the hard way; coming from a Java background almost all of our objects defined custom __eq__ and __hash__ methods, and that was fine until we started to get a lot of objects, when it became a performance burden. Now it turns out that many such of those objects don't need these methods.

Worth adding to the docs?

ZODB.serialize.ObjectWriter should fall back to __reduce__() right?

I know I am trying some 'black magic', but I am creating dynamic classes. I would like to store these in ZODB. These run-time classes implement __getnewargs__ and __reduce__ as well as subclass Persistent. Pickle will dump and load them, because of __reduce__ providing a viable callable to reproduce these dynamic classes.

ZODB fails with PicklingError because the type(obj) is not the same as the global
Now I have gotten it to work using a dirty workaround

from ZODB.serialize import ObjectWriter
orig_serialize = ObjectWriter.serialize
def _serialize(self, obj):
    try:
        return orig_serialize(self, obj)
    except:
        reduce = obj.__reduce__()  # contains (callable/class, newargs, __getstate__() )
        if reduce[1]:
            meta = reduce[0], reduce[1]
        else:
            meta = reduce[0]
        return self._dump(meta, reduce[2])
ObjectWriter.serialize = _serialize

Now my dirty workaround works, but I would much rather have the existing serialize method have this functionality

Closing a Connection should set `transaction_manager` to None

See this thread: https://github.com/zopefoundation/ZODB/pull/105/files/67388dcd9bab50051510e72c76b8ed29a8773a9c#r78302105

MANIFEST.in file includes too much

The MANIFEST.in file doesn't fit

warning: no previously-included files matching '.dll' found anywhere in distribution
warning: no previously-included files matching '.pyc' found anywhere in distribution
warning: no previously-included files matching '.pyo' found anywhere in distribution
warning: no previously-included files matching '.so' found anywhere in distribution

PersistentMapping copy behaves weirdly on latest versions

I've always had troubles with PersistentMapping.copy(), which caused connection._registered_objects to fill itself as if the PersistentMapping had been modified.

So so far, i used the "copy" module to do this job. But with 4.X releases, there has been a semantic change somewhere, and copy.copy() now creates a new mapping which however changes when the original mapping has an item deleted. Below is the test which PASSES with ZODB 4.3.1.
Is it a known behaviour ?

 def test_mapping_copy(self):

    a = PersistentMapping(dict(a=3, b=5))
    b = a.copy()
    c = copy.copy(a)
    d = copy.deepcopy(a)

    assert a == b
    assert a == c
    assert a == d
    assert a is not b
    assert a is not c
    assert b is not c
    assert a is not d

    del a["a"]
    assert b["a"]  # NOT impacted
    assert "a" not in c, c  # BUG, is impacted!
    assert d["a"]  # NOT impacted

Attempts at Python2-compatible pickles cause unpicklable objects

Under Py3k, ZODB._compat uses 'zodbpickle', but restricts the maximum protocol level to 2, and passes 'bytes_as_strings=True' to the pickling / unpickling methods. This combination causes some values to be unpicklable. E.g., see this gist:

https://gist.github.com/tseaver/5726422

Against the ZODB trunk, the second stanza fails, because the first has written unpicklable data to the 'tree' object.

Write "Blobs" topic

From @jimfulton on August 22, 2016 17:42

Copied from original issue: zopefoundation/zodbdocs#18

ConflictResolvingStorage should implement new_instance?

The registerDB method of CRS sets some instance attributes (which are needed by zc.zlibstorage). However, CRS doesn't implement new_instance so those attributes aren't copied to IMVCCStorage children.

This quietly breaks conflict resolution for RelStorge+zc.zlibstorage because the records can't be decompressed. (See zodb/relstorage#71)

registerDB is only called once, by the DB object on the "root" storage, not by each Connection that calls new_instance. Now that all storages are wrapped by an IMVCCStorage proxy, I wonder if this problem has gotten worse.

It should be easier to change one's mind about persistence

You can't currently change your mind about subclassing persistent. This can be a painful pitfall. We should see if we can make this possible, or perhaps less painful.

activity monitors???

In documenting DB, I was reminded about activity monitors. This is a feature that should be documented or dropped.

If the feature is kept, does it need to be pluggable? It's rather complex right now.

pack FileStorage: No space left on device, Data.fs.pack should be removed

Data.fs.pack is the file that gets created while packing.
It is left alone if the disk gets full, leaving 0 bytes free on the disk.
0 bytes free then practically blocks any operation, because the live Data.fs is on the same disk.

 File "d:\home\.buildout\eggs\zodb3-3.10.5-py2.7-win-amd64.egg\ZODB\DB.py", line 812, in pack
   self.storage.pack(t, self.references)
 File "d:\home\.buildout\eggs\zodb3-3.10.5-py2.7-win-amd64.egg\ZODB\FileStorage\FileStorage.py", line 1078, in pack
   pack_result = self.packer(self, referencesf, stop, gc)
 File "d:\home\.buildout\eggs\zodb3-3.10.5-py2.7-win-amd64.egg\ZODB\FileStorage\FileStorage.py", line 1034, in packer
   opos = p.pack()
 File "d:\home\.buildout\eggs\zodb3-3.10.5-py2.7-win-amd64.egg\ZODB\FileStorage\fspack.py", line 407, in pack
   ipos, opos = self.copyToPacktime()
 File "d:\home\.buildout\eggs\zodb3-3.10.5-py2.7-win-amd64.egg\ZODB\FileStorage\fspack.py", line 462, in copyToPacktime
   new_tpos, pos = self.copyDataRecords(pos, th)
 File "d:\home\.buildout\eggs\zodb3-3.10.5-py2.7-win-amd64.egg\ZODB\FileStorage\fspack.py", line 544, in copyDataRecords
   self.writePackedDataRecord(h, data, new_tpos)
 File "d:\home\.buildout\eggs\zodb3-3.10.5-py2.7-win-amd64.egg\ZODB\FileStorage\fspack.py", line 573, in writePackedDataRecord
   self._tfile.write(data)
IOError: [Errno 28] No space left on device

Write "Multi-databases" topic

From @jimfulton on August 22, 2016 17:43

Copied from original issue: zopefoundation/zodbdocs#20

pip install zodb fails

C:\Users\admin>pip install zodb
Collecting zodb
  Downloading ZODB-5.0.0.tar.gz (467kB)
    100% |################################| 471kB 1.5MB/s
    Complete output from command python setup.py egg_info:
    warning: no previously-included files matching '*.dll' found anywhere in distribution
    warning: no previously-included files matching '*.pyc' found anywhere in distribution
    warning: no previously-included files matching '*.pyo' found anywhere in distribution
    warning: no previously-included files matching '*.so' found anywhere in distribution
    warning: no previously-included files matching 'coverage.xml' found anywhere in distribution
    no previously-included directories found matching 'docs\_build'
    no previously-included directories found matching 'persistent\__pycache__'
    Traceback (most recent call last):
      File "C:\Users\admin\Anaconda3\lib\distutils\core.py", line 148, in setup
        dist.run_commands()
      File "C:\Users\admin\Anaconda3\lib\distutils\dist.py", line 955, in run_commands
        self.run_command(cmd)
      File "C:\Users\admin\Anaconda3\lib\distutils\dist.py", line 974, in run_command
        cmd_obj.run()
      File "c:\users\admin\anaconda3\lib\site-packages\setuptools-23.0.0-py3.5.egg\setuptools\command\bdist_egg.py", line 161, in run
      File "c:\users\admin\anaconda3\lib\site-packages\setuptools-23.0.0-py3.5.egg\setuptools\command\bdist_egg.py", line 147, in call_command
      File "C:\Users\admin\Anaconda3\lib\distutils\cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "C:\Users\admin\Anaconda3\lib\distutils\dist.py", line 974, in run_command
        cmd_obj.run()
      File "c:\users\admin\anaconda3\lib\site-packages\setuptools-23.0.0-py3.5.egg\setuptools\command\install_lib.py", line 10, in run
      File "C:\Users\admin\Anaconda3\lib\distutils\command\install_lib.py", line 107, in build
        self.run_command('build_ext')
      File "C:\Users\admin\Anaconda3\lib\distutils\cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "C:\Users\admin\Anaconda3\lib\distutils\dist.py", line 974, in run_command
        cmd_obj.run()
      File "c:\users\admin\anaconda3\lib\site-packages\setuptools-23.0.0-py3.5.egg\setuptools\command\build_ext.py", line 66, in run
      File "C:\Users\admin\Anaconda3\lib\site-packages\Cython\Distutils\build_ext.py", line 164, in run
        _build_ext.build_ext.run(self)
      File "C:\Users\admin\Anaconda3\lib\distutils\command\build_ext.py", line 338, in run
        self.build_extensions()
      File "C:\Users\admin\Anaconda3\lib\site-packages\Cython\Distutils\build_ext.py", line 172, in build_extensions
        self.build_extension(ext)
      File "c:\users\admin\anaconda3\lib\site-packages\setuptools-23.0.0-py3.5.egg\setuptools\command\build_ext.py", line 178, in build_extension
      File "C:\Users\admin\Anaconda3\lib\distutils\command\build_ext.py", line 532, in build_extension
        depends=ext.depends)
      File "C:\Users\admin\Anaconda3\lib\distutils\_msvccompiler.py", line 306, in compile
        self.initialize()
      File "C:\Users\admin\Anaconda3\lib\distutils\_msvccompiler.py", line 199, in initialize
        vc_env = _get_vc_env(plat_spec)
      File "C:\Users\admin\Anaconda3\lib\distutils\_msvccompiler.py", line 85, in _get_vc_env
        raise DistutilsPlatformError("Unable to find vcvarsall.bat")
    distutils.errors.DistutilsPlatformError: Unable to find vcvarsall.bat

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "C:\Users\admin\Anaconda3\lib\site-packages\setuptools-23.0.0-py3.5.egg\setuptools\sandbox.py", line 154, in save_modules
      File "C:\Users\admin\Anaconda3\lib\site-packages\setuptools-23.0.0-py3.5.egg\setuptools\sandbox.py", line 195, in setup_context
      File "C:\Users\admin\Anaconda3\lib\site-packages\setuptools-23.0.0-py3.5.egg\setuptools\sandbox.py", line 243, in run_setup
      File "C:\Users\admin\Anaconda3\lib\site-packages\setuptools-23.0.0-py3.5.egg\setuptools\sandbox.py", line 273, in run
      File "C:\Users\admin\Anaconda3\lib\site-packages\setuptools-23.0.0-py3.5.egg\setuptools\sandbox.py", line 242, in runner
      File "C:\Users\admin\Anaconda3\lib\site-packages\setuptools-23.0.0-py3.5.egg\setuptools\sandbox.py", line 46, in _execfile
      File "C:\Users\admin\AppData\Local\Temp\easy_install-_b_vbbt0\persistent-4.2.1\setup.py", line 115, in <module>
        read_file("CHANGES.rst").decode('latin-1')))
      File "C:\Users\admin\Anaconda3\lib\distutils\core.py", line 163, in setup
        raise SystemExit("error: " + str(msg))
    SystemExit: error: Unable to find vcvarsall.bat

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "C:\Users\admin\Anaconda3\lib\site-packages\setuptools-23.0.0-py3.5.egg\setuptools\command\easy_install.py", line 1100, in run_setup
      File "C:\Users\admin\Anaconda3\lib\site-packages\setuptools-23.0.0-py3.5.egg\setuptools\sandbox.py", line 246, in run_setup
      File "C:\Users\admin\Anaconda3\lib\contextlib.py", line 77, in __exit__
        self.gen.throw(type, value, traceback)
      File "C:\Users\admin\Anaconda3\lib\site-packages\setuptools-23.0.0-py3.5.egg\setuptools\sandbox.py", line 195, in setup_context
      File "C:\Users\admin\Anaconda3\lib\contextlib.py", line 77, in __exit__
        self.gen.throw(type, value, traceback)
      File "C:\Users\admin\Anaconda3\lib\site-packages\setuptools-23.0.0-py3.5.egg\setuptools\sandbox.py", line 166, in save_modules
      File "C:\Users\admin\Anaconda3\lib\site-packages\setuptools-23.0.0-py3.5.egg\setuptools\sandbox.py", line 141, in resume
      File "C:\Users\admin\Anaconda3\lib\site-packages\setuptools-23.0.0-py3.5.egg\pkg_resources\_vendor\six.py", line 685, in reraise
        raise value.with_traceback(tb)
      File "C:\Users\admin\Anaconda3\lib\site-packages\setuptools-23.0.0-py3.5.egg\setuptools\sandbox.py", line 154, in save_modules
      File "C:\Users\admin\Anaconda3\lib\site-packages\setuptools-23.0.0-py3.5.egg\setuptools\sandbox.py", line 195, in setup_context
      File "C:\Users\admin\Anaconda3\lib\site-packages\setuptools-23.0.0-py3.5.egg\setuptools\sandbox.py", line 243, in run_setup
      File "C:\Users\admin\Anaconda3\lib\site-packages\setuptools-23.0.0-py3.5.egg\setuptools\sandbox.py", line 273, in run
      File "C:\Users\admin\Anaconda3\lib\site-packages\setuptools-23.0.0-py3.5.egg\setuptools\sandbox.py", line 242, in runner
      File "C:\Users\admin\Anaconda3\lib\site-packages\setuptools-23.0.0-py3.5.egg\setuptools\sandbox.py", line 46, in _execfile
      File "C:\Users\admin\AppData\Local\Temp\easy_install-_b_vbbt0\persistent-4.2.1\setup.py", line 115, in <module>
        read_file("CHANGES.rst").decode('latin-1')))
      File "C:\Users\admin\Anaconda3\lib\distutils\core.py", line 163, in setup
        raise SystemExit("error: " + str(msg))
    SystemExit: error: Unable to find vcvarsall.bat

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "C:\Users\admin\AppData\Local\Temp\pip-build-f_mm4j39\zodb\setup.py", line 159, in <module>
        include_package_data = True,
      File "C:\Users\admin\Anaconda3\lib\distutils\core.py", line 108, in setup
        _setup_distribution = dist = klass(attrs)
      File "C:\Users\admin\Anaconda3\lib\site-packages\setuptools-23.0.0-py3.5.egg\setuptools\dist.py", line 269, in __init__
      File "C:\Users\admin\Anaconda3\lib\site-packages\setuptools-23.0.0-py3.5.egg\setuptools\dist.py", line 313, in fetch_build_eggs
      File "C:\Users\admin\Anaconda3\lib\site-packages\setuptools-23.0.0-py3.5.egg\pkg_resources\__init__.py", line 826, in resolve
      File "C:\Users\admin\Anaconda3\lib\site-packages\setuptools-23.0.0-py3.5.egg\pkg_resources\__init__.py", line 1092, in best_match
      File "C:\Users\admin\Anaconda3\lib\site-packages\setuptools-23.0.0-py3.5.egg\pkg_resources\__init__.py", line 1104, in obtain
      File "C:\Users\admin\Anaconda3\lib\site-packages\setuptools-23.0.0-py3.5.egg\setuptools\dist.py", line 380, in fetch_build_egg
      File "C:\Users\admin\Anaconda3\lib\site-packages\setuptools-23.0.0-py3.5.egg\setuptools\command\easy_install.py", line 664, in easy_install
      File "C:\Users\admin\Anaconda3\lib\site-packages\setuptools-23.0.0-py3.5.egg\setuptools\command\easy_install.py", line 694, in install_item
      File "C:\Users\admin\Anaconda3\lib\site-packages\setuptools-23.0.0-py3.5.egg\setuptools\command\easy_install.py", line 875, in install_eggs
      File "C:\Users\admin\Anaconda3\lib\site-packages\setuptools-23.0.0-py3.5.egg\setuptools\command\easy_install.py", line 1114, in build_and_install
      File "C:\Users\admin\Anaconda3\lib\site-packages\setuptools-23.0.0-py3.5.egg\setuptools\command\easy_install.py", line 1102, in run_setup
    distutils.errors.DistutilsError: Setup script exited with error: Unable to find vcvarsall.bat

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in C:\Users\admin\AppData\Local\Temp\pip-build-f_mm4j39\zodb\

DB root-object creation should use a connection

When a DB is created, it tried to make sure that the root object exists. It doesn't use a connection to do this, but implements the transaction logic itself. It hasn't kept up, however with recent changes in the MVCC implementation, as I discovered in some failing ZEO tests.

DB logic should be updated to use a connection to do the transaction processing, even if changes are needed to Connection to deal with object 0 not being present.

Question: How to integrate with asyncio flow?

Hi,

How could and should ZODB be integrated into asyncio loop based application? What kind of issued would be expected? Is there known existing work for that somewhere?

My naive assumptions are that:

• The easiest solution would be a custom asyncio task executor, which would queue ZODB tasks into limited amount of worker threads, matching the size of ZODB connection pool. This probably works OOTB ZODB and with both local ZODB and ZEO.
• The next "easiest" solution would be to implement asyncio ZEO client connection. MVCC would probably come an issue.

Thanks a lot for your time and thoughts!

Writing persistent objects

From @jimfulton on August 26, 2016 13:10

Plus:

• Added machinery to text documentation examples
• Cleaned up config a bit

Copied from original issue: zopefoundation/zodbdocs/pull/22

Write "Threading and processes" topic

From @jimfulton on August 22, 2016 17:42

Copied from original issue: zopefoundation/zodbdocs#16

Write "Packing and garbage collection" topic

From @jimfulton on August 22, 2016 17:42

Copied from original issue: zopefoundation/zodbdocs#17

dynamic install_requires breaks universal wheels

pip 7 builds and caches wheels, which speeds up repeated installs.

Unfortunately, if you build a wheel of ZODB on Python 2.6, it will compute install_requires that works on 2.6 but fails on 2.7 (because no 'zodbpickle'). This wheel will be named ZODB-4.2.0b1-py2-none-any.whl, and so pip will try to use it on 2.7. This breaks people's buildbots.

STEPS TO REPRODUCE:

$ virtualenv --version
13.0.1

$ cat tox.ini
[tox]
envlist = py26,py27
skipsdist = true

[testenv]
skip_install = true
deps = ZODB
commands =
  python -c 'import ZODB'

$ rm ~/.cache/pip/wheels/*/*/*/*/ZODB-*.whl

$ tox --pre
py26 installed: argparse==1.3.0,BTrees==4.1.3,persistent==4.1.0,six==1.9.0,transaction==1.4.4,wheel==0.24.0,zc.lockfile==1.1.0,ZConfig==3.0.4,zdaemon==4.1.0,ZODB==4.2.0b1,zope.interface==4.1.2
py26 runtests: PYTHONHASHSEED='2859500558'
py26 runtests: commands[0] | python -c import ZODB
py27 recreate: /tmp/bork/.tox/py27
py27 installdeps: ZODB
py27 installed: BTrees==4.1.3,persistent==4.1.0,six==1.9.0,transaction==1.4.4,wheel==0.24.0,zc.lockfile==1.1.0,ZConfig==3.0.4,zdaemon==4.1.0,ZODB==4.2.0b1,zope.interface==4.1.2
py27 runtests: PYTHONHASHSEED='2859500558'
py27 runtests: commands[0] | python -c import ZODB
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/tmp/bork/.tox/py27/local/lib/python2.7/site-packages/ZODB/__init__.py", line 28, in <module>
    from ZODB.DB import DB, connection
  File "/tmp/bork/.tox/py27/local/lib/python2.7/site-packages/ZODB/DB.py", line 23, in <module>
    from ZODB.broken import find_global
  File "/tmp/bork/.tox/py27/local/lib/python2.7/site-packages/ZODB/broken.py", line 23, in <module>
    from ZODB._compat import IMPORT_MAPPING
  File "/tmp/bork/.tox/py27/local/lib/python2.7/site-packages/ZODB/_compat.py", line 42, in <module>
    import zodbpickle.pickle
ImportError: No module named zodbpickle.pickle
ERROR: InvocationError: '/tmp/bork/.tox/py27/bin/python -c import ZODB'
___________________________________ summary ___________________________________
  py26: commands succeeded
ERROR:   py27: commands failed

$ ls ~/.cache/pip/wheels/*/*/*/*/ZODB*
/home/mg/.cache/pip/wheels/98/e4/85/1b42bbea2eb9f3d1bd2043843104a15121cdeceaa24ad659d1/ZODB-4.2.0b1-py2-none-any.whl

I can see three possible solutions:

• require zodbpickle on Python 2.6
• use PEP 426 environment markers in extras_require instead of a dynamically-computed install_requires list
• tell bdist_wheel to use more specific Python tags

`python setup.py test` does not run the full test suite

lp440234_Setting__p_changed_of_a_Blob_w_no_uncomitted_changes_is_noop (from src/ZODB/tests/testblob.py) is an example of test that is not run by:

$ python setup.py test
...
Ran 1145 tests in 179.874s

OK

I test by inserting an error in this test, and I get no failure. However:

$ python setup.py test -m ZODB.tests.testFileStorage
...
FAIL: lp440234_Setting__p_changed_of_a_Blob_w_no_uncomitted_changes_is_noop (ZODB.tests.testblob)
...

I also wonder if Travis runs all tests.

Provide a way to get blobstorage size

i'd like to extend munin.zope or Products.ZNagios to return the size of blobstorages too.

is there a way to get these numbers via an api?
running du on linux systems can take ages (and is not platform independent)

maybe it's better to keep track of added blobfiles and keep the size in bytes as a separate value redundantly

``load(oid)`` must die!

Update: The discussion below primarily pertains to non-RelStorage storages. We ended up moving to a model where ZODB's MVCC implementation moves to an adapter applied to storages other than RelStorage. The discussion below pertains (now) to how that adapter interacts with storages. ZODB connections will still use load to load objects from IMVCCStorage instances, which will either be RelStorage instances or adapted instances.

---

Currently, we use load(oid) to load current data for an object
unless we've recieved an invalidation for it (the object). This leads to
complex code that:

• checkes to see if an object has been invalidated and if not:
• loads the object using load, and
• if while loading, the object was invalidated, it then reloads using
  loadBefore.

The IStorage.load method is problematic because it's required to
load current data. This leads to delicate ordering of load
results and invalidations, which also provide information about the
current version of an object. In FileStorage, we also have to get a
read lock on the file before we load current data, to prevent writes.

In reviewing NEO, I realized that there's no reason for load to
exist, and in fact, NEO doesn't really implement it.

First of all, it makes no sense to use load if the transaction
time (Connection._txn_time) is known. We should just use
loadBefore with the transaction time.

Second, one can choose a suitable transaction time at the beginning of
a transaction. A suitable transaction time is just past (because we
use loadBefore) the database transaction time. We don't want to
read any data that was written later, regardless of what we've gotten
invalidations for. We can use IStorage.lastTransaction() to
determine the last transaction id(/time). For storages with remote
data, like ZEO or NEO, we might choose to request the last transaction
id from the remote server to make sure we've processed any in-flight
transactions. This is what NEO does. (More about this in a separate
issue.)

By determining the transaction time at the start of a transaction, we
can:

• Always call loadBefore which can be much more efficiently
  implemented than load().
• Greatly simplify Connection._setstate.
• Greatly simplify logic of ZEO, and perhaps NEO or RelStorage.

So, in conclusion:

• Reimplement transaction handling in Connection to set _txn_time
  at the beginning of a transaction using
  p64(u64(storage.lastTransaction()) + 1).

• Simplify _setstate by always calling loadBefore using _txn_time.

  Update get(oid) in a similar fashion. (This is probably not as big
  an issue in this case, as get(oid) doesn't load object state,
  but I still want to deprecate load.)

5.0.0. DemoStorage Exception

Hi,

In line 330 of DemoStorage.py (see below) we are not returning after we calling self.changes.storeBlob so the previous attribute error is always raised.

Thanks a lot,

Carlos

       if self._blobify():
            self.changes.storeBlob(
                oid, oldserial, data, blobfilename, '', transaction)
        raise

POSError as a subclass from BaseException?

Python 2.5 brought us BaseException, the parent Exception of Exception.
If POSError was a Subclass of BaseException, It would not be a base class or Exception any longer.

While bare exceptions would still be dangerous,
this:

try:
    something_weird()
except ConflictError:
    raise
except:
    do_something()

could be replaced with:

try:
    something_weird()
except Exception:
    do_something()

Any comments?

Explicit PyPy Support (in progress)

ZODB currently does not cleanly pass a test run under PyPy --- there are numerous failures.

Part of this is due to the lack of noload in PyPy's "cPickle" module (which is actually implemented in Python) --- this is easily solved through the use of zodbpickle (which is already a dependency). At least part of the rest of it is due to differences in the C and pure-Python implementations of persistent (e.g., the Python PickleCache lacks some size attributes found in the C version).

Hopefully the rest are minor differences, but I'm not sure yet. I'm working on a branch at https://github.com/NextThought/ZODB/tree/pypy to find out, and hopefully fix them (looks like some other PRs in persistent will be useful, too).

I'm opening this issue to let people know someone is working on this, and maybe to have a place to ask for help if I get stuck :)

ZODB.FileStorage.format: TxnHeader cannot handle Unicode 'descr'

When adding an object with a Unicode name in Pyramid, the application stores the path via 'transaction.note()', which leads to this error during commit::

Traceback (most recent call last):
  File "/home/tseaver/projects/agendaless/agendaless.com/eggs/ZEO-4.0.0b1-py3.3.egg/ZEO/zrpc/connection.py", line 469, in handle_request
    ret = meth(*args)
  File "/home/tseaver/projects/agendaless/agendaless.com/eggs/ZEO-4.0.0b1-py3.3.egg/ZEO/StorageServer.py", line 447, in vote
    return self._try_to_vote()
  File "/home/tseaver/projects/agendaless/agendaless.com/eggs/ZEO-4.0.0b1-py3.3.egg/ZEO/StorageServer.py", line 487, in _try_to_vote
    serials = self.storage.tpc_vote(self.transaction)
  File "/home/tseaver/projects/agendaless/agendaless.com/eggs/ZODB-4.0.0b3-py3.3.egg/ZODB/FileStorage/FileStorage.py", line 717, in tpc_vote
    self._file.write(h.asString())
  File "/home/tseaver/projects/agendaless/agendaless.com/eggs/ZODB-4.0.0b3-py3.3.egg/ZODB/FileStorage/format.py", line 287, in asString
    return b"".join(map(as_bytes, [s, self.user, self.descr, self.ext]))
  File "/home/tseaver/projects/agendaless/agendaless.com/eggs/ZODB-4.0.0b3-py3.3.egg/ZODB/utils.py", line 90, in as_bytes
    return str(obj).encode("ascii")
UnicodeEncodeError: 'ascii' codec can't encode character '\u20ac' in position 24: ordinal not in range(128)

Documentation

If you look in

https://github.com/zopefoundation/ZODB/tree/ZODB5/doc

There are three files.

zodb-guide.txt refers to a url, perhaps should be updated to be zodb.org

storage.pdf from 2004 still talks about versions. Should that document still be there.
How about renaming it to 2004-storage.pdf to give readers a hint as to its age.

And History was last updated in 2011. maybe that should be merged with changes.rst int he root folder.

POSKeyError during commit after savepoint rollback

When a BTree is added and modified in a savepoint before this savepoint is rolled back, consecutive transaction commit fails with a POSKeyError.

This is triggered in certain cases in Plone 3 to 4 migration.

See also http://rpatterson.net/blog/poskeyerror-during-commit

Nondeterministic failure in testBogusObject (ZODB.tests.testCache.CacheErrors)?

I saw this error once

Failure in test testBogusObject (ZODB.tests.testCache.CacheErrors)
Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/case.py", line 329, in run
    testMethod()
  File "/home/mg/src/zopefoundation/ZODB/src/ZODB/tests/testCache.py", line 367, in testBogusObject
    self.assertEqual(sys.getrefcount(None), nones)
  File "/usr/lib/python2.7/unittest/case.py", line 513, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/usr/lib/python2.7/unittest/case.py", line 506, in _baseAssertEqual
    raise self.failureException(msg)
AssertionError: 172989 != 172967

but was unable to reproduce it.

I was using coverage run -m zope.testrunner --test-path=src on Python 2.7.

Opening a blobstorage w/o read permissions ignores blobs

Coming from rejected issue zodb/relstorage#12 over here as @jamadden suggested, because it seems more a general issue.

Problem:
If a filestorage + blobstorage was opened and the blobstorage directory has not enough permissions to access the blobs, then the blobs are just ignored.

Expectations:
I would at expect an error or at least a very visible warning if a read/write storage has non-read-/writeable blobs and if a read-only storage has no readable blobs.

Fix possible data corruption after FileStorage is truncated to roll back a transaction

See: https://mail.zope.org/pipermail/zodb-dev/2014-February/015182.html

I believe Julien is going to submit a PR.

HistoricalStorageAdapter does not forward release to its base

This causes a connection leak on RelStorage.