[ZODB-Dev] Copying persistent objects between databases

[Tim Peters]

[Thomas Guettler]
> I need to copy some objects from one database to an other. Both are
> accessed with ZEO. It will use one thread.  Are there any pitfalls?

I'm sure there are many, but as it's something I haven't tried, I'm not sure
what they may all be.  For starters, I expect you'll see a lot of these:

ZODB.POSException.InvalidObjectReference: Attempt to store an object
                    from a foreign database connection

> I would like to abort the (readonly) transaction which reads the data
> from the source and commit the transaction which writes the data to the
> destination.
>
> Which transaction does get_transaction() return?

The transaction associated with the current thread.  If you have only one
thread, then by default there is only one transaction object.  If you don't
want the "one thread <=> one transaction" default, you could perhaps worm
around it via the setLocalTransaction() and getTransaction() methods on
Connection objects.  But you still (AFAICT) can't load an object in one
connection and store it via a different connection.

> Is this possible at all?

I don't know.  Offhand I don't see an obvious way.

> I could write the objects to pickles and read them again,

Are you sure of that?  What if a persistent object P has a reference to
another persistent object O?  P's pickle will refer to O's oid then, and,
e.g., there's no reason to suppose O will have the *same* oid in the other
database.  Sounds intractable to me.

> this would avoid two open connections. But the first solution would be
> better.

I must have missed that one <wink>.

> Version: ZODB 3.2.1

It's possible that ZODB's ExportImport.py is relevant.  Here are the docs:

    """Support for database export and import."""

Anyway, Connection inherits from the ExportImport class there, and it
appears to contain code for writing an object to a file in such a way that
it can be read up again.  I've never used it.  There's one test for it,
checkExportImport() in ZODB/tests/testZODB.py.  Staring at that, my first
attempt to use it "worked":

"""
import ZODB
from ZODB.FileStorage import FileStorage as fs
from BTrees.OOBTree import *

fs1 = fs('Data1.fs')
db1 = ZODB.DB(fs1)
cn1 = db1.open()
# Create a persistent object that refers to another.
b = OOBTree([(1, 2), (3, OOBTree([(8, 9)]))])
cn1.root()['tree'] = b
get_transaction().commit()
print 'original b'
print list(b.items())
print '3 ->', list(b[3].items())

# "Export" b to a temp file.
import tempfile
f = tempfile.TemporaryFile()
cn1.exportFile(b._p_oid, f)

# Rewind the temp file for reading.
f.seek(0)

# Try to clone b to a different .fs.
fs2 = fs('Data2.fs')
db2 = ZODB.DB(fs2)
cn2 = db2.open()

# Import b from the temp file.
new_b = cn2.importFile(f)
cn2.root()['tree'] = new_b
get_transaction().commit()
print 'cloned b'
print list(new_b.items())
print '3 ->', list(new_b[3].items())

db1.close()
db2.close()
"""

This kind of thing may well be why ExportImport.py exists, but I really
don't know.  Anyone know better?  Someone must <wink>.