[ZODB-Dev] Synchronizing Mirror Sites

[Norfleet, Sheppard S.]

Shane,

Nice to hear from you again.  

>Is there any chance that the data from the master and secondaries will need
to be merged?

Yes

>Or does each node write exclusively to a particular part of the database?

No

>If you have to merge data, you probably need to replicate at the
application level.

Ok, in this case how would I go about replication at the application level.
This is my first foray into ZODB.  Is there already a replication service
that doesnt require the master server's IP to be always available(queued). I
believe ZEO requires the IP to be available at any time.

>If you don't need to merge data, you can use rsync or
copyTransactionsFrom()

I thought copyTransactionsFrom copied all the transactions in a database,
not just those from a point in time.

Possible Idea:
During the course of the day, each transaction is copied to a separate
storage unit.  At the appropriate time of day a replication script transmits
that storage unit to every othe r node which then copysTransactionsFrom()
that storage unit.  On completion that storage unit is destroyed and a fresh
put in its place for the process to start over.  Would that not accomplish
the merging aspect?  

>Alternatively, you can use Ape with a version control tool to merge data at
the database level.

Sorry, what would the effect of a corrupted ZODB cache be?  Not good, eh?

Thanks for the response

Shep










-----Original Message-----
From: Shane Hathaway [mailto:shane at zope.com]
Sent: Monday, July 21, 2003 2:04 PM
To: Norfleet, Sheppard S.
Cc: 'zodb-dev at zope.org'
Subject: Re: [ZODB-Dev] Synchronizing Mirror Sites


Norfleet, Sheppard S. wrote:
> It seems like all the available replication is centered on an available IP
> connection. The environment I am in, requries that I store up changes, and
> at a certain time of the day, burst the changes to master server, that
will
> send me updates from other servers.  Only in that short window of time is
> the master server available.
> 
> Therefore, I need to find a way to store the transactions in an outbound
> queue, and handle incoming transactions from an inbound source.  Does
anyone
> have any ideas about how I can accomplish this?

Is there any chance that the data from the master and secondaries will 
need to be merged?  Or does each node write exclusively to a particular 
part of the database?  If you have to merge data, you probably need to 
replicate at the application level.  If you don't need to merge data, 
you can use rsync or copyTransactionsFrom(), momentarily taking a node 
offline while importing data.  You might even take advantage of the 
restore() API to avoid the short downtime.

Alternatively, you can use Ape with a version control tool to merge data 
at the database level.  I'd like someone to try this, but I can't 
recommend it quite yet, since it wouldn't update the ZODB cache properly 
yet.

Shane