[ZODB-Dev] zodb conversion questions

[Jürgen Herrmann]

Am 07.02.2013 17:11, schrieb Jim Fulton:
> On Thu, Feb 7, 2013 at 10:48 AM, Jürgen Herrmann
> <Juergen.Herrmann at xlhost.de> wrote:
>> Am 06.02.2013 15:05, schrieb Jürgen Herrmann:
>>
>>> Hi there!
>>>
>>> I hav a relstorage with mysql backend that grew out of bounds
>>> and we're looking into different backend solutions now. Possibly
>>> also going back to FileStorage and using zeo...
>>>
>>> Anyway we'll have to convert the databases at some point. As these
>>> are live DBs we cannot shut them down for longer than the
>>> ususal maintenance interval during the night, so for maybe 2-3h.
>>>
>>> a full conversion process will never complete in this time so
>>> we're looking for a process that can split the conversion into
>>> two phases:
>>>
>>> 1. copy transactions from backup of the source db to the 
>>> destination
>>>    db. this can take a long time, we don't care. note the last
>>>    timestamp/transaction_id converted.
>>> 2. shut down the source db
>>> 3. copy transactions from the source db to the destination db, 
>>> starting
>>>    at the last converted transaction_id. this should be fast, as 
>>> only
>>>    a few transactions need to be converted, say < 1% .
>>>
>>>
>>> if i would reimplement copyTransactionsFrom() to accept a start
>>> transaction_id/timestamp, would this result in dest being an exact
>>> copy of source?
>>>
>>> source = open_my_source_storage()
>>> dest = open_my_destination_storage()
>>> dest.copyTransactionsFrom(source)
>>> last_txn_id = source.lastTransaction()
>>> source.close()
>>> dest.close()
>>>
>>> source = open_my_source_storage()
>>> # add some transactions
>>> source.close()
>>>
>>> source = open_my_source_storage()
>>> dest = open_my_destination_storage()
>>> dest.copyTransactionsFrom(source, last_txn_id=last_txn_id)
>>> source.close()
>>> dest.close()
>>
>>
>> I will reply to myself here :) This actually works, tested with a
>> modified version of FileStorage for now. I modified the signature
>> of copyTransactionsFrom to look like this:
>>
>> def copyTransactionsFrom(self, source, verbose=0, 
>> not_before_tid=None):
>
> ``start`` would be better to be consistent with the iterator API.
>
>> not_before_tid is a packed tid or None, None meaning "copy all"
>> (the default, so no existing API usage would break).
>>
>> Is there public interest in modifying this API permamently?
>
> +.1
>
> This "API" is a bit of an attractive nuisance.  I'd rather people
> learn how to use iterators in their own scripts, as they are very
> useful and powerful.  This API just hides that.
>
> The second part, replaying old transactions is a bit more subtle,
> but it's still worth it for people to be aware of it.
>
> If I were doing this today, I'd make this documentation
> rather than API. But then, documentation ... whimper.
>
>> Anybody want to look at the actual code changes?
>
> Sure, if they have tests.  Unfortunately, we can only accept
> pull requests from zope contributors. Are you one?
> Wanna be one? :)
>
> Jim

1. SUCCESS. I migrated to FileStorage/ZEO successfully with the 
modified
    copyTransactionsFrom() with downtime of only 10 minutes. Very cool 
:)
    DB sizes are reasonable again and repozo backups are set up, just 
like
    before the migration to RelStorage. Feels robust again, a good 
feeling.
    I'd advise anybody thinking about migrating to RelStorage to think
    about a proper (incremental) backup strategy for the underlying sql 
db
    first, this might become crucial! repozo makes your life easy there
    when using FileStorage.

2. Regarding the tests: I did not find any test for 
copyTransactionsFrom()
    in the current test suite, am I blind? If there isn't any test yet,
    maybe you could suggest a proper file for the test, maybe even a 
similar
    one to start from?

3. Regarding my request to have this in ZODB 3.10.5:
    After reading my own statement I quickly realized that changing an 
API
    can only happen in a major version. Anyway, this was a one-shot 
action
    so I'll probably never need my own code again :)

best regards,
Jürgen
-- 
>> XLhost.de ® - Webhosting von supersmall bis eXtra Large <<

XLhost.de GmbH
Jürgen Herrmann, Geschäftsführer
Boelckestrasse 21, 93051 Regensburg, Germany

Geschäftsführer: Jürgen Herrmann
Registriert unter: HRB9918
Umsatzsteuer-Identifikationsnummer: DE245931218

Fon:  +49 (0)800 XLHOSTDE [0800 95467833]
Fax:  +49 (0)800 95467830
Web:  http://www.XLhost.de