[ZODB-Dev] zodb conversion questions

Thu Feb 14 20:56:01 UTC 2013

Hi,

Will be cool if you create a package with this script and with instructions
to patch ZODB and maybe new major version could have the new API.

If you need help I'm interested to do it. :-)

Cheers,

On Thu, Feb 14, 2013 at 6:29 PM, Jürgen Herrmann <Juergen.Herrmann at xlhost.de
> wrote:

> Am 07.02.2013 17:11, schrieb Jim Fulton:
>
>> On Thu, Feb 7, 2013 at 10:48 AM, Jürgen Herrmann
>>
>> <Juergen.Herrmann at xlhost.de> wrote:
>>
>>> Am 06.02.2013 15:05, schrieb Jürgen Herrmann:
>>>
>>>  Hi there!
>>>>
>>>> I hav a relstorage with mysql backend that grew out of bounds
>>>> and we're looking into different backend solutions now. Possibly
>>>> also going back to FileStorage and using zeo...
>>>>
>>>> Anyway we'll have to convert the databases at some point. As these
>>>> are live DBs we cannot shut them down for longer than the
>>>> ususal maintenance interval during the night, so for maybe 2-3h.
>>>>
>>>> a full conversion process will never complete in this time so
>>>> we're looking for a process that can split the conversion into
>>>> two phases:
>>>>
>>>> 1. copy transactions from backup of the source db to the destination
>>>>    db. this can take a long time, we don't care. note the last
>>>>    timestamp/transaction_id converted.
>>>> 2. shut down the source db
>>>> 3. copy transactions from the source db to the destination db, starting
>>>>    at the last converted transaction_id. this should be fast, as only
>>>>    a few transactions need to be converted, say < 1% .
>>>>
>>>>
>>>> if i would reimplement copyTransactionsFrom() to accept a start
>>>> transaction_id/timestamp, would this result in dest being an exact
>>>> copy of source?
>>>>
>>>> source = open_my_source_storage()
>>>> dest = open_my_destination_storage()
>>>> dest.copyTransactionsFrom(**source)
>>>> last_txn_id = source.lastTransaction()
>>>> source.close()
>>>> dest.close()
>>>>
>>>> source = open_my_source_storage()
>>>> # add some transactions
>>>> source.close()
>>>>
>>>> source = open_my_source_storage()
>>>> dest = open_my_destination_storage()
>>>> dest.copyTransactionsFrom(**source, last_txn_id=last_txn_id)
>>>> source.close()
>>>> dest.close()
>>>>
>>>
>>>
>>> I will reply to myself here :) This actually works, tested with a
>>> modified version of FileStorage for now. I modified the signature
>>> of copyTransactionsFrom to look like this:
>>>
>>> def copyTransactionsFrom(self, source, verbose=0, not_before_tid=None):
>>>
>>
>> ``start`` would be better to be consistent with the iterator API.
>>
>>  not_before_tid is a packed tid or None, None meaning "copy all"
>>> (the default, so no existing API usage would break).
>>>
>>> Is there public interest in modifying this API permamently?
>>>
>>
>> +.1
>>
>> This "API" is a bit of an attractive nuisance.  I'd rather people
>> learn how to use iterators in their own scripts, as they are very
>> useful and powerful.  This API just hides that.
>>
>> The second part, replaying old transactions is a bit more subtle,
>> but it's still worth it for people to be aware of it.
>>
>> If I were doing this today, I'd make this documentation
>> rather than API. But then, documentation ... whimper.
>>
>>  Anybody want to look at the actual code changes?
>>>
>>
>> Sure, if they have tests.  Unfortunately, we can only accept
>> pull requests from zope contributors. Are you one?
>> Wanna be one? :)
>>
>> Jim
>>
>
> 1. SUCCESS. I migrated to FileStorage/ZEO successfully with the modified
>    copyTransactionsFrom() with downtime of only 10 minutes. Very cool :)
>    DB sizes are reasonable again and repozo backups are set up, just like
>    before the migration to RelStorage. Feels robust again, a good feeling.
>    I'd advise anybody thinking about migrating to RelStorage to think
>    about a proper (incremental) backup strategy for the underlying sql db
>    first, this might become crucial! repozo makes your life easy there
>    when using FileStorage.
>
> 2. Regarding the tests: I did not find any test for copyTransactionsFrom()
>    in the current test suite, am I blind? If there isn't any test yet,
>    maybe you could suggest a proper file for the test, maybe even a similar
>    one to start from?
>
> 3. Regarding my request to have this in ZODB 3.10.5:
>    After reading my own statement I quickly realized that changing an API
>    can only happen in a major version. Anyway, this was a one-shot action
>    so I'll probably never need my own code again :)
>
> best regards,
> Jürgen
>
> --
>
>> XLhost.de ® - Webhosting von supersmall bis eXtra Large <<
>>>
>>
> XLhost.de GmbH
> Jürgen Herrmann, Geschäftsführer
> Boelckestrasse 21, 93051 Regensburg, Germany
>
> Geschäftsführer: Jürgen Herrmann
> Registriert unter: HRB9918
> Umsatzsteuer-**Identifikationsnummer: DE245931218
>
> Fon:  +49 (0)800 XLHOSTDE [0800 95467833]
> Fax:  +49 (0)800 95467830
> Web:  http://www.XLhost.de
> ______________________________**_________________
> For more information about ZODB, see http://zodb.org/
>
> ZODB-Dev mailing list  -  ZODB-Dev at zope.org
> https://mail.zope.org/mailman/**listinfo/zodb-dev<https://mail.zope.org/mailman/listinfo/zodb-dev>
>

-- 
Rudá Porto Filgueiras
http://python-blog.blogspot.com
http://twitter.com/rudaporto
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.zope.org/pipermail/zodb-dev/attachments/20130214/312fd8d0/attachment.html>